Distributed storage data writing method and device

Abstract

The invention discloses a distributed storage data writing method and device. The method comprises the following steps: constructing a writing channel of a user space file system and bottom file storage; according to write-in operation triggered by the large model training task, a user space file system is directly connected with an underlying file storage, and the task data is written into the underlying file storage based on a write-in channel; and synchronizing the written task data with the distributed storage cluster. The writing operation triggered by the large model training task is directly written into the underlying file storage through the writing channel between the user space file system and the underlying file storage, so that the writing operation interruption risk of the large model training task can be avoided, the writing failure rate is reduced, and the storage stability is improved.

Description

Technical Field

[0001] This application relates to the field of data storage technology, specifically to a distributed storage data writing method and apparatus. Background Technology

[0002] Training large-scale models demands substantial data, requiring high-throughput, high-capacity data storage media. Data access and processing involve data cleaning and manipulation, accessing underlying data access interfaces. Unified data access handling is necessary to shield heterogeneous underlying storage and simplify upper-layer development, integration, and maintenance. Furthermore, data storage must ensure high availability and stable read / write performance to prevent fluctuations in I / O operations, node failures, and other anomalies from impacting training progress and computational resource utilization. Based on these requirements, basic storage, such as a single distributed file system, is insufficient for large-scale model training. Therefore, data storage is constructed using underlying file storage (including distributed file systems), distributed storage clusters, and user-space file systems. The user-space file system maps network resources to a local file system for easy user operation. It allows direct access to the distributed storage cluster, which can synchronize with the distributed file system, pre-loading data for caching acceleration, ensuring data access stability, and maintaining data consistency between the distributed file system and the distributed storage cluster.

[0003] However, training large models involves data writing. When a distributed storage cluster uses a single-replica write approach, a node failure can trigger a write failure. The asynchronous multi-replica write strategy also faces the same risk in scenarios where worker nodes fail. Write failures can trigger exceptions, leading to the loss of transmitted data, resulting in training progress losses for large models. Data inconsistency can also easily cause feature engineering failures. Especially in high-concurrency write scenarios, the cascading failure effect can drastically increase the cluster-level write failure rate, creating a core stability bottleneck in the large model training process. Summary of the Invention

[0004] In view of the above problems, embodiments of this application are proposed to provide a distributed storage data writing method and apparatus that overcomes or at least partially solves the above problems.

[0005] According to a first aspect of the embodiments of this application, a distributed storage data writing method is provided, comprising:

[0006] Construct a write channel between the user-space file system and the underlying file storage;

[0007] Based on the write operation triggered by the large model training task, the user space file system directly connects to the underlying file storage, and writes the task data to the underlying file storage based on the write channel.

[0008] The written task data is synchronized with the distributed storage cluster.

[0009] Optionally, the underlying file storage includes a distributed file system;

[0010] The write operation triggered by the large model training task, which involves direct connection between the user-space file system and the underlying file storage, further includes writing task data to the underlying file storage via the write channel:

[0011] The priority of the task data for the write operation is determined based on the write operation triggered by the large model training task.

[0012] Based on the priority of the task data, the task data is written to the corresponding storage layer of the distributed file system through the write channel; the priority is determined according to the usage frequency of the task data.

[0013] Optionally, writing the task data to the corresponding storage layer of the distributed file system based on the write channel, according to the priority of the task data, further includes:

[0014] Based on the write channel, the first priority task data is written to the solid-state drive storage layer of the distributed file system; the first priority task data includes training result data.

[0015] Optionally, writing the task data to the corresponding storage layer of the distributed file system based on the write channel, according to the priority of the task data, further includes:

[0016] Based on the write channel, the second priority task data is written to the mechanical hard disk storage layer of the distributed file system using a multi-replica method; the second priority task data includes preprocessed data.

[0017] Optionally, synchronizing the written task data with the distributed storage cluster further includes:

[0018] Based on the priority of the task data, the task data written with different synchronization granularities is synchronized with the distributed storage cluster.

[0019] Optionally, the step of synchronizing the task data written with the distributed storage cluster using different synchronization granularities according to the priority of the task data further includes:

[0020] Once the first-priority task data is written, it is immediately synchronized with the distributed storage cluster based on the written first-priority task data.

[0021] Optionally, the step of synchronizing the task data written with the distributed storage cluster using different synchronization granularities according to the priority of the task data further includes:

[0022] After the second-priority task data is written, it is synchronized with the distributed storage cluster according to the preset synchronization period.

[0023] According to a second aspect of the embodiments of this application, a distributed storage data writing apparatus is provided, comprising:

[0024] The channel module is suitable for building write channels between the user-space file system and the underlying file storage;

[0025] The direct-connect storage module is suitable for write operations triggered by large model training tasks. It is directly connected to the underlying file storage via the user space file system and writes task data to the underlying file storage based on the write channel.

[0026] A synchronization module is adapted to synchronize the written task data with the distributed storage cluster.

[0027] According to a third aspect of the embodiments of this application, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0028] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-described distributed storage data writing method.

[0029] According to a fourth aspect of the present application, a computer storage medium is provided, the storage medium storing at least one executable instruction, the executable instruction causing a processor to perform an operation corresponding to the distributed storage data writing method described above.

[0030] According to a fifth aspect of the present application, a computer program product is provided, including at least one executable instruction that causes a processor to perform an operation corresponding to the distributed storage data writing method described above.

[0031] According to the distributed storage data writing method and apparatus provided in this application, the write operation triggered by the large model training task is directly written to the underlying file storage through the write channel between the user space file system and the underlying file storage, which can avoid the risk of interruption of the write operation of the large model training task, reduce the write failure rate, and improve storage stability.

[0032] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0033] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0034] Figure 1 A flowchart of a distributed storage data writing method according to an embodiment of this application is shown;

[0035] Figure 2 This diagram illustrates an existing worker node write failure.

[0036] Figure 3 This diagram illustrates a user-space file system performing task data writing based on a write channel directly connected to the underlying file storage.

[0037] Figure 4 A flowchart of a distributed storage data writing method according to another embodiment of this application is shown;

[0038] Figure 5 This diagram illustrates the intensive writing process for a large model training task.

[0039] Figure 6 A schematic diagram of a distributed storage data writing apparatus according to an embodiment of this application is shown;

[0040] Figure 7 A schematic diagram of the structure of a computing device according to an embodiment of this application is shown. Detailed Implementation

[0041] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0042] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0043] User-space file system: By providing a simplified API library, it supports non-privileged users to securely create and mount file systems without requiring in-depth knowledge of kernel module programming, significantly lowering the development threshold. It maps network resources to the local file system, allowing users to treat remote files as local files.

[0044] Underlying file storage: The storage space of underlying file storage is not managed by the distributed storage cluster and can store data from external file systems, including distributed file systems. A distributed storage cluster can connect to one or more underlying file stores and present these underlying storage systems uniformly within a single namespace. Underlying file storage is designed for persistent storage of large amounts of data over a considerable period.

[0045] Distributed file systems: The physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network; or a complete hierarchical file system formed by combining several different logical disk partitions or volume labels; it is a highly fault-tolerant system, suitable for deployment on inexpensive machines, providing high-throughput data access, and is very suitable for applications on large-scale datasets.

[0046] Distributed storage clusters bridge the gap between computing frameworks and storage systems, enabling the computing framework to connect to multiple independent storage systems through a common interface, thus isolating computing from storage. Distributed storage clusters unify data across different storage systems, providing a unified interface for upper-layer data frameworks. They can comprehensively utilize various storage resources such as memory, solid-state drives (SSDs), and hard disks to provide distributed shared caching services, facilitating the caching of frequently accessed data, providing near-memory-level I / O throughput, and improving stability.

[0047] Management node: A node in the distributed storage cluster that stores and manages all metadata of the distributed file system, including file and directory structure information, permission settings, timestamps, etc., so that the required data blocks can be quickly located.

[0048] Worker nodes: These are nodes in a distributed storage cluster. They manage local storage space and store data blocks cached from the underlying distributed storage system, including files or objects from distributed file systems. This reduces direct access to the underlying storage and improves access speed. When the computing framework needs to access data, the worker nodes transfer the cached data to the nodes of the computing framework.

[0049] Solid State Drive (SSD): Uses flash memory as the storage medium, resulting in faster read speeds compared to mechanical hard drives. SSDs do not have read / write heads, have virtually zero seek time, and data is read and written directly via electronic signals, eliminating the need for physical moving parts.

[0050] Hard disk drive (HDD): A traditional hard drive mainly consists of platters, read / write heads, a spindle and control motor, a head controller, a data converter, an interface, and a cache. The read / write heads can be positioned at specific locations on the platter to perform data read / write operations. Information is written to the disk by an electromagnetic current that changes the polarity of the head, which is very close to the magnetic surface.

[0051] Figure 1 A flowchart of a distributed storage data writing method according to an embodiment of this application is shown, as follows: Figure 1 As shown, the method includes the following steps:

[0052] Step S101: Construct a write channel between the user space file system and the underlying file storage.

[0053] A storage structure comprised of a user-space file system, a distributed storage cluster, and underlying file storage provides fast and convenient data retrieval and storage services for large models during training. Accessing files through the user-space file system is as easy and quick as operating local data. The distributed storage cluster can pre-read data from the underlying file storage; the management node stores structural information such as file directories, while the worker nodes store the files, objects, and other data from the underlying file storage. Accessing the distributed storage cluster through the user-space file system allows for quick retrieval of required files and data by locating the data first through the management node, thus improving data acquisition efficiency. Data consistency is maintained between the distributed storage cluster and the underlying file storage through synchronization and other processes.

[0054] When a large model training task is completed and data needs to be written, the existing writing method is as follows: Figure 2 As shown, large model training tasks write data through the user-space file system. The user-space file system is then routed through a distributed storage cluster. The management node of the distributed storage cluster creates metadata storage, data is written to the corresponding worker nodes, and finally, synchronization with the underlying file storage is performed. However, this writing method will fail when a worker node fails. In cases with multiple worker nodes, if a single worker node fails, even if other worker nodes successfully write, the entire writing process will still trigger an ABORT exception, resulting in the loss of transmitted data. To address these issues, this embodiment constructs a write channel between the user-space file system and the underlying file storage, as follows: Figure 3 As shown, the user-space file system can directly connect to the underlying file storage for data writing through the write channel.

[0055] The write channel can be constructed by building an intermediate layer between the user-space file system and the underlying file storage, allowing the user-space file system to call the intermediate layer to write data to the underlying file storage. The intermediate layer can use access interfaces suitable for the underlying file storage, etc., and there are no restrictions here.

[0056] Step S102: Based on the write operation triggered by the large model training task, the task data is written to the underlying file storage via a direct connection from the user space file system and a write channel.

[0057] During the training of large models, write operations are involved. Task data generated during the training process is first written to storage and then retrieved when needed. This includes preprocessed data and training results. Preprocessed data includes data after various preprocessing steps, such as cleaning, normalization, standardization, and outlier handling. Training results include data generated during large model training, such as data from intermediate and output layers, which can be used for subsequent training or applied to specific scenarios.

[0058] The large model training task can be configured with specific write operations based on the specific implementation. When the large model training task triggers a write operation, the task data is written by calling the user-space file system. In this embodiment, the user-space file system can be directly connected to the underlying file storage. Through the write channel between the user-space file system and the underlying file storage, the underlying file storage can be directly manipulated, thereby writing the task data generated by the large model training task to the underlying file storage.

[0059] The underlying file storage can include distributed file systems, which can include storage layers corresponding to different storage media. Different task data can be stored in different storage layers to meet the storage needs of different task data and ensure storage stability. For example, training results data can be stored in an SSD solid-state drive storage layer. SSDs offer fast write speeds and are suitable for frequent small file write tasks, ensuring low latency and high durability. For large model training tasks that frequently switch between model training and data saving, such as saving data after model training and then continuing model training and saving data again, the SSD solid-state drive storage layer can provide efficient and stable data storage. Preprocessed data can be stored in an HDD (hard disk drive) storage layer. Single points of failure can be avoided through methods such as multi-replica fault tolerance. If one replica fails to write data, a new replica can be used to rewrite the task data, ensuring correct and complete data writing. The above are examples; specific settings depend on the implementation and are not limited here.

[0060] Specifically, such as Figure 3As shown, large model training tasks trigger write operations. When data needs to be written, the user-space file system directly connects to the underlying file storage via the write channel shown by the red line on the right, based on the write operation. The underlying file storage can then write the data to the corresponding storage layer based on the write task data. This direct execution of write operations by the underlying file storage improves the stability of distributed data writing.

[0061] Step S103: Synchronize the written task data with the distributed storage cluster.

[0062] After writing the task data to the underlying file storage, subsequent large model training requires reading the written task data. Therefore, after writing the task data, it is also necessary to synchronize the task data written to the underlying file storage with the distributed storage cluster to ensure that the large model training can quickly access the distributed storage cluster and read the corresponding task data through the user-space file system. Figure 3 As shown, after the underlying file storage completes the data writing, it synchronizes with the distributed storage cluster to synchronize the task data written by the underlying file storage to the distributed storage cluster. This makes it convenient for subsequent large model training tasks to read data from the distributed storage cluster through the user space file system for large model training, etc.

[0063] During synchronization, the metadata stored on the management nodes in the distributed storage cluster can be updated based on the task data written to the underlying file storage. This includes updating the directory structure information and time of the task data storage for easier location. It also updates the worker nodes in the distributed storage cluster, caching the corresponding task data in the underlying file storage. If the distributed storage cluster uses a master-slave architecture, the standby management node can also be updated via methods such as logs when updating the master management node; this is not limited here.

[0064] According to the distributed storage data writing method provided in this application, the write operation triggered by the large model training task is directly written to the underlying file storage through the write channel between the user space file system and the underlying file storage. This can avoid the risk of interruption of the write operation of the large model training task, reduce the write failure rate, and improve storage stability.

[0065] Figure 4 A flowchart of a distributed storage data writing method according to an embodiment of this application is shown, as follows: Figure 4 As shown, the method includes the following steps:

[0066] Step S401: Construct a write channel between the user space file system and the underlying file storage.

[0067] Writing task data involves more than just saving the interim results of a large model training task; failure to write the task data can also affect the restart of model training for that large model. For example... Figure 5 As shown, when the original write operations were completed by the worker nodes of the distributed storage cluster, the large model training task would generate task data and save it after executing a segment of model training. After saving, model training would continue, and the generated task data would be saved again. This process would continue, and the task data would be saved again. When the write operation failed on the third save attempt, the impact was not only on the task data but also on the progress loss of the third model training. The model training corresponding to the write failure needed to be restarted after the failure, i.e., restarting the third model training, thus affecting the overall model training progress. Even if the distributed storage cluster uses a multi-node approach, when a single worker node fails and its write operation fails, even if other replicas succeed, the overall write process will still trigger an ABORT exception, resulting in the loss of transmitted data. The large model training task would need to restart the interrupted model training. The multi-replica configuration of the distributed storage cluster not only fails to improve availability but also increases resource consumption. Especially in high-concurrency write scenarios, the failure avalanche effect causes the write failure rate of the distributed storage cluster to increase sharply, becoming the core stability bottleneck of the model training process.

[0068] For write operations in large model training tasks, this embodiment constructs a write channel to establish a channel for write operations between the user-space file system and the underlying file storage. When a large model training task needs to write data, the user-space file system can directly write the task data to the underlying file storage through the write channel. The underlying file storage supports high throughput and high fault tolerance, resulting in high stability of task data writing. Furthermore, the underlying file storage can use different storage media to meet the storage needs of different task data, thereby improving the stability of write operations for large model training tasks and reducing progress loss in large model training tasks.

[0069] Step S402: Based on the write operation triggered by the large model training task, determine the priority of the task data to be written, and write the task data to the corresponding storage layer of the distributed file system based on the write channel according to the priority of the task data.

[0070] When a large model training task triggers a write operation, such as Figure 5The high-density write operations shown can include tasks where the written data can include, for example, training results generated during a model training phase. These results can be used for the next phase of model training, such as as input data. Alternatively, they can include preprocessed data generated from preprocessing data from the previous phase during model training. Preprocessed data ensures the accuracy of feature extraction during model training; if the write operation fails, inconsistent feature extraction may lead to feature engineering failures. Preprocessed data also needs to be written during large model training tasks for later use. The above is an example; the task data can also include various data generated by large model training tasks, depending on the implementation. This is not limited here.

[0071] When a large model training task requires saving task data, a write operation is triggered, such as... Figure 3 As shown, the user-space file system can write data to the underlying file storage based on the write channel. The underlying file storage can include, for example, a distributed file system. Distributed file systems offer high fault tolerance and reliability, automatically saving data copies through mechanisms such as multi-replica storage, distributing them across different nodes to ensure data storage reliability. Distributed file systems can also employ different storage media, such as solid-state drive (SSD) storage layers and hard disk drive (HDD) storage layers. Using SSD storage layers can improve read / write performance, for example, by using flash memory chips for fast read / write speeds; using HDD storage layers can reduce costs and provide large-capacity storage. For task data intensively stored for large model training tasks, if the task data is read and used in real-time after being written, an SSD storage layer can be used to improve storage performance; if the task data needs to be used for a long time, an HDD storage layer can be used to reduce storage costs.

[0072] When writing task data, the priority of the task data can be determined based on its usage frequency. Different priorities allow task data to be written to different storage layers of different storage media, improving storage performance while reducing storage costs. Priorities can include, for example, first priority and second priority. Priority is determined based on the usage frequency of the task data. If task data is frequently accessed and used after storage, its priority is first priority; if it is not frequently accessed and its priority is low, its priority is second priority. The above is an example; specific settings can be configured according to the implementation. Different task data can be assigned different priorities and written according to priority to more specifically meet the needs of different task data. This also provides reliable data retrieval during subsequent large model training tasks, satisfying storage requirements while reducing costs.

[0073] Specifically, for first-priority task data, which includes training result data such as data generated by intermediate or output layers of a large model during training, after being written to the distributed file system, it can be used for the next stage of model training. This means it needs to be read in real-time as input for the next stage of model training. Therefore, the task data with high usage frequency and requiring real-time reading, i.e., training result data, can be designated as first-priority. Different write operations triggered by large model training tasks correspond to different task data. When the task data is training result data, it can be determined as first-priority task data. The user-space file system can write first-priority task data to the solid-state drive (SSD) storage layer of the distributed file system based on the write channel. The SSD storage layer enables fast writing of first-priority task data and has lower access latency. Subsequent synchronization of first-priority task data can be performed using this low latency to improve synchronization efficiency.

[0074] For second-priority task data, which includes preprocessed data such as data cleaning, normalization, standardization, and outlier removal performed on various data during large model training tasks, this preprocessing reduces data noise, ensures data feature consistency, and minimizes interference with large model training. Preprocessed data can be used in large model training tasks for tasks such as feature extraction. Since preprocessed data does not require real-time reading after being written, it can be read as needed by the large model training task. As this data is used less frequently and is not read in real-time, its priority can be determined as second-priority. After preprocessing the data, the large model training task triggers a write operation; at this point, the task data is preprocessed data, thus designated as second-priority task data. The user-space file system can write the second-priority task data to the mechanical hard disk storage layer of the distributed file system via a write channel. The mechanical hard disk storage layer uses high-capacity storage media, facilitating the storage of large amounts of data at a low cost. The distributed file system can employ a multi-replica approach when storing second-priority task data to avoid single-point failures affecting the writing of second-priority task data. Multiple replicas can be configured in a distributed file system, such as with three replicas. If one replica fails to write data, a new replica can be built, and the data for writing second-priority tasks can be recovered from the other replicas, reducing the impact of single-point failures on the writing of second-priority task data. The multiple replicas of a distributed file system can be distributed across different racks, such as different physical locations or logical groups of different physical devices. The multiple replicas are unaffected by the failure of a single replica, and the overall writing of second-priority task data will not be affected by the failure of one replica.

[0075] For task data with different priorities, storage layers using different storage media within a distributed file system are employed. This approach considers both the usage frequency of task data with varying priorities and the cost of the distributed file system, satisfying task data usage requirements while avoiding excessive costs. The underlying file storage described above uses a distributed file system as an example. In practical applications, other file systems, such as distributed object systems, can also be used as the underlying file storage; this is not a limitation here.

[0076] Step S403: Based on the priority of the task data, the task data written with different synchronization granularities is synchronized with the distributed storage cluster.

[0077] After writing task data to different storage layers of the distributed file system according to priority, the usage frequency of different task data is different. Correspondingly, the timing of large model training tasks reading task data will also be different, and the timing of synchronization between the distributed file system and the distributed storage cluster will also be different. It is necessary to use different synchronization granularities to synchronize the written task data according to the priority of the task data, so that the task data with the first priority can meet the real-time reading needs of large model training tasks, while the task data with the second priority can be synchronized periodically to meet the reading needs while reducing the frequent pressure of real-time synchronization.

[0078] Specifically, for first-priority task data, after the first-priority task data is written to the distributed file storage solid-state drive storage layer through the user-space file system, such as... Figure 3 As shown, data needs to be synchronized with the distributed storage cluster so that subsequent large model training tasks can read data from the distributed storage cluster through the user-space file system. First-priority task data includes training result data, which can be used for the next stage of model training. This data has high real-time read requirements and is used frequently. Therefore, after the first-priority task data is written to the solid-state drive storage layer of the distributed file storage, it can be synchronized with the distributed storage cluster immediately to achieve real-time strong consistency between the distributed file storage and the distributed storage cluster. When synchronizing the first-priority task data to the distributed storage cluster, synchronization can be based on the write results when writing the first-priority task data to the distributed file storage. The write results can be stored on the management node of the distributed storage cluster. The distributed storage cluster can update the metadata in the management node based on the write results stored in the management node, such as updating the directory structure information corresponding to the first-priority task data. During synchronization, the worker nodes in the distributed storage cluster also need to be synchronized with the first-priority task data written in the distributed file storage and cached in the worker nodes.

[0079] For second-priority task data, after the second-priority task data is written to the mechanical hard disk storage layer of the distributed file storage through the user-space file system, such as Figure 3 As shown, data needs to be synchronized with the distributed storage cluster so that subsequent large model training tasks can read data from the distributed storage cluster through the user-space file system. Second-priority task data includes preprocessed data, which is obtained after preprocessing the data before the execution of the large model training task. This preprocessed data is used less frequently and does not require real-time reading. Therefore, the second-priority task data written to the distributed file system can be synchronized using a preset synchronization period, such as hourly or daily synchronization. The preset synchronization period can be set or adjusted according to the implementation situation and is not limited here. When synchronizing the second-priority task data to the distributed storage cluster, the preset synchronization period can be obtained first, and then synchronization can be performed according to the preset synchronization period. For example, if the preset synchronization period is hourly, synchronization can be performed every hour on the hour. If the current time reaches the synchronization time, the second-priority task data is synchronized; if the current time has not reached the synchronization time, it waits until the synchronization time arrives before synchronizing the second-priority task data. If the preset synchronization period is daily, a specific time within a day can be set as the synchronization time. If the current time reaches the synchronization time, the second-priority task data is synchronized. If the current time has not reached the synchronization time, the synchronization is waited until the synchronization time arrives before synchronizing the second-priority task data. The above is an example; the specific settings depend on the implementation and are not limited here. When synchronizing second-priority task data, the distributed file storage may have written one or more batches of second-priority task data. All unsynchronized second-priority task data needs to be synchronized. After writing second-priority task data to the distributed file storage, the write results can be stored in the management node of the distributed storage cluster. According to the preset synchronization period, when the synchronization time arrives, the metadata in the management node of the distributed storage cluster is updated based on the write results stored in the management node. This includes updating the directory structure information corresponding to the second-priority task data. During synchronization, the worker nodes in the distributed storage cluster also need to be synchronized with the second-priority task data written to the distributed file storage and cached in the worker nodes.

[0080] Furthermore, the stability of the management node in a distributed storage cluster affects the cache read and write operations of all nodes. Therefore, a distributed storage cluster can adopt a primary-standby structure, including a primary management node and a standby management node, to ensure the stability of the management node. Based on the primary-standby structure, after the primary management node is updated, it can be synchronized with the standby management node through methods such as logs, ensuring that the metadata in the standby management node is synchronized with the metadata of the primary management node during primary-standby failover.

[0081] Furthermore, considering the stability issues of worker nodes in a distributed storage cluster, situations such as mixed storage of memory blocks of different sizes and failure to clean up memory in a timely manner can easily lead to worker node failures. This embodiment can prevent memory overflow problems by scanning worker nodes to promptly detect invalid memory usage and setting warnings for memory usage. For example, a background thread can periodically scan worker nodes and release invalid memory usage, such as memory linked lists with size=0, based on the scan results. A memory warning threshold can also be set, such as setting 80% of the total memory as the warning threshold. When memory usage triggers the warning threshold, unused non-standard capacity memory blocks can be forcibly merged, and garbage collection can be used for coordinated release, reducing the likelihood of memory overflow problems when worker nodes are loading massive amounts of data.

[0082] After synchronization, when a large model training task needs to read relevant task data, such as... Figure 3 As shown, data can be read from the distributed storage cluster via the user-space file system to obtain the corresponding task data. The management node in the distributed storage cluster can first locate the directory information of the task data. Based on this directory information, the task data can be quickly located and read from the worker nodes of the distributed storage cluster, improving read efficiency. Furthermore, if there is an anomaly in the worker nodes of the distributed storage cluster, it may cause the task data reading to fail. Since the task data has already been written to the underlying file storage, a fallback read method can be used to read the task data from the underlying file storage, ensuring the normal execution of data reading for large model training tasks. This is not limited here.

[0083] According to the distributed storage data writing method provided in this application, the write operation triggered by the large model training task determines the priority of the task data based on the task data to be written. Based on different priorities, the task data of different priorities are stored in the corresponding storage layers of the storage media through the write channel between the user-space file system and the underlying file storage. The first priority task data is stored in the solid-state drive storage layer, and the second priority task data is stored in the mechanical hard drive storage layer. This facilitates meeting the read and write requirements of the large model training task for task data of different priorities and also ensures the stability of task data writing. After writing, the task data of different priorities is synchronized to the distributed storage cluster using different synchronization granularities, facilitating retrieval by the large model training task.

[0084] Figure 6 A schematic diagram of a distributed storage data writing device according to an embodiment of this application is shown. Figure 6 As shown, the device includes:

[0085] Channel module 610 is suitable for constructing a write channel between the user space file system and the underlying file storage;

[0086] The direct-connect storage module 620 is suitable for write operations triggered by large model training tasks. It connects directly to the underlying file storage from the user space file system and writes task data to the underlying file storage based on the write channel.

[0087] The synchronization module 630 is suitable for synchronizing the written task data with the distributed storage cluster.

[0088] Optionally, the underlying file storage includes a distributed file system;

[0089] The direct-connect storage module 620 is further adapted to:

[0090] The priority of the task data to be written is determined based on the write operations triggered by the large model training task.

[0091] Based on the priority of the task data, the task data is written to the corresponding storage layer of the distributed file system through the write channel; the priority is determined according to the frequency of use of the task data.

[0092] Optionally, the direct-connect storage module 620 is further adapted to:

[0093] Based on the write channel, the first priority task data is written to the solid-state drive storage layer of the distributed file system; the first priority task data includes training result data.

[0094] Optionally, the direct-connect storage module 620 is further adapted to:

[0095] Based on the write channel, the second-priority task data is written to the mechanical hard disk storage layer of the distributed file system using a multi-replica method; the second-priority task data includes preprocessed data.

[0096] Alternatively, the synchronization module 630 is further adapted to:

[0097] Based on the priority of the task data, the task data written with different synchronization granularities is synchronized with the distributed storage cluster.

[0098] Alternatively, the synchronization module 630 is further adapted to:

[0099] Once the first-priority task data is written, it is immediately synchronized with the distributed storage cluster based on the written first-priority task data.

[0100] Alternatively, the synchronization module 630 is further adapted to:

[0101] After the second-priority task data is written, it is synchronized with the distributed storage cluster according to the preset synchronization period.

[0102] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be repeated here.

[0103] According to the distributed storage data writing device provided in this application, the write operation triggered by the large model training task is directly written to the underlying file storage through the write channel between the user space file system and the underlying file storage. This can avoid the risk of interruption of the write operation of the large model training task, reduce the write failure rate, and improve storage stability.

[0104] This application also provides a non-volatile computer storage medium storing at least one executable instruction that can perform the operation corresponding to the distributed storage data writing method in any of the above method embodiments.

[0105] This application also provides a computer program product comprising at least one executable instruction or computer program that enables a processor to perform the operation corresponding to the distributed storage data writing method in any of the above method embodiments.

[0106] Figure 7 The diagram shows a structural schematic of a computing device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0107] like Figure 7As shown, the computing device may include: a processor 702, a communications interface 704, a memory 706, and a communications bus 708.

[0108] in:

[0109] The processor 702, communication interface 704, and memory 706 communicate with each other via communication bus 708.

[0110] The communication interface 704 is used to communicate with other network elements such as clients or other servers.

[0111] The processor 702 is used to execute program 710, specifically to perform the relevant steps in the above-described distributed storage data writing method embodiment.

[0112] Specifically, program 710 may include program code that includes computer operation instructions.

[0113] The processor 702 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement this application. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0114] Memory 706 is used to store program 710. Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0115] Specifically, program 710 can be used to cause processor 702 to execute the distributed storage data writing method in any of the above method embodiments. The specific implementation of each step in program 710 can be found in the corresponding descriptions of the steps and units in the above distributed storage data writing embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0116] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing preferred embodiments of this application.

[0117] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0118] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0119] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0120] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0121] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0122] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A distributed storage data writing method, comprising: constructing a write channel of a user space file system and an underlying file storage; based on a write operation triggered by a large model training task, writing task data to the underlying file storage by the user space file system directly connecting the underlying file storage based on the write channel; synchronizing the written task data with a distributed storage cluster.

2. The method of claim 1, wherein, The underlying file storage comprises a distributed file system. The write operation triggered by the large model training task, writing task data to the underlying file storage by the user space file system directly connecting the underlying file storage based on the write channel further comprises: determining the priority of the task data of the write operation triggered by the large model training task; based on the priority of the task data, writing the task data to the corresponding storage layer of the distributed file system based on the write channel; the priority is determined according to the frequency of use of the task data.

3. The method of claim 2, wherein, Based on the priority of the task data, writing the task data to the corresponding storage layer of the distributed file system based on the write channel further comprises: based on the write channel, writing task data of a first priority to a solid state disk storage layer of the distributed file system; the task data of the first priority comprises training result data.

4. The method of claim 2, wherein, Based on the priority of the task data, writing the task data to the corresponding storage layer of the distributed file system based on the write channel further comprises: based on the write channel, writing task data of a second priority to a mechanical hard disk storage layer of the distributed file system using a multi-copy method; the task data of the second priority comprises pre-processing data.

5. The method of any one of claims 1-4, wherein, The synchronization of the written task data with the distributed storage cluster further comprises: based on the priority of the task data, synchronizing the written task data with the distributed storage cluster using different synchronization granularities.

6. The method of claim 5, wherein, The synchronization of the written task data with the distributed storage cluster using different synchronization granularities based on the priority of the task data further comprises: after writing task data of a first priority, synchronizing the written task data of the first priority with the distributed storage cluster immediately.

7. The method of claim 5, wherein, The synchronization of the written task data with the distributed storage cluster using different synchronization granularities based on the priority of the task data further comprises: after writing task data of a second priority, synchronizing the written task data of the second priority with the distributed storage cluster according to a preset synchronization period.

8. A distributed storage data writing device, comprising: a channel module adapted to construct a write channel of a user space file system and an underlying file storage; a direct connection storage module adapted to write task data to the underlying file storage based on a write operation triggered by a large model training task by the user space file system directly connecting the underlying file storage based on the write channel; a synchronization module adapted to synchronize the written task data with a distributed storage cluster.

9. A computing device comprising: a processor, a memory, a communication interface, and a communication bus, which enable mutual communication among the processor, the memory, and the communication interface through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation corresponding to the distributed storage data writing method in any one of claims 1-7.

10. A computer storage medium, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform the operation corresponding to the distributed storage data writing method in any one of claims 1-7.

11. A computer program product, comprising at least one executable instruction, and the executable instruction enables a processor to perform the operation corresponding to the distributed storage data writing method in any one of claims 1-7.

Images