Data storage method, device and computer-readable storage medium

Abstract

The present application provides a data storage method, device, and computer-readable storage medium, which relate to the fields of cloud technology and blockchain. The data storage method is applied to a data storage device, the data storage device is used to perform a first task, and the local disk of the data storage device is used to store first data to be stored, which is data generated when the first task is performed. The data storage method includes: running a mounting client, wherein the mounting client is used to store data to a distributed storage device; obtaining second data to be stored generated when a second task is performed, wherein the priority of the second task is lower than the priority of the first task; and sending a data storage request carrying the second data to be stored to the distributed storage device through the running mounting client, so that the distributed storage device responds to the data storage request and stores the second data to be stored in the distributed storage device. Through this application, the mixed deployment effect can be improved.

Description

Technical Field

[0001] The present application relates to storage technology in the field of computer networks, and in particular to a data storage method, device, and computer-readable storage medium. Background Art

[0002] Colocation refers to deploying tasks with different priorities on the same device. For example, in online and offline colocation, online and offline tasks are deployed on the same device, with the online task having a higher priority than the offline task.

[0003] Generally speaking, during colocation, submitting higher-priority tasks typically requires more resources. Furthermore, higher-priority tasks experience peaks and valleys in their execution, especially during valleys, when resources are underutilized. Colocation improves resource utilization by filling the gaps between higher-priority tasks with lower-priority tasks. However, colocation often results in competition for disk space between lower-priority and higher-priority tasks, impacting the normal execution of higher-priority tasks and resulting in poor colocation effectiveness. Summary of the Invention

[0004] The embodiments of the present application provide a data storage method, apparatus, device, and computer-readable storage medium, which can improve the colocation effect.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a data storage method, which is applied to a data storage device, wherein the data storage device is used to perform a first task, and a local disk of the data storage device is used to store first data to be stored, wherein the first data to be stored is data generated when performing the first task, and includes:

[0007] Running a mount client, wherein the mount client is used to store data in a distributed storage device;

[0008] Acquire second data to be stored generated when executing a second task, wherein the priority of the second task is lower than the priority of the first task;

[0009] By running the mount client, a data storage request carrying the second data to be stored is sent to the distributed storage device, so that

[0010] The distributed storage device stores the second data to be stored in the distributed storage device in response to the data storage request.

[0011] An embodiment of the present application provides a data storage device, including:

[0012] A task execution module, configured to execute a first task and store first data to be stored generated when executing the first task to a local disk of the data storage device;

[0013] A client running module, used to run a mounting client, wherein the mounting client is used to store data in a distributed storage device;

[0014] a data acquisition module, configured to acquire second data to be stored generated when executing a second task, wherein the priority of the second task is lower than the priority of the first task;

[0015] The data storage module is used to send a data storage request carrying the second data to be stored to the distributed storage device through the running mounting client, so that the distributed storage device responds to the data storage request and stores the second data to be stored in the distributed storage device.

[0016] In an embodiment of the present application, the data storage device also includes a status detection module for detecting the storage status corresponding to the data storage request; counting the number of states in which the storage status is an uninterruptible sleep state; when the number of states is greater than a quantity threshold, stopping scheduling the next second task.

[0017] In an embodiment of the present application, the state detection module is further configured to reduce the disk read / write rate of the second task when the state quantity is greater than the quantity threshold.

[0018] In an embodiment of the present application, the status detection module is also used to generate an alarm prompt information when the number of the statuses is greater than the number threshold, wherein the alarm prompt information is information used to prompt that the distributed storage device is working abnormally; and display the alarm prompt information.

[0019] In an embodiment of the present application, the state detection module is also used to obtain update information of the state quantity and determine the updated state quantity based on the updated information of the state quantity; when the updated state quantity is less than or equal to the quantity threshold, the next second task is scheduled.

[0020] In an embodiment of the present application, the status detection module is further configured to restore the disk read / write rate of the second task when the number of update states is less than or equal to the number threshold.

[0021] In an embodiment of the present application, the status detection module is also used to generate recovery prompt information when the number of update states is less than or equal to the number threshold, wherein the recovery prompt information is information used to prompt that the operation of the distributed storage device has returned to normal; and display the recovery prompt information.

[0022] In an embodiment of the present application, the data storage device also includes a space limitation module for obtaining the disk space occupancy corresponding to the first task; determining the disk free space amount in combination with the disk space occupancy and the total disk space amount, wherein the total disk space amount is the space amount of the local disk; when scheduling the second task, creating a local storage directory associated with the second task; and generating space limit information for the local storage directory based on the disk free space amount.

[0023] In an embodiment of the present application, the space limitation module is further configured to store the second data to be stored in the local storage directory based on the space limitation information.

[0024] In an embodiment of the present application, the space limitation module is further used to obtain the next disk space occupancy corresponding to the first task; and update the disk free space amount in combination with the next disk space occupancy and the total disk space amount.

[0025] In an embodiment of the present application, the space limitation module is further configured to generate the space limitation information of the local storage directory based on the updated free space amount of the disk.

[0026] In an embodiment of the present application, the space limitation module is also used to build a container corresponding to the second task when scheduling the second task; create at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory; mount at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory to the container corresponding to the second task to obtain the local storage directory.

[0027] In an embodiment of the present application, the space limitation module is also used to generate the space limitation information based on the amount of free space on the disk; and establish an association with the space limitation information for at least one data storage directory in the local storage directory and at least one sub-data storage directory corresponding to each data storage directory.

[0028] In an embodiment of the present application, the space limitation module is also used to obtain the amount of stored space in the local storage directory; determine the amount of storable space based on the amount of free disk space and the amount of stored space corresponding to the space limitation information; when the amount of storage space corresponding to the second data to be stored is less than or equal to the amount of storable space, store the second data to be stored in the local storage directory; when the amount of storage space corresponding to the second data to be stored is greater than the amount of storable space, end the data storage process of the second data to be stored in the local storage directory.

[0029] In an embodiment of the present application, the data storage device further includes a remote storage module for establishing a storage association relationship with a remote storage device.

[0030] In an embodiment of the present application, the remote storage module is also used to send a remote storage request carrying the second data to be stored to the remote storage device based on the storage association relationship, so that the remote storage device responds to the remote storage request and stores the second data to be stored in the remote storage device.

[0031] An embodiment of the present application provides a data storage device, including:

[0032] a memory for storing executable instructions;

[0033] The processor is used to implement the data storage method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0034] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute instructions to implement the data storage method provided in the embodiment of the present application.

[0035] The embodiments of the present application have at least the following beneficial effects: on a data storage device that implements mixed deployment, a virtual disk is mounted by mounting a client, so that the second data to be stored generated when executing a second task with a lower priority can be stored in a distributed storage device, and thus isolated from the storage of the first data to be stored on the local disk when executing the first task on the data storage device; that is, the storage of the first data to be stored for the first task and the storage of the second data to be stored for the second task are independent and do not affect each other, thereby improving the mixed deployment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of an optional architecture of a data storage system provided in an embodiment of the present application;

[0037] Figure 2 is another optional architectural diagram of the data storage system provided in an embodiment of the present application;

[0038] Figure 3 This embodiment of the present application provides Figure 1 A schematic diagram of the composition structure of a server in FIG;

[0039] Figure 4 This is an optional flowchart of the data storage method provided in the embodiment of the present application;

[0040] Figure 5This is a schematic diagram of an exemplary data storage method provided in an embodiment of the present application;

[0041] Figure 6 This is another optional flowchart of the data storage method provided in the embodiment of the present application;

[0042] Figure 7 This is another optional flowchart of the data storage method provided in the embodiment of the present application;

[0043] Figure 8 This is another optional flowchart of the data storage method provided in the embodiment of the present application;

[0044] Figure 9 This is a schematic diagram of the architecture of an exemplary data storage method provided in an embodiment of the present application;

[0045] Figure 10 is a schematic diagram of the architecture of another exemplary data storage method provided in an embodiment of the present application;

[0046] Figure 11 This is a flowchart of an exemplary method of adjusting the execution of a second task based on a storage state, provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0048] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0050] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the embodiments of this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0051] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0052] 1) Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software and network within a wide area network or local area network to realize data computing, storage, processing and sharing.

[0053] 2) A container is a group of processes that are resource-constrained and isolated from each other; container technology is an isolation technology based on operating system capabilities. In this embodiment of the present application, the second task can be executed in the form of a container.

[0054] 3) Container orchestration: Container orchestration refers to the deployment, management, expansion, and networking of automated containers. The same application can be deployed in different environments without redesigning. Container orchestration tools, such as "Kubernetes," or K8s for short, are used to manage containerized applications on multiple hosts in a cloud platform and serve as the foundation of the container cloud platform. Tasks submitted through the container orchestration tool are executed in the form of containers. In this embodiment of the present application, the second task may be submitted through "Kubernetes."

[0055] 4) Control Groups (cgroups) are process groups used to limit resources such as CPU, memory, storage, and network. In this embodiment of the present application, the control group can be used to limit the disk read and write rate of the second task.

[0056] 5) Online tasks refer to at least one service with strong real-time performance. Online tasks usually require relatively few resources but require a short response time, such as web services. In the embodiment of the present application, the first task can be an online task.

[0057] 6) Offline tasks refer to at least one task that is not real-time. Typically, offline tasks do not require a quick response to tasks, but are relatively computationally intensive and resource-intensive, such as machine learning. In this embodiment of the present application, the second task may be an offline task.

[0058] 7) Distributed storage system (Ceph) refers to a physical disk space for file storage management that is not necessarily directly connected to a local node but is connected to nodes via a computer network. Alternatively, it may be a complete, hierarchical storage system formed by combining several different logical disk partitions or volumes, offering high reliability and scalability. In the embodiments of this application, the distributed storage device can be a distributed storage system, and the remote storage device can also be a distributed storage system.

[0059] 8) Block storage: In a block storage system, all data is divided into blocks of fixed size, and each block is assigned a number for addressing. In the embodiment of the present application, the storage format corresponding to the mounted client can be block storage.

[0060] 9) Disk Quota Management (DiskQuota) is used to limit disk space usage. After setting a maximum disk quota, the DiskQuota process monitors disk usage and maintains a blacklist of tasks that exceed the maximum quota. When inserting data into a blacklisted task, the operation is prohibited. In this embodiment of the application, disk quota management can be used to implement a maximum quota limit on the local storage directory, where the maximum quota is the amount of free disk space.

[0061] 10) Storage directories, used to support container storage, such as "emptyDir" and "hostPath" storage directories. The "emptyDir" type of storage directory is automatically created when the "pod" container is assigned to the host machine; the "hostPath" type of storage directory is a directory or file mounted on the host machine by the "pod" container, allowing the container to use the host machine's file system for storage. In this embodiment of the application, the local storage directory is the storage directory.

[0062] 11) Blockchain is an encrypted, chain-like transaction storage structure formed by blocks.

[0063] 12) Blockchain Network: A collection of nodes that incorporate new blocks into the blockchain through consensus.

[0064] Generally speaking, in order to achieve mixed deployment, the remaining space after the data corresponding to the higher priority tasks is usually obtained, and the data corresponding to the lower priority tasks are stored in the remaining space. However, there is no limit on the space occupied by the data corresponding to the lower priority tasks, which may cause the data corresponding to the lower priority tasks to fill the entire disk partition, thereby affecting the normal execution of the higher priority tasks.

[0065] In addition, to achieve colocation, some lower-priority tasks can be deployed on the colocation devices. However, since the number of lower-priority tasks deployed is reduced, this approach is not universal and cannot effectively improve the utilization of resources such as processors and memory.

[0066] For example, when online and offline are mixed, the online task occupies the storage space of the host machine, but there is still remaining free space. The remaining free space can be allocated to the offline task, and the offline task uses the remaining free space resources; since there is no restriction on the storage space allocated to the offline task, when the offline task needs more temporary space resources during a certain period of time, the offline task may occupy more space, which affects the operation of the online task.

[0067] For example, to achieve online and offline colocation, you can deploy only offline tasks that require less disk space, or deploy a fixed number of offline tasks based on available disk space. However, deploying only offline tasks that require less disk space narrows the scope of colocation offline tasks and is not universal. Running only a fixed number of offline tasks also underutilizes idle CPU or memory resources, resulting in low resource utilization.

[0068] Based on this, the embodiments of the present application provide a data storage method, apparatus, device and computer-readable storage medium, which can physically separate the storage of data corresponding to a first task with a higher priority and the data corresponding to a second task with a lower priority, without affecting each other, and have a storage space large enough to store the data corresponding to the second task with a lower priority. It has high versatility and can improve the utilization of resources such as processors and memory.

[0069] The following describes exemplary applications of the data storage device provided in the embodiments of the present application. The data storage device provided in the embodiments of the present application can be implemented as various types of terminals, such as smartphones, smart watches, laptops, tablet computers, desktop computers, smart TVs, set-top boxes, smart car devices, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, and smart speakers. It can also be implemented as a server. The following describes exemplary applications of the data storage device implemented as a server.

[0070] See also Figure 1 , Figure 1 This is an optional architectural diagram of a data storage system provided in an embodiment of the present application; Figure 1 As shown, to support a data storage application, in a data storage system 100, a server 400 (distributed storage device) is connected to a server 200 (data storage device). In addition, the data storage system 100 also includes a terminal 500 connected to the server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0071] Server 200 is configured to receive a first task and a second task submitted by terminal 500 via network 300; execute the first task and store first data to be stored generated during the execution of the first task in a local disk; run a mount client, wherein the mount client is configured to store data to server 400; obtain second data to be stored generated during the execution of the second task, wherein the priority of the second task is lower than that of the first task; and send a data storage request carrying the second data to be stored to server 400 via the running mount client, so that server 400 responds to the data storage request and stores the second data to be stored in a distributed storage device.

[0072] The server 400 is configured to receive a data storage request carrying the second data to be stored sent by the server 200 , and store the second data to be stored in the server 400 in response to the data storage request.

[0073] The terminal 500 is used to submit the first task and the second task to the server 200 via the network 300 .

[0074] In some embodiments of the present application, server 200 and server 400 can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. Terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0075] The space limitation information involved in the data storage method provided in the embodiment of the present application can be stored on the blockchain.

[0076] In addition, in the data storage method provided in the embodiment of the present application, the data storage device can be used as a node on the blockchain; see Figure 2 , Figure 2 This is another optional architectural diagram of the data storage system provided in an embodiment of the present application. Figure 2 In the data storage system 100 shown, data is transmitted to multiple servers (server 400-1 and server 400-2, i.e., Figure 1 The server 400 sends a data storage request carrying the second data to be stored.

[0077] In some embodiments of the present application, server 200, server 400-1, and server 400-2 can join blockchain network 600 and become a node therein. The type of blockchain network 600 is flexible and diverse, and can be, for example, any of a public chain, a private chain, or a consortium chain. Taking a public chain as an example, any electronic device of a business entity can access blockchain network 600 without authorization and serve as a consensus node of blockchain network 600. For example, server 400-1 is mapped as consensus node 600-1 in blockchain network 600, server 200 is mapped as consensus node 600-2 in blockchain network 600, and server 400-2 is mapped as consensus node 600-3 in blockchain network 600.

[0078] Taking blockchain network 600 as a consortium chain, for example, server 200, server 400-1, and server 400-2 can access blockchain network 600 and become nodes after obtaining authorization. Server 200 can send a data storage request containing the second data to be stored to blockchain network 600 for consensus by executing a smart contract. Once consensus is reached, the server then sends the data storage request containing the second data to be stored to servers 400-1 and 400-2. This shows that by having multiple nodes in the blockchain network reach consensus on the data storage request containing the second data to be stored before sending it to servers 400-1 and 400-2, the reliability and accuracy of the transmission of the data storage request containing the second data to be stored can be improved during the data storage process.

[0079] See also Figure 3 , Figure 3 This embodiment of the present application provides Figure 1 A schematic diagram of the composition structure of a server in Figure 3 The server 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 3 Various buses are labeled as bus system 240 .

[0080] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0082] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0083] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0084] In some embodiments of the present application, the memory 250 can store data to support various operations. Examples of these data include programs, modules, and data structures, or subsets or supersets thereof, as exemplarily described below.

[0085] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0086] A network communication module 252 for reaching other computer devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0087] a presentation module 253 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with a user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0088] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0089] In some embodiments of the present application, the data storage device provided in the embodiments of the present application can be implemented in a software manner. Figure 3 The data storage device 255 stored in the memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a task execution module 2551, a client operation module 2552, a data acquisition module 2553, a data storage module 2554, a status detection module 2555, a space restriction module 2556, and a remote storage module 2557. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0090] In other embodiments of the present application, the data storage device provided in the embodiments of the present application can be implemented in hardware. As an example, the data storage device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data storage method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0091] Below, the data storage method provided in the embodiment of the present application will be described in combination with the exemplary application and implementation of the data storage device provided in the embodiment of the present application.

[0092] See also Figure 4 , Figure 4 This is an optional flow chart of the data storage method provided in the embodiment of the present application, which will be combined with Figure 4 The data storage method is applied to a data storage device, the data storage device is used to perform a first task, and a local disk of the data storage device is used to store first data to be stored, the first data to be stored being data generated when performing the first task.

[0093] S401. Run the mount client.

[0094] In an embodiment of the present application, a mount client is installed on the data storage device, and the mount client is used to store data in the distributed storage device; thus, the data storage device runs the installed mount client to execute the data storage method through the running mount client.

[0095] It should be noted that the distributed storage device is used to distribute data stored in the data storage device and is a device independent of the data storage device. Furthermore, the distributed storage device can be a single device or a device cluster, which is not limited in the embodiments of this application. The mounting client is a bridge between the data storage device and the distributed storage device, and is used to store data from the data storage device on the distributed storage device in a certain storage format. For example, when the distributed storage device is a "Ceph" distributed storage system, the mounting client is the RBD (RADOS Block Device) in the "Ceph" distributed storage system.

[0096] S402: Acquire second data to be stored generated when executing the second task.

[0097] In an embodiment of the present application, the data storage device is also used to perform a second task to achieve mixed deployment; the data storage device will generate data that needs to be stored when performing the second task, and obtain the data that needs to be stored generated when performing the second task (for example, temporary data calculated locally by the data storage device), thereby obtaining the second data to be stored.

[0098] It should be noted that the priority of the second task is lower than that of the first task, where priority refers to the priority of task execution; the data storage device implements mixed deployment by executing the first task and the second task. The first task can be a containerized task or a non-containerized task, which is not limited in the embodiments of the present application; the second task can be a containerized task or a non-containerized task, which is not limited in the embodiments of the present application; and the first task can be an offline task or an online task. When the first task is an offline task, the second task is an offline task with a lower priority than the first task; when the first task is an online task, the second task is an offline task or an online task with a lower priority than the first task.

[0099] S403: Send a data storage request carrying the second data to be stored to the distributed storage device through the running mounting client, so that the distributed storage device responds to the data storage request and stores the second data to be stored in the distributed storage device.

[0100] In an embodiment of the present application, the data storage device stores the obtained second data to be stored through a running mount client. Here, the data storage device generates a data storage request carrying the second data to be stored through the running mount client, and sends the data storage request to the distributed storage device through the running mount client. At this time, the distributed storage device also receives the data storage request. Since the data storage request is used to request that the second data to be stored be stored in the distributed storage device, the distributed storage device responds to the data storage request and stores the second data to be stored locally on the distributed storage device, thereby completing the storage of the second data to be stored corresponding to the second task.

[0101] It should be noted that the running mounting client is the mounted virtual disk, and the data storage device stores the second data to be stored in the running mounting client by storing data in the local disk, so as to store the second data to be stored in the distributed storage device through the running mounting client.

[0102] It should also be noted that the first data to be stored generated when executing the first task can be stored in the local disk of the data storage device, and can also be stored on other devices. The embodiment of the present application does not limit this. In addition, when the first data to be stored generated when executing the first task is stored on other devices, the other device may be a device different from the distributed storage device; and when the storage space of the distributed storage device is large enough, the other device may also be a distributed storage device.

[0103] For example, see Figure 5 , Figure 5 is a schematic diagram of an exemplary data storage method provided in an embodiment of the present application; Figure 5 As shown, a mounting client 5-11 is installed in the data storage device 5-1. When the data storage device 5-1 executes the first task 5-21, the first data to be stored is stored in the local disk 5-12 of the data storage device 5-1; when the data storage device 5-1 executes the second task 5-22, the second data to be stored is stored in the distributed storage device 5-3 through the mounting client 5-11.

[0104] It is understandable that on the data storage device that implements mixed deployment, the mounting of the virtual disk is achieved by mounting the client, so that the second data to be stored generated when executing the second task with a lower priority can be stored in the distributed storage device, and thus isolated from the storage of the first data to be stored generated when executing the first task on the data storage device on the local disk; that is, the storage of the first data to be stored of the first task and the storage of the second data to be stored of the second task are independent and do not affect each other, thus improving the implementation effect of mixed deployment. In addition, by mounting the client, the storage requirements of the second data to be stored of the second task can be met, thereby improving the applicability of the second task and the resource utilization of the data storage device.

[0105] See also Figure 6 , Figure 6 This is another optional flow chart of the data storage method provided in the embodiment of the present application; Figure 6 As shown, in an embodiment of the present application, S403 also includes S404 to S406; that is, after the running mounting client sends a data storage request carrying the second data to be stored to the distributed storage device, the data storage method also includes S404 to S406, and each step is explained below.

[0106] S404: Detect the storage status corresponding to the data storage request.

[0107] It should be noted that after the data storage device sends a data storage request to the distributed storage device through the running mount client, the data storage device monitors the data storage request to detect the corresponding status of the data storage request, thereby obtaining the storage status. In other words, the storage status is the processing status of the data storage request, for example, it can be a process status or a thread status.

[0108] S405: Count the number of states in which the storage state is the uninterruptible sleep state.

[0109] In an embodiment of the present application, the data storage device judges the storage status. If it is determined that the storage status is an uninterruptible sleep state, it indicates that the distributed storage device has not completed the response to the data storage request. At this time, one is added to the number of states to count the number of states in which the storage status is an uninterruptible sleep state.

[0110] It should be noted that the uninterruptible sleep state refers to a state in which the data storage device does not receive a response from the distributed storage device after sending a data storage request to the distributed storage device, for example, the "UNINTE RRUPTIBLE" state.

[0111] Here, the number of states can also be obtained by obtaining resource limit data, which includes the number of various storage states.

[0112] S406: When the state quantity is greater than the quantity threshold, stop scheduling the next second task.

[0113] It should be noted that the number of states is negatively correlated with the stability of the distributed storage device: the larger the number of states, the lower the stability of the distributed storage device; the smaller the number of states, the higher the stability of the distributed storage device. Here, a preset threshold is used to measure the stability of the distributed storage device. When the number of states is greater than the threshold, it indicates that the distributed storage device is unstable and cannot meet the data storage needs of the data storage device. In this case, the data storage device stops scheduling the next second task. When the number of states is less than or equal to the threshold, it indicates that the distributed storage device is stable and can meet the data storage needs of the data storage device. In this case, the data storage device schedules the next second task.

[0114] It can be understood that the data storage device realizes the mounting of the virtual disk by running the mounting client; since the storage of the second data to be stored of the second task is realized by mounting the virtual disk, the stability of the distributed storage device is an important factor affecting the storage of the second data to be stored. Therefore, by detecting the storage status and counting the number of states in which the storage status is an uninterruptible sleep state, the scheduling of the second task is stopped when the number of states is greater than the number threshold, the normal operation of the data storage device can be ensured.

[0115] In the embodiment of the present application, S405 also includes S407; that is, after the data storage device counts the number of states in which the storage state is an uninterruptible sleep state, the data storage method further includes S407, and this step is described below.

[0116] S407: When the number of states is greater than the number threshold, reduce the disk read and write rate of the second task.

[0117] In an embodiment of the present application, when the data storage device determines that the number of states is greater than a quantity threshold, it can continue to execute the second task currently being executed, but the disk read and write rate of the second task is reduced to reduce the number of states, thereby reducing the load on the distributed storage device and ensuring the normal operation of the data storage device.

[0118] In the embodiment of the present application, S405 also includes S408 and S409; that is, after the data storage device counts the number of states in which the storage state is an uninterruptible sleep state, the data storage method also includes S408 and S409, which are described below.

[0119] S408: When the state quantity is greater than the quantity threshold, generate an alarm prompt message.

[0120] In an embodiment of the present application, the data storage device may also generate an alarm prompt message when determining that the number of states is greater than a quantity threshold; wherein the alarm prompt message is information used to prompt that the distributed storage device is operating abnormally.

[0121] S409: Display warning information.

[0122] It should be noted that the data storage device displays an alarm prompt message to prompt that the distributed storage device is working abnormally.

[0123] In the embodiment of the present application, the data storage device may execute all of S408 and S409, S406, and S407, or only one of them, or two of them, and the embodiment of the present application does not limit this.

[0124] In an embodiment of the present application, S406 also includes S410 and S411; that is, when the number of states is greater than the number threshold, the data storage device stops scheduling the next second task, and the data storage method also includes S410 and S411. Each step is explained below.

[0125] S410: Acquire update information of the state quantity, and determine the updated state quantity based on the update information of the state quantity.

[0126] It should be noted that if the storage state changes from an uninterruptible sleep state to a state in which processing can continue (for example, a running state, a sleep state, or a stopped state, etc.), or the number of uninterruptible sleep states increases, it indicates that the state quantity has been updated; thus, the data storage state obtains the updated information of the state quantity, that is, the increase or decrease information of the state quantity, and determines the current state quantity based on the updated information of the state quantity, and the current state quantity is the updated state quantity.

[0127] S411: When the number of updated states is less than or equal to the number threshold, schedule the next second task.

[0128] In an embodiment of the present application, when the update information of the state quantity is the subtraction information of the state quantity, if the updated state quantity is less than or equal to the quantity threshold, the scheduling of the next second task can continue.

[0129] In the embodiment of the present application, S410 also includes S412; that is, after the data storage device determines the updated state quantity based on the updated information of the state quantity, the data storage method also includes S412, which is described below.

[0130] S412: When the number of update states is less than or equal to the number threshold, restore the disk read / write rate of the second task.

[0131] It should be noted that if the number of states is greater than the quantity threshold, the data storage device reduces the disk read and write rate of the second task, that is, when the data storage device executes S407, the disk read and write rate of the second task is restored when the number of updated states is less than or equal to the quantity threshold.

[0132] In the embodiment of the present application, S410 also includes S413; that is, after the data storage device determines the updated state quantity based on the updated information of the state quantity, the data storage method also includes S413 and S414, which are described below.

[0133] S413: When the number of update states is less than or equal to the number threshold, generate a recovery prompt message.

[0134] S414: Display recovery prompt information.

[0135] It should be noted that the recovery prompt information is information used to prompt that the operation of the distributed storage device has returned to normal; if the status quantity is greater than the quantity threshold, the data storage device displays an alarm prompt information, that is, when the data storage device executes S408 and S409, then when the update status quantity is less than or equal to the quantity threshold, the recovery prompt information is displayed; by displaying the recovery prompt information, it is prompted that the operation of the distributed storage device has returned to normal.

[0136] It is understandable that by running the mounting client to mount a distributed storage virtual disk, such as the "RBD" disk of "CEPH", it is possible to expand the storage space of the local disk, make full use of idle CPU or memory resources, and improve the resource utilization of colocation; at the same time, it can also improve the versatility of the second task.

[0137] See also Figure 7 , Figure 7 This is another optional flow chart of the data storage method provided in the embodiment of the present application; Figure 7 As shown, in the embodiment of the present application, S415 to S418 are also included before S402; that is, before the data storage device obtains the second data to be stored generated when executing the second task, the data storage method also includes S415 to S418, and each step is explained below.

[0138] S415: Obtain the disk space usage corresponding to the first task.

[0139] In an embodiment of the present application, the data storage device can achieve independent storage of the first task and the second task by mounting a virtual disk, thereby resolving the storage space competition problem during the colocation process. The data storage device can also resolve the storage space competition problem during the colocation process by limiting the storage space of the second task on the local disk. Here, when the data storage device limits the storage space of the second task on the local disk, it first obtains the disk space usage corresponding to the first task, that is, the usage of the first data to be stored when stored on the local disk.

[0140] S416: Determine the amount of free disk space based on the disk space usage and the total disk space.

[0141] It should be noted that the data storage device may determine the amount of free disk space as the difference between the total disk space and the occupied disk space; or it may determine the amount of free disk space as the amount of reserved disk space removed from the difference; this embodiment of the application is not limited thereto. The total disk space refers to the amount of space on the local disk.

[0142] S417: When scheduling the second task, create a local storage directory associated with the second task.

[0143] In an embodiment of the present application, when scheduling the second task, the data storage device creates a storage directory for storing data corresponding to the second task, thereby completing the creation of a local storage directory associated with the second task.

[0144] S418: Generate space limit information for the local storage directory based on the free disk space.

[0145] It should be noted that the data storage device uses the amount of free disk space as the maximum quota of the disk space of the local storage directory to limit the amount of data stored in the local storage directory; wherein the maximum quota of the disk space of the local storage directory is the space limit information.

[0146] Continue to see Figure 7 In an embodiment of the present application, S402 also includes S419; that is, after the data storage device obtains the second data to be stored generated when executing the second task, the data storage method also includes S419, and this step is explained below.

[0147] S419: Based on the space limitation information, store the second data to be stored in the local storage directory.

[0148] It should be noted that, after the data storage device obtains the second data to be stored generated when executing the second task, when it stores the second data to be stored in the local storage directory, if the data storage device determines that after the second data to be stored is stored in the local storage directory, the total disk occupancy of the data in the local storage directory is less than or equal to the disk free space amount in the space limit information, the second data to be stored will be stored in the local storage directory; if the data storage device determines that after the second data to be stored is stored in the local storage directory, the total disk occupancy of the data in the local storage directory is greater than the disk free space amount in the space limit information, the second data to be stored will no longer be stored in the local storage directory.

[0149] In an embodiment of the present application, S416 also includes S420 and S421; that is, after the data storage device determines the amount of free disk space based on the disk space occupancy and the total disk space, the data storage method also includes S420 and S421. Each step is explained below.

[0150] S420: Obtain the next disk space occupancy corresponding to the first task.

[0151] It should be noted that the data storage device dynamically adjusts the maximum quota of the space limit information based on the real-time usage of the first to-be-stored data corresponding to the first task on the local disk; thus, the data storage device periodically or in an event-triggered manner obtains the disk space usage corresponding to the first task. Here, the next disk space usage is the latest disk space usage obtained after the disk space usage is obtained.

[0152] S421. Update the free disk space based on the next disk space usage and the total disk space.

[0153] It should be noted that the data storage device updates the disk free space amount in combination with the next disk space occupancy amount and the total disk space amount, so that the updated disk free space amount is obtained in real time.

[0154] Accordingly, in an embodiment of the present application, in S418, the data storage device generates space limit information for the local storage directory based on the amount of free disk space, including: the data storage device generates space limit information for the local storage directory based on the updated amount of free disk space, so that the maximum quota in the space limit information is dynamically related to the amount of space occupied by the data of the first task on the local disk.

[0155] It is understandable that the data storage device can solve the competition problem between the first task and the second task in data storage by dynamically obtaining the occupancy of the first task's data on the local disk and dynamically controlling the storage of the second task's data on the current disk.

[0156] In an embodiment of the present application, S417 can be implemented through S4171 to S4173; that is, when the data storage device schedules the second task, it creates a local storage directory associated with the second task, including S4171 to S4173. Each step is described below.

[0157] S4171. When scheduling the second task, construct a container corresponding to the second task.

[0158] It should be noted that the second task is executed in the form of a container; therefore, when scheduling the second task, the data storage device constructs a container corresponding to the second task to execute the second task.

[0159] S4172. Create at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory.

[0160] It should be noted that the storage directory created by the data storage device in the local disk includes at least one data storage directory, and each data storage directory in the at least one data storage directory corresponds to at least one sub-data storage directory.

[0161] S4173. Mount at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory to the container corresponding to the second task to obtain a local storage directory.

[0162] In an embodiment of the present application, the data storage device mounts at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory to the container corresponding to the second task. In this way, the at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory are associated with the second task, and the data storage device completes the creation of a local storage directory associated with the second task. In other words, the at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory mounted to the container corresponding to the second task are local storage directories.

[0163] In an embodiment of the present application, S418 can be implemented through S4181 and S4182; that is, the data storage device generates space limit information of the local storage directory based on the amount of free space on the disk, including S4181 and S4182. Each step is explained below.

[0164] S4181. Generate space limit information based on the amount of free space on the disk.

[0165] It should be noted that the data storage device uses the amount of free disk space as the maximum quota to generate space limitation information.

[0166] S4182: Establish an association with space limitation information for at least one data storage directory in the local storage directory and at least one sub-data storage directory corresponding to each data storage directory.

[0167] It should be noted that the data storage device associates at least one data storage directory in the local storage directory, and at least one sub-data storage directory corresponding to each data storage directory, with space limit information to enable sharing of the space limit information. In other words, the space limit information is used to limit the total storage capacity of the at least one data storage directory and the at least one sub-data storage directory corresponding to each data storage directory.

[0168] It is understandable that at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory share a space limit information, which can effectively and accurately limit the storage of data of the second task.

[0169] In an embodiment of the present application, S419 can be implemented through S4191 to S4194; that is, the data storage device stores the second data to be stored in the local storage directory based on the space limitation information, including S4191 to S4194, and each step is explained below.

[0170] S4191. Obtain the amount of stored space in the local storage directory.

[0171] It should be noted that the data storage device counts the amount of data stored in all storage directories in the local storage directory, and thus obtains the amount of stored space. It is easy to know that the amount of stored space is the storage amount of data corresponding to all second tasks in the local storage directory.

[0172] S4192. Determine the amount of available storage space based on the amount of free disk space and the amount of stored space corresponding to the space limit information.

[0173] In an embodiment of the present application, the difference between the amount of free disk space and the amount of stored space obtained by the data storage device is the amount of storable space; that is, the amount of storable space refers to the amount of free space in the local storage directory used to store data for the second task.

[0174] S4193. When the amount of storage space corresponding to the second data to be stored is less than or equal to the amount of available storage space, the second data to be stored is stored in the local storage directory.

[0175] It should be noted that when the storage space corresponding to the second data to be stored is less than or equal to the storable space, it indicates that the local storage directory can store the second data to be stored, and thus the data storage device stores the second data to be stored in the local storage directory.

[0176] S4194: When the amount of storage space corresponding to the second data to be stored is greater than the amount of available storage space, the data storage process of the second data to be stored in the local storage directory is terminated.

[0177] It should be noted that when the amount of storage space corresponding to the second data to be stored is greater than the amount of storable space, it indicates that the storage space of the local storage directory is insufficient to meet the storage needs of the second data to be stored. As a result, the data storage device ends the data storage process of the second data to be stored in the local storage directory; at this time, a prompt message indicating that the space is full can also be generated, and the prompt message indicating that the space is full can be displayed to indicate that the disk space of the local storage directory is full.

[0178] It can be understood that in the process of implementing mixed deployment, the disk space usage of the second task is managed to prevent the second task from using too much disk space and affecting the normal operation of the first task; the second task shares the disk space resources of the local disk partition, but the amount of space used by the second task is determined based on the space usage of the first task to ensure the normal operation of the first task.

[0179] See also Figure 8 , Figure 8 This is another optional flow chart of the data storage method provided in the embodiment of the present application; Figure 8 As shown, in the embodiment of the present application, S422 is also included before S402; that is, before the data storage device obtains the second data to be stored generated when executing the second task, the data storage method also includes S422. These steps are explained below.

[0180] S422: Establish a storage association relationship with a remote storage device.

[0181] It should be noted that the data storage device establishes a storage association relationship with the remote storage device to store the second to-be-stored data corresponding to the second task through the storage association relationship. The remote storage device may be a file storage device, an object storage device, or a storage device with separate computing and storage, etc., which is not limited in the present embodiment.

[0182] Continue to see Figure 8, S402 also includes S423; that is, after the data storage device obtains the second data to be stored generated when executing the second task, the data storage method also includes S423, and this step is explained below.

[0183] S423: Based on the storage association relationship, send a remote storage request carrying the second data to be stored to the remote storage device, so that the remote storage device responds to the remote storage request and stores the second data to be stored in the remote storage device.

[0184] It should be noted that, based on the storage association relationship, the data storage device sends a remote storage request carrying the second data to be stored to the remote storage device; at this time, the remote storage device also receives the remote storage request, and since the remote storage request is a request to store the second data to be stored on the remote storage device, the remote storage device responds to the remote storage request and stores the second data to be stored in the remote storage device.

[0185] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0186] See also Figure 9 , Figure 9 This is a schematic diagram of an exemplary data storage method provided in an embodiment of the present application; Figure 9As shown, in the colocation host 9-1 (data storage device), a new agent component 9-11 is added to dynamically obtain the disk space usage 9-14 (disk space occupancy) of the online task 9-13 (first task) in the partition of the local disk 9-12; and then update the disk available space 9-16 (disk free space) of the offline task 9-15 (second task); and, based on the disk available space 9-16, manage the disk space usage of the offline task 9-15, that is, for the offline directory 9-17 (local storage directory, exemplarily showing the "HostPath" storage directory 9-171, the "HostPath" storage directory 9-172, the "EmptyDir" storage directory 9-173 and the "EmptyDir" storage directory 9-174) corresponding to the offline task 9-15, use the project quota management (project quota management) of the disk quota management 9-18 (DiskQuota) The quota function limits the disk space usage of offline directory 9-17 to the available disk space 9-16 (thus obtaining the aforementioned space limit information). A project ID is also generated for offline directory 9-17. All storage directories (at least one data storage directory) created under offline directory 9-17, as well as each storage directory's sub-storage directories (at least one sub-data storage directory), inherit the same project ID and are collectively subject to the available disk space 9-16 limit, thus achieving restricted sharing. Thus, the available disk space for offline task 9-15's offline directory 9-17 is determined by the disk space usage 9-14 of online task 9-13. Offline directory 9-17 is created by creating a storage directory in a partition on local disk 9-12 and mounting it to the container containing offline task 9-15. Within the container, offline task 9-15 can access and write to offline directory 9-17 just as it would a storage directory on a local disk.

[0187] It should be noted that since the colocation host is based on Kubernetes, DiskQuota is applied to the EmptyDir and HostPath storage directories of Kubernetes, which basically covers the mainstream storage directory mounting methods of Kubernetes.

[0188] As you can see, by obtaining disk space usage information for online tasks, we can adjust the available disk space for offline tasks and use the "DiskQuota" feature to limit offline task disk space usage, resolving data storage competition between offline and online tasks in hybrid deployment. Furthermore, each storage directory in the offline directory is not subject to individual limits; instead, they share the same limit, preventing offline tasks from using excessive disk resources. Furthermore, limiting offline tasks through "DiskQuota" is completely transparent to the business, requiring no changes to the business.

[0189] See also Figure 10 , Figure 10 is a schematic diagram of the architecture of another exemplary data storage method provided in an embodiment of the present application; Figure 10 As shown, the co-located host machine 10-1 (data storage device) is used to execute online task 10-11 (first task) and offline task 10-12 (second task, exemplarily showing offline task 10-121, offline task 10-122, and offline task 10-123). Among them, the data 10-31 (first data to be stored) generated when the online task 10-11 is executed is stored in the local disk 10-4; the data 10-32 (second data to be stored) generated when the offline task 10-12 is executed is stored in the distributed storage virtual disk 10-5 (mounted on the client, such as the block storage implementation form "RBD" in the "Ceph" cluster), and the distributed storage virtual disk 10-5 then stores the data 10-32 in the back-end distributed storage service device 10-6 (distributed storage device, such as the "Ceph" cluster).

[0190] It is understandable that, since offline tasks generally require a large amount of disk space during operation to store temporary data in the calculation, if the free disk space remaining in the colocation host is small and cannot meet the storage needs of the offline tasks, only a small number of offline tasks can be colocation; thus, the idle CPU and memory resources are not fully utilized, and the improvement in the colocation effect is limited. In order to solve the bottleneck problem of disk space, the embodiment of the present application adopts the method of mounting a distributed storage virtual disk, and dynamically mounts a virtual disk of any size or adapted to the data storage capacity of the offline task on the colocation host to store the data generated by the offline task during execution, which is completely imperceptible to the offline task. Continue to see Figure 9 The data storage of offline tasks and online tasks is completely isolated, and there is no need to configure space limit information; moreover, it can solve the problem of insufficient disk space and can be expanded arbitrarily according to the storage requirements of offline tasks.

[0191] It should be noted that when using the distributed storage virtual disk mounting method to dynamically mount virtual disks on the colocation host, the problem of unstable storage backend distributed storage service equipment often occurs. If the backend distributed storage service equipment is unstable, the colocation host will be stuck because the input / output data sent by the offline process of the offline task does not receive a response message. As offline tasks are continuously scheduled, the number of offline task processes accumulates, which will eventually affect the stability of the colocation host. For the problem of unstable storage backend distributed storage service equipment, see Figure 11 , Figure 11 This is a flow chart of an exemplary method of adjusting the execution of the second task based on the storage state provided in an embodiment of the present application; Figure 11 As shown, the exemplary process of adjusting the execution of the second task based on the storage state includes:

[0192] S1101. Obtain the control group of the offline process.

[0193] It should be noted that the offline task is in the "cgroup", and the control group of the offline process refers to the "cgroup" of the offline process.

[0194] S1102: Obtain kernel control group status data.

[0195] It should be noted that the colocation host obtains the kernel's control group status data (cgroupstats data) based on the obtained cgroup of the offline process. Here, the colocation host obtains the kernel's cgroupstats data through the Netlink mechanism.

[0196] S1103. Obtain the number of D-state processes (number of states).

[0197] It should be noted that the colocation host obtains the number of processes in D state (UNINTERRUPTIBLE, uninterruptible sleep state) from the control group status data.

[0198] It should also be noted that if the input / output data sent by the offline process does not receive a response message, the offline process will be in the D state. The colocation host obtains the number of processes in the D state from the kernel's control group status data ("cgroupstats" data). The "cgroupstats" data can be described by the following code:

[0199]

[0200] Among them, "__u64 nr_sleeping" refers to the number of offline processes in the sleeping state; "__u64 nr_running" refers to the number of offline processes in the running state; "__u64 nr_stopped" refers to the number of offline processes in the stopped state; "__u64 nr_uninterruptible" refers to the number of offline processes in the D state; "__u64 nr_io_wait" refers to the number of offline processes in the waiting state.

[0201] S1104: Determine whether the number of processes in D state is greater than a threshold (number threshold). If yes, execute S1105; if not, execute S1108.

[0202] Here, if the number of D-state processes is greater than the threshold, it indicates that the back-end distributed storage service device is unstable; if the number of D-state processes is less than or equal to the threshold, it indicates that the back-end distributed storage service device is stable. Even if a small number of D-state processes are currently accumulated, it does not affect the normal operation of online tasks.

[0203] S1105: Stop scheduling offline tasks.

[0204] It should be noted that when the backend distributed storage service equipment is unstable, the colocation host adopts a strategy of prohibiting the scheduling of offline tasks, freezing offline task scheduling, avoiding receiving more offline tasks, and putting greater pressure on the backend distributed storage service equipment.

[0205] S1106. Mount the virtual disk at a limited speed.

[0206] It should be noted that when the backend distributed storage service device is unstable, input / output read / write operations will continue to be generated for offline tasks that are still running. The colocation host also uses a method of limiting the speed of the mounted disk to slow down the generation of input / output data and alleviate the pressure on the backend distributed storage service device.

[0207] Here, the colocation host uses Blkio cgroup to limit the speed of mounted virtual disks. You can configure the speed for the cgroup root directory (for example, when limiting the speed for cgroup version 1) or for each cgroup directory containing offline tasks (for example, when limiting the speed for cgroup version 2). By limiting the configured speed, the read and write rates of input and output data generated by currently running offline tasks are reduced, thereby reducing the pressure on the backend distributed storage service equipment.

[0208] S1107: Send warning information (warning prompt information). Continue to execute S1102.

[0209] It should be noted that the colocation host sends an alarm message to notify people to intervene and resolve the problem.

[0210] S1108. Resume scheduling of offline tasks.

[0211] It should be noted that after the backend distributed storage service device is restored, the D state process will be automatically restored; after the D state process is released, the scheduling of offline tasks will be restarted.

[0212] S1109: Restore the speed of the mounted virtual disk. Continue with S1102.

[0213] It should be noted that after the D state process is released, the rate limit of the read and write operations of the input / output data is released, and the read and write operations of the input / output data are restored.

[0214] In the embodiment of the present application, with the subsequent development of storage and computing separation technology (for example, Remote Shuffle Service), the data storage corresponding to the offline task can be directly accessed remotely to achieve storage.

[0215] It's understandable that on Kubernetes-based colocation hosts, the DiskQuota feature limits disk space usage for offline tasks to avoid competition with online tasks for storage resources. Alternatively, mounting distributed storage virtual disks can provide sufficient storage space for offline tasks, fully utilizing idle CPU and memory resources. Furthermore, mounting distributed storage virtual disks expands the scope of offline tasks and broadens the scope of colocation scenarios, eliminating the need for limited offline tasks requiring minimal disk space.

[0216] The following continues to describe the exemplary structure of the data storage device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 3 As shown, the software modules stored in the data storage device 255 of the memory 250 may include:

[0217] The task execution module 2551 is configured to execute a first task and store first data to be stored generated when executing the first task to a local disk of the data storage device;

[0218] The client running module 2552 is used to run the mounting client, wherein the mounting client is used to store data in the distributed storage device;

[0219] a data acquisition module 2553, configured to acquire second data to be stored generated when executing a second task, wherein the priority of the second task is lower than the priority of the first task;

[0220] The data storage module 2554 is used to send a data storage request carrying the second data to be stored to the distributed storage device through the running mounting client, so that the distributed storage device responds to the data storage request and stores the second data to be stored in the distributed storage device.

[0221] In an embodiment of the present application, the data storage device 255 also includes a status detection module 2555, which is used to detect the storage status corresponding to the data storage request; count the number of states in which the storage status is an uninterruptible sleep state; when the number of states is greater than a quantity threshold, stop scheduling the next second task.

[0222] In the embodiment of the present application, the state detection module 2555 is further configured to reduce the disk read / write rate of the second task when the state quantity is greater than the quantity threshold.

[0223] In an embodiment of the present application, the status detection module 2555 is also used to generate an alarm prompt information when the number of the status is greater than the number threshold, wherein the alarm prompt information is information used to prompt the abnormal operation of the distributed storage device; and display the alarm prompt information.

[0224] In an embodiment of the present application, the state detection module 2555 is also used to obtain the update information of the state quantity and determine the updated state quantity based on the updated information of the state quantity; when the updated state quantity is less than or equal to the quantity threshold, the next second task is scheduled.

[0225] In the embodiment of the present application, the status detection module 2555 is further configured to restore the disk read / write rate of the second task when the number of update states is less than or equal to the number threshold.

[0226] In an embodiment of the present application, the status detection module 2555 is also used to generate recovery prompt information when the number of update states is less than or equal to the number threshold, wherein the recovery prompt information is information used to prompt that the operation of the distributed storage device has returned to normal; and display the recovery prompt information.

[0227] In an embodiment of the present application, the data storage device 255 also includes a space limitation module 2556, which is used to obtain the disk space occupancy corresponding to the first task; determine the disk free space amount in combination with the disk space occupancy and the total disk space amount, wherein the total disk space amount is the space amount of the local disk; when scheduling the second task, create a local storage directory associated with the second task; based on the disk free space amount, generate space limit information for the local storage directory.

[0228] In the embodiment of the present application, the space limitation module 2556 is further configured to store the second data to be stored in the local storage directory based on the space limitation information.

[0229] In an embodiment of the present application, the space limitation module 2556 is further configured to obtain the next disk space occupancy corresponding to the first task; and update the disk free space amount in combination with the next disk space occupancy and the total disk space amount.

[0230] In the embodiment of the present application, the space limitation module 2556 is further configured to generate the space limitation information of the local storage directory based on the updated free space amount of the disk.

[0231] In an embodiment of the present application, the space limitation module 2556 is also used to build a container corresponding to the second task when scheduling the second task; create at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory; mount at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory to the container corresponding to the second task to obtain the local storage directory.

[0232] In an embodiment of the present application, the space limitation module 2556 is also used to generate the space limitation information based on the amount of free space on the disk; and establish an association with the space limitation information for at least one data storage directory in the local storage directory and at least one sub-data storage directory corresponding to each data storage directory.

[0233] In an embodiment of the present application, the space limitation module 2556 is also used to obtain the amount of stored space in the local storage directory; determine the amount of storable space based on the amount of free disk space and the amount of stored space corresponding to the space limitation information; when the amount of storage space corresponding to the second data to be stored is less than or equal to the amount of storable space, store the second data to be stored in the local storage directory; when the amount of storage space corresponding to the second data to be stored is greater than the amount of storable space, end the data storage process of the second data to be stored in the local storage directory.

[0234] In the embodiment of the present application, the data storage device 255 further includes a remote storage module 2557 for establishing a storage association relationship with a remote storage device.

[0235] In an embodiment of the present application, the remote storage module 2557 is also used to send a remote storage request carrying the second data to be stored to the remote storage device based on the storage association relationship, so that the remote storage device responds to the remote storage request and stores the second data to be stored in the remote storage device.

[0236] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data storage method described above in the present invention.

[0237] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the data storage method provided by the embodiment of the present application, for example, Figure 4 The data storage method is shown.

[0238] In some embodiments of the present application, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0239] In some embodiments of the present application, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0240] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (for example, files storing one or more modules, subroutines, or code portions).

[0241] As an example, executable instructions may be deployed to be executed on one computer device (in which case, the one computer device is a data storage device), or on multiple computer devices located at one location (in which case, the multiple computer devices located at one location are data storage devices), or on multiple computer devices distributed at multiple locations and interconnected by a communication network (in which case, the multiple computer devices distributed at multiple locations and interconnected by a communication network are data storage devices).

[0242] In summary, the embodiment of the present application implements the mounting of a virtual disk by mounting a client on a data storage device that implements mixed deployment, so that the second data to be stored generated when executing a second task with a lower priority can be stored in a distributed storage device, and thus isolated from the storage of the first data to be stored generated when executing a first task on the data storage device on the local disk; that is, the storage of the first data to be stored for the first task and the storage of the second data to be stored for the second task are independent and do not affect each other, thus improving the implementation effect of mixed deployment. In addition, by mounting the client, the storage requirements of the second data to be stored for the second task can be met, thereby improving the applicability of the second task and the resource utilization of the data storage device.

[0243] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data storage method, applied to a data storage device, characterized in that: The data storage device is used to perform a first task, and the local disk of the data storage device is used to store first data to be stored, where the first data to be stored is data generated when performing the first task, including: Running a mount client, wherein the mount client is used to store data in a distributed storage device; Acquire second data to be stored generated when executing a second task, wherein the priority of the second task is lower than the priority of the first task; By running the mount client, a data storage request carrying the second data to be stored is sent to the distributed storage device, so that The distributed storage device stores the second data to be stored in the distributed storage device in response to the data storage request.

2. The method according to claim 1, characterized in that After the running mounting client sends a data storage request carrying the second data to be stored to the distributed storage device, the method further includes: detecting a storage status corresponding to the data storage request; Counting the number of states where the storage state is an uninterruptible sleep state; When the state quantity is greater than the quantity threshold, the scheduling of the next second task is stopped.

3. The method according to claim 2, characterized in that After counting the number of states in which the storage state is an uninterruptible sleep state, the method further includes: When the number of states is greater than the number threshold, the disk read and write rate of the second task is reduced.

4. The method according to claim 2 or 3, characterized in that After counting the number of states in which the storage state is an uninterruptible sleep state, the method further includes: When the number of states is greater than the number threshold, generating an alarm prompt information, wherein the alarm prompt information is information for prompting an abnormal operation of the distributed storage device; The warning prompt information is displayed.

5. The method according to claim 2 or 3, characterized in that After stopping scheduling the next second task when the number of states is greater than the number threshold, the method further includes: Acquire update information of the state quantity, and determine an updated state quantity based on the update information of the state quantity; When the number of updated states is less than or equal to the number threshold, the next second task is scheduled.

6. The method according to claim 5, characterized in that After determining the updated state quantity based on the update information of the state quantity, the method further includes: When the number of update states is less than or equal to the number threshold, the disk read and write rate of the second task is restored.

7. The method according to claim 5 or 6, characterized in that After determining the updated state quantity based on the update information of the state quantity, the method further includes: When the number of update states is less than or equal to the number threshold, generating recovery prompt information, wherein the recovery prompt information is information used to prompt that the distributed storage device is restored to normal operation; The recovery prompt information is displayed.

8. The method according to claim 1, characterized in that Before acquiring the second data to be stored generated when executing the second task, the method further includes: Obtaining the disk space usage corresponding to the first task; Determine the amount of free disk space by combining the disk space usage and the total disk space, wherein the total disk space is the space amount of the local disk; When scheduling the second task, creating a local storage directory associated with the second task; Based on the free space of the disk, generating space limit information of the local storage directory; After obtaining the second data to be stored generated when executing the second task, the method further includes: Based on the space limitation information, the second data to be stored is stored in the local storage directory.

9. The method according to claim 8, characterized in that After determining the amount of free disk space by combining the disk space usage and the total disk space, the method further includes: Obtaining the next disk space usage corresponding to the first task; In combination with the next disk space occupancy amount and the total disk space amount, updating the disk free space amount; The generating of the space limit information of the local storage directory based on the free space of the disk includes: Based on the updated free disk space, the space limit information of the local storage directory is generated.

10. The method according to claim 8 or 9, characterized in that The step of creating a local storage directory associated with the second task when scheduling the second task includes: When scheduling the second task, building a container corresponding to the second task; Creating at least one data storage directory and at least one sub-data storage directory corresponding to each data storage directory; Mount at least one of the data storage directories and at least one sub-data storage directory corresponding to each of the data storage directories to the container corresponding to the second task to obtain the local storage directory.

11. The method according to claim 8 or 9, characterized in that The generating of the space limit information of the local storage directory based on the free space of the disk includes: Based on the free space amount of the disk, generating the space amount limit information; For at least one data storage directory in the local storage directory and at least one sub-data storage directory corresponding to each data storage directory, an association with the space limitation information is established.

12. The method according to claim 8 or 9, characterized in that The storing the second to-be-stored data in the local storage directory based on the space limitation information includes: Obtaining the amount of stored space in the local storage directory; Determining the amount of available storage space based on the amount of free disk space and the amount of stored space corresponding to the space limit information; When the amount of storage space corresponding to the second data to be stored is less than or equal to the amount of storable space, storing the second data to be stored in the local storage directory; When the amount of storage space corresponding to the second data to be stored is greater than the amount of storable space, the data storage process of the second data to be stored in the local storage directory is terminated.

13. The method according to claim 1, wherein Before acquiring the data to be stored generated by executing the second task, the method further includes: Establishing a storage association relationship with a remote storage device; After obtaining the second data to be stored generated when executing the second task, the method further includes: Based on the storage association relationship, a remote storage request carrying the second data to be stored is sent to the remote storage device, so that The remote storage device stores the second data to be stored in the remote storage device in response to the remote storage request.

14. A data storage device, characterized in that include: a memory for storing executable instructions; A processor, configured to implement the data storage method according to any one of claims 1 to 13 when executing executable instructions stored in the memory.

15. A computer-readable storage medium, characterized in that Executable instructions are stored, which are used to implement the data storage method according to any one of claims 1 to 13 when executed by a processor.

Images