Data filling method, device, apparatus, medium and product

Abstract

The present disclosure relates to the technical field of computers, and particularly provides a data filling method, device, equipment, medium and product. In the present disclosure, a to-be-processed task is written into a task bucket of an object storage cluster; a target task is read based on time attributes of each to-be-processed task in the task bucket; when the target task is a water level filling task, a target object is constructed and written into an OSD; when the water level of the OSD meets a preset first threshold, water level filling is completed. By setting a data filling device in the object storage cluster, the data filling device can directly communicate with an object storage device OSD, the to-be-processed task is directly written into the task bucket of the object storage cluster, the transmission path in the data reading and writing process is reduced, the limitation of the gateway node is avoided, and the data filling efficiency and response speed can be improved. Therefore, the efficiency of data filling can be improved.

Description

Technical Field

[0001] This disclosure belongs to the field of computer technology, and specifically relates to a data filling method, apparatus, device, medium and product. Background Technology

[0002] Object storage is a technology that stores and manages data in an unstructured format (called objects). OSD (Object Storage Device) is a core component in a distributed storage cluster, responsible for data storage, management, and high availability assurance. OSD uses objects as the basic unit, encapsulating data and metadata into a single entity to achieve highly scalable, highly reliable, and high-performance data storage services. To ensure the stability of distributed data storage cluster performance, the cluster state is typically tested under high-water mark and high-concurrency conditions to verify its stability.

[0003] In general, common data filling tools such as Cosbench can be used to simulate real-world usage by sending client data filling requests to the OSD service process, which then fills the data on the OSD. However, this method requires a relatively long data write path, thus reducing data filling efficiency. Summary of the Invention

[0004] This disclosure is made in view of the above-mentioned problems. This disclosure provides a data filling method, apparatus, device, medium, and product that can improve the accuracy of data filling.

[0005] According to one aspect of this disclosure, a data population method is provided, applied to a data population device in an object storage cluster, the data population device communicating directly with an object storage device (OSD); the method includes:

[0006] The tasks to be processed are written to the task bucket of the object storage cluster;

[0007] Based on the time attributes of each task to be processed in the task bucket, the target task is read;

[0008] When the target task is a water level filling task, the target object is constructed and written to the OSD;

[0009] When the water level of the OSD meets the preset first threshold, the water level filling is completed.

[0010] Optionally, when the target task is a water level filling task, constructing the target object and writing it to the OSD includes:

[0011] Based on the task parameters of the water level filling task, determine the data volume of the target object;

[0012] Based on the task parameters and the data volume, the target object is constructed;

[0013] Start the water level filling thread; wherein, the water level filling thread is used to construct the water level filling request of the target object and submit the water level filling request to the OSD message queue, and the OSD message queue is read and executed by the internal thread of the OSD.

[0014] Optionally, the task parameters include at least one of the following: target filling water level, filling object name, and filling data size.

[0015] Optionally, the method further includes:

[0016] Start a listening thread, which is used to monitor the water level of the OSD.

[0017] Optionally, the method further includes:

[0018] When the water level of the OSD meets the preset second threshold, the actual water level of the target task is updated; wherein, the second threshold is one or more.

[0019] Optionally, the task status of each task to be processed in the task bucket includes at least one of the following: processing, suspended, completed, and pending deletion; the method further includes:

[0020] Update the task status of each pending task.

[0021] Optionally, updating the task status of each pending task includes:

[0022] Update the task status of the currently executing target task to "processing";

[0023] When the target task is completed, the task status of the target task will be updated to "completed".

[0024] Optionally, the method further includes:

[0025] When the target task is a suspended task, stop water level filling;

[0026] Update the task status of the target task to suspended.

[0027] Optionally, the method further includes:

[0028] When the target task is a continuation task for a suspended task, the task status of the target task is updated to "processing".

[0029] Continue performing the target task.

[0030] Optionally, the method further includes:

[0031] When the target task is a deletion task, stop water level filling and update the task status of the target task to be deleted;

[0032] A deletion request is constructed and submitted to the OSD message queue, which is read and executed by the internal thread of the OSD.

[0033] Optionally, the method further includes:

[0034] When the data filling device resumes normal operation, it checks whether there are any unfinished tasks in the task bucket.

[0035] If there are any unfinished tasks in progress, continue executing those tasks.

[0036] Optionally, before writing the task to be processed into the task bucket of the object storage cluster, the method further includes:

[0037] Based on the data filling device, acquire task parameters and water level filling commands;

[0038] Based on the task parameters and the water level filling command, the task to be processed is constructed.

[0039] According to another aspect of this disclosure, a data filling apparatus is provided that communicates directly with an object storage device (OSD), the apparatus comprising:

[0040] The write module is used to write the tasks to be processed into the task bucket of the object storage cluster;

[0041] The reading module is used to read the target task based on the time attributes of each task to be processed in the task bucket;

[0042] The construction module is used to construct the target object and write it to the OSD when the target task is a water level filling task;

[0043] The filling module is used to complete the water level filling when the water level of the OSD meets a preset first threshold.

[0044] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the above-described data filling method.

[0045] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the program being executed by a processor to implement the above-described data filling method.

[0046] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described data filling method.

[0047] In this disclosure, tasks to be processed are written into task buckets of an object storage cluster; based on the time attributes of each task to be processed in the task bucket, target tasks are read; when the target task is a water level filling task, the target object is constructed and written to the OSD; when the water level of the OSD meets a preset first threshold, water level filling is completed. By setting a data filling device inside the object storage cluster, the data filling device can communicate directly with the object storage device OSD, writing tasks to be processed directly into the task buckets of the object storage cluster. This reduces the transmission path during data read and write, avoids the limitations of gateway nodes, and improves data filling efficiency and response speed. Therefore, data filling efficiency can be improved.

[0048] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0049] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0050] Figure 1 A flowchart of a data filling method provided in this disclosure.

[0051] Figure 2 This is an architectural diagram of a data filling device provided in this disclosure.

[0052] Figure 3 Another flowchart for a data population method provided in this disclosure.

[0053] Figure 4 This is a schematic diagram of the structure of a data filling device provided in this disclosure.

[0054] Figure 5 This is a hardware block diagram of an electronic device provided in this disclosure.

[0055] Figure 6 This is a schematic diagram of a computer program product provided in this disclosure. Detailed Implementation

[0056] To enable those skilled in the art to better understand the technical solution of this application, the application scenario of this application will be described first below.

[0057] Object storage is a technology that stores and manages data in an unstructured format (called objects). A typical object storage architecture includes components such as access interfaces, metadata storage clusters, and distributed data storage clusters. The OSD (Object Storage Device) is the core component of the distributed storage cluster, responsible for data storage, management, and high availability assurance. The OSD uses objects as the basic unit, encapsulating data and metadata into a single unit to achieve highly scalable, highly reliable, and high-performance data storage services. To ensure the stability of the distributed data storage cluster's performance, the cluster state is typically tested under high-water mark and high concurrency conditions to verify its stability. In the above architecture, the OSD is the key component for actually storing data, responsible for storing objects on the physical device and providing read and write access. Other components are responsible for providing interfaces, controlling data access, and managing metadata and data.

[0058] In object storage clusters, water level filling refers to artificially increasing the amount of object data in the storage system to achieve a specific level of storage utilization. Testing the cluster status under high water level and high concurrency is a crucial step before cluster release and deployment. Its main purpose is to verify stability under high load, avoid deployment issues, analyze system response time, optimize resource utilization, ensure system scalability, and improve user experience. Through water level filling testing, potential problems can be identified and resolved, contingency plans can be developed, and the system can be ensured to operate smoothly and continuously optimize under various load and water level conditions.

[0059] Currently, common data population tools, such as Cosbench, can be used to simulate real-world usage by sending client data population requests to the OSD service process, which then performs the data population on the OSDs. The process is as follows: First, client traffic is load-balanced via SLB and sent to the object storage gateway RGW. The RGW analyzes the user's authentication and access information from the request and initiates authentication with the object storage cluster. After successful authentication by reading information from the metadata storage cluster, the RGW connects to the data storage cluster based on the obtained token. Upon successful connection, it queries the cluster's key configuration information. Next, the RGW parses the read / write requests sent by the client and obtains the backend storage service address based on the cluster configuration information. It then establishes a connection with the backend storage service process using the librados interface connect, sending the client's read / write requests to the backend storage service process. Finally, the backend process performs the read / write operations on the physical disk OSDs and returns the results to the client. However, this data population method requires a relatively long data write path, thus reducing data population efficiency.

[0060] To address the aforementioned technical problems, this disclosure provides an inventive concept: a data filling device is set up inside the object storage cluster, which communicates directly with the object storage device OSD. This allows for concurrent writing of data directly onto the OSD physical disk, bypassing the gateway node, reducing intermediate steps in the data read / write transmission path, avoiding the limitations of the gateway node, and improving filling efficiency and response speed.

[0061] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0062] Figure 1 This is a flowchart illustrating a data population method provided in this disclosure. Figure 1 As shown, this method is applied to a data population device in an object storage cluster. The data population device communicates directly with the object storage device (OSD), and includes:

[0063] S101: Write the tasks to be processed to the task bucket of the object storage cluster.

[0064] Specifically, upon receiving a task to be processed, the validity and legality of the task parameters can be checked based on the task type. If the check passes, the task can be written as a new object to the task bucket of the object storage cluster. Specifically, when checking task parameters, if the task type is a water level filling task, it is necessary to check whether the task parameters include the following fields: target water level, filling data size, and filling object name, and to check the validity and legality of the data based on the field types. If the task type is a suspended task, a resumed suspended task, or a deleted task, it is only necessary to check whether the task parameters include the filling object name field, and to check the validity and legality of the data based on the field type.

[0065] S102: Read the target task based on the time attributes of each task to be processed in the task bucket.

[0066] Specifically, when a task is written to the task bucket, its write time attribute is updated, indicating the time of the task's most recent modification. During task scheduling, the task with the most recently updated time attribute in the current task bucket is read as the target task. That is, the most recently updated task is executed first, ensuring timely response to changes in task status.

[0067] In addition, listeners can be set on task buckets. When a task in a task bucket changes, the OSD will respond immediately and perform corresponding actions based on the task status. Each task has a designated target OSD, and OSDs in the object storage cluster can execute in parallel without affecting each other.

[0068] S103: When the target task is a water level filling task, construct the target object and write it to the OSD.

[0069] Specifically, when the target task is a water level filling task, the gateway node is bypassed and the data is written directly to the OSD. After reading the target task, the task parameters can be obtained. Based on the task parameters, the number of target objects to be built can be determined, and the target objects are then built. A water level filling thread is started on the OSD to write the built target objects into the OSD.

[0070] S104: When the water level of the OSD meets the preset first threshold, the water level filling is completed.

[0071] Specifically, in this embodiment, the preset first threshold can be the target filling water level in the task parameters. When the OSD water level reaches the target filling water level, it means that the water level filling task has been completed.

[0072] In this disclosure, tasks to be processed are written into task buckets of an object storage cluster; based on the time attributes of each task to be processed in the task bucket, target tasks are read; when the target task is a water level filling task, the target object is constructed and written to the OSD; when the water level of the OSD meets a preset first threshold, water level filling is completed. By setting a data filling device in the object storage cluster, the data filling device can communicate directly with the object storage device OSD, and the tasks to be processed can be written directly into the task buckets of the object storage cluster. This reduces the transmission path during data reading and writing, avoids the limitations of gateway nodes, improves data filling efficiency and response speed, compresses testing time, and improves the efficiency of version iteration. Therefore, data filling efficiency can be improved.

[0073] In one possible implementation, when the target task is a water level filling task, an exemplary method for constructing the target object and writing it to the OSD includes:

[0074] Based on the task parameters of the water level filling task, determine the data volume of the target object; construct the target object based on the task parameters and data volume; and start the water level filling thread.

[0075] Specifically, when constructing the target object, it is necessary to obtain the task parameters of the target task in the task bucket and update the task status of the target task in the task bucket to "processing". For example, the task parameters include at least one of the following: target fill level, fill object name, and fill data size. The fill object name is used to match the target task in the task bucket when writing the target object. The current total capacity and used capacity of the OSD are obtained from the object storage cluster. Based on the total capacity, used capacity, and target fill level, the amount of data required to reach the target fill level can be calculated. The following formula is an example formula for calculating the amount of data required to reach the target fill level:

[0076]

[0077] Then, based on the amount of data to be filled and the size of the filling data, the data volume of the target object can be calculated. In this embodiment, a bufferlist can be used to construct Y / X target objects obj with a filling data size of X and an object name of m_osdid_random.

[0078] Finally, a water level filling thread is started on the OSD. This thread constructs water level filling requests for the target objects and submits them to the OSD message queue. The OSD message queue is read and executed by an internal thread of the OSD. Specifically, in this embodiment, a pseudo-random algorithm can be used to uniformly map objects onto virtual nodes, constructing a water level filling request for each target object and submitting it to the OSD message queue. The OSD's internal thread then reads and writes the request to disk.

[0079] This water level filling method avoids the need for third-party filling tools to require users to manually configure multiple complex parameters, such as concurrency and write duration. It can accurately control the usage capacity of the object storage cluster at the target filling water level, automatically calculate the amount of data written to the OSD, and monitor the cluster capacity in real time, estimate the time required to reach the target filling water level, and ensure that the object storage cluster is within the safe water level.

[0080] In one possible implementation, the method further includes:

[0081] Start the listening thread.

[0082] Specifically, a listening thread can be started simultaneously with the water level filling thread. This listening thread monitors the OSD water level, which in an object storage cluster represents the utilization rate of storage space. This allows users to easily view the filling progress and continuously monitor until the water level reaches the target filling level.

[0083] In one possible implementation, the method further includes:

[0084] When the OSD water level meets the preset second threshold, update the actual water level of the target task.

[0085] Specifically, when writing target objects to the OSD begins, to ensure the OSD's water level is safe, a listening thread can be started when the OSD's water level reaches a preset second threshold. This thread monitors the current water level of the OSD, updates the actual water level of the target task in real time, and visualizes the actual water level for easy monitoring of the filling progress. When the actual water level reaches the target filling level, the task status of the target task in the task bucket is changed to "completed," indicating that the water level filling task is finished. The second threshold can be one or more; for example, the actual water level can be updated when it reaches 10%, 20%, and / or 30%.

[0086] In one possible implementation, the task status of each pending task in the task bucket includes at least one of the following: processing, suspended, completed, and pending deletion; the method further includes:

[0087] Update the task status of each pending task.

[0088] Specifically, the task status can be updated when a pending task is executing its current stage or completing its current stage. For updates to tasks with statuses of suspended, completed, or pending deletion, the corresponding pending task in the task bucket can be located and modified using the object name populated in the task parameters.

[0089] For example: update the task status of the currently executing target task to "processing"; when the target task is completed, update the task status of the target task to "completed".

[0090] Specifically, when the target task is read as a water level filling task from the task bucket, the task status of the currently executing target task is updated to "processing". At this time, the thread that writes the target object to the OSD is executed. When the target task is completed, that is, when the actual water level in the OSD reaches the target filling water level, the task status of the target task in the task bucket is updated to "completed".

[0091] In addition, after the OSD water level filling is completed, the task status of the target task in the task bucket can be updated to pending deletion, and a deletion task request can be constructed and submitted to the OSD message queue. If the target object that matches the name of the object to be filled cannot be found in the object storage cluster, it means that the deletion is complete. At this time, the target task in the task bucket can be deleted.

[0092] It is understandable that the same logic can be applied to update the states of other tasks, which will not be elaborated here.

[0093] In one possible implementation, the method further includes:

[0094] When the target task is a suspended task, stop water level filling; update the target task's task status to suspended.

[0095] Specifically, when the target task is a suspended task, it means that the water level filling thread needs to be paused. The name of the object to be filled is retrieved from the suspended task's task parameters. Based on this name, a search is performed in the task bucket to identify the water level filling task with the same name as the suspended task. The suspended task and the water level filling task are target tasks operating on the same object. The water level filling thread for the OSD corresponding to the water level filling task is stopped to ensure no further filling operations are performed, and the task status of the target task in the task bucket is updated to suspended.

[0096] In one possible implementation, the method further includes:

[0097] When the target task is a continuation task of a suspended task, update the target task's task status to "processing"; continue executing the target task.

[0098] Specifically, when the target task is a continuation task for a suspended task, this continuation task is one that continues filling the water level based on the paused water level filling thread, rather than filling paused due to OSD reasons. The name of the filling object is obtained from the target task's task parameters. Based on the name of the filling object of the pending water level task, a search is performed in the task bucket to identify the water level filling task with the same name as the continuation task. Since the continuation task and the water level filling task are target tasks operating on the same object, the task status of the target task is updated to "processing," and execution of the target task continues.

[0099] In one possible implementation, the method further includes:

[0100] When the target task is a deletion task, stop water level filling and update the task status of the target task to pending deletion; construct a deletion request and submit it to the OSD message queue.

[0101] Specifically, when the target task is a deletion task, the name of the object to be filled is retrieved from the task parameters of the deletion task. Based on this name, a search is performed in the task bucket to identify a water level filling task with the same name. The deletion task and the water level filling task are considered target tasks operating on the same object. The water level filling thread of the OSD corresponding to the water level filling task is stopped to ensure no further filling operations are performed, and the task status of the target water level task is updated to deleted. A deletion request is constructed and submitted to the OSD message queue. The corresponding water level filling request is deleted in the water level filling thread. The OSD message queue is read by the OSD's internal thread, which then executes the deletion operation. Furthermore, if the object storage cluster cannot find a target object matching the name to be filled, it indicates that the request corresponding to that target object has already been cleaned up. In this case, the corresponding target task can be directly deleted from the task bucket.

[0102] In one possible implementation, the method further includes:

[0103] When the data filling device resumes normal operation, it checks whether there are any unfinished tasks in the task bucket; if there are unfinished tasks, it continues to execute the tasks.

[0104] Specifically, during the water level filling task, OSD (Optical System Device) anomalies may cause the task to be interrupted. After the OSD recovers, such as through re-connection or a restart, the data filling device resumes normal operation. At this point, the data filling device can proactively check the task bucket for any unfinished tasks related to that OSD that are currently in progress. This allows for automatic recovery and continuation of processing after a task interruption, ensuring the continuity and stability of the water level filling task, avoiding resource waste and filling interruptions, and reducing the need for manual intervention.

[0105] If the target object exists and its actual water level does not meet the preset first threshold, then the task parameters of the target task are read, the data volume of the target object is recalculated, and the target object is constructed based on the task parameters and data volume, and the water level filling thread is started. The execution flow of the water level filling task was mentioned earlier and will not be repeated here.

[0106] If the target task's status is "pending deletion," then the name of the populated object for the target task is read, and the deletion task is executed. The execution flow for deleting a task was mentioned earlier and will not be repeated here.

[0107] In one possible implementation, the method further includes the following step before writing the task to be processed to the task bucket of the object storage cluster:

[0108] Based on the data filling device, the task parameters and water level filling command are obtained; based on the task parameters and water level filling command, the task to be processed is constructed.

[0109] Specifically, the data population device can be viewed as an internal population interface that can directly connect to the backend cluster and object storage cluster. The data population device can directly receive callbacks of task parameters and water level population commands from the backend cluster. The backend cluster can directly send water level population commands to the data population device using the admin socaket mechanism (inter-process communication mechanism). Based on the task parameters and water level population commands, the task to be processed can be constructed.

[0110] Figure 2 Here is an architectural diagram of a data filling device provided in this disclosure, such as Figure 2 As shown. The backend cluster 210 is directly connected to the object storage cluster 230 through the data population device 220. The data population device 220 can be set in the object storage cluster 230, which includes multiple OSDs.

[0111] Figure 3 Another flowchart for a data population method provided in this disclosure. Figure 3 As shown, the method includes:

[0112] S301: Construct tasks to be processed.

[0113] Specifically, based on the data filling device, it directly connects to the backend cluster and the object storage cluster, and obtains task parameters and water level filling commands from the backend cluster via callback; based on the task parameters and water level filling commands, it constructs the task to be processed.

[0114] S302: Inspect the task to be processed.

[0115] Specifically, based on the task type of the task to be processed, the parameter validity and legality of the task to be processed are checked. If the check passes, the process proceeds to S303. If the check fails, it means that the task to be processed is invalid and the process ends.

[0116] S303: Write the tasks to be processed to the task bucket of the object storage cluster.

[0117] Specifically, the task to be processed is written as a new object into the task bucket of the object storage cluster.

[0118] S304: Based on the time attributes of each pending task in the task bucket, read the target task and execute the target task.

[0119] Specifically, during task scheduling, the task with the most recently updated time attribute in the current task bucket is read as the target task based on the time attribute of each pending task in the task bucket, and then executed. The execution method of the target task, as mentioned earlier, will not be elaborated upon here.

[0120] S305: Start the OSD water level monitoring thread.

[0121] Specifically, while starting the water level filling thread, a listening thread can also be started. The listening thread is used to monitor the OSD water level. In an object storage cluster, the water level is the utilization rate of storage space.

[0122] Figure 4 This is a schematic diagram of the structure of a data filling device provided in this disclosure. Figure 4 As shown, the device 400 includes: a writing module 410, a reading module 420, a building module 430, and a filling module 440.

[0123] The writing module 410 is used to write the tasks to be processed into the task bucket of the object storage cluster;

[0124] The reading module 420 is used to read the target task based on the time attributes of each task to be processed in the task bucket;

[0125] Module 430 is used to construct a target object and write it to the OSD when the target task is a water level filling task;

[0126] The filling module 440 is used to complete the water level filling when the water level of the OSD meets the preset first threshold.

[0127] Optionally, the building module is used for:

[0128] Based on the task parameters of the water level filling task, determine the data volume of the target object;

[0129] Based on the task parameters and the data volume, the target object is constructed;

[0130] Start the water level filling thread; wherein, the water level filling thread is used to construct the water level filling request of the target object and submit the water level filling request to the OSD message queue, and the OSD message queue is read and executed by the internal thread of the OSD.

[0131] Optionally, the device further includes:

[0132] Start a listening thread, which is used to monitor the water level of the OSD.

[0133] Optionally, the device further includes:

[0134] When the water level of the OSD meets the preset second threshold, the actual water level of the target task is updated; wherein, the second threshold is one or more.

[0135] Optionally, the device further includes:

[0136] Update the task status of each pending task.

[0137] Optionally, the device further includes:

[0138] Update the task status of the currently executing target task to "processing";

[0139] When the target task is completed, the task status of the target task will be updated to "completed".

[0140] Optionally, the device further includes:

[0141] When the target task is a suspended task, stop water level filling;

[0142] Update the task status of the target task to suspended.

[0143] Optionally, the device further includes:

[0144] When the target task is a continuation task for a suspended task, the task status of the target task is updated to "processing".

[0145] Continue performing the target task.

[0146] Optionally, the device further includes:

[0147] When the target task is a deletion task, stop water level filling and update the task status of the target task to be deleted;

[0148] A deletion request is constructed and submitted to the OSD message queue, which is read and executed by the internal thread of the OSD.

[0149] Optionally, the device further includes:

[0150] When the data filling device resumes normal operation, it checks whether there are any unfinished tasks in the task bucket.

[0151] If there are any unfinished tasks in progress, continue executing those tasks.

[0152] Optionally, the device further includes:

[0153] Before writing the task to be processed into the task bucket of the object storage cluster, the task parameters and water level filling command are obtained based on the data filling device.

[0154] Based on the task parameters and the water level filling command, the task to be processed is constructed.

[0155] This application also provides an electronic device for performing the above-described data filling method. Please refer to... Figure 5 It illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 5 As shown, the electronic device 5 includes: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected via the bus 502. The memory 501 stores a computer program that can run on the processor 500. When the processor 500 runs the computer program, it executes the data filling method provided in any of the foregoing embodiments of this application.

[0156] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0157] Bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 501 is used to store programs. After receiving an execution instruction, the processor 500 executes the program. The data filling method disclosed in any of the foregoing embodiments of this application can be applied to the processor 500, or implemented by the processor 500.

[0158] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0159] The electronic device provided in this application embodiment and the data filling method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0160] This application also provides a computer-readable storage medium corresponding to the data filling method provided in the foregoing embodiments. The computer-readable storage medium shown may be an optical disc, on which a computer program is stored. When the computer program is run by a processor, it executes the data filling method provided in any of the foregoing embodiments.

[0161] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0162] The computer-readable storage medium provided in the above embodiments of this application and the data filling method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0163] This application also provides a computer program product 600, such as... Figure 6 As shown. This computer program product carries a computer program 601, the instructions of which can be used to execute the steps of the data filling method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0164] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0165] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0166] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0167] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0168] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0169] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0170] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0171] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method of data padding, characterized by, The application discloses a data filling device applied to an object storage cluster, wherein the data filling device directly communicates with an object storage device (OSD); the method comprises the following steps: writing a to-be-processed task into a task bucket of the object storage cluster; reading a target task based on a time attribute of each to-be-processed task in the task bucket; when the target task is a water level filling task, constructing a target object and writing the target object into the OSD; when a water level of the OSD meets a preset first threshold, completing water level filling.

2. The method of claim 1, wherein, The step of, when the target task is the water level filling task, constructing the target object and writing the target object into the OSD, comprises the following steps: determining a data amount of the target object based on a task parameter of the water level filling task; constructing the target object based on the task parameter and the data amount; starting a water level filling thread; wherein the water level filling thread is used for constructing a water level filling request of the target object and submitting the water level filling request into an OSD message queue, and the OSD message queue is read and executed by an internal thread of the OSD.

3. The method of claim 2, wherein, The task parameter comprises at least one of the following: a target filling water level, a filling object name and a filling data size.

4. The method of claim 1, wherein, The method further comprises the following steps: starting a monitoring thread, wherein the monitoring thread is used for monitoring the water level of the OSD.

5. The method of claim 1, wherein, The method further comprises the following steps: when the water level of the OSD meets a preset second threshold, updating an actual water level of the target task; wherein the second threshold is one or more.

6. The method of claim 1, wherein, The task state of each to-be-processed task in the task bucket comprises at least one of the following: processing, suspension, completion and deletion; the method further comprises the following steps: updating the task state of each to-be-processed task.

7. The method of claim 6, wherein, The step of updating the task state of each to-be-processed task comprises the following steps: updating the task state of the target task currently executed to processing; when the target task is executed, updating the task state of the target task to completion.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises the following steps: when the target task is a suspension task, stopping water level filling; updating the task state of the target task to suspension.

9. The method according to any one of claims 1 to 7, characterized in that, The method further comprises the following steps: when the target task is a continuation task for the suspension task, updating the task state of the target task to processing; continuing to execute the target task.

10. The method according to any one of claims 1 to 7, characterized in that, The method further comprises the following steps: when the target task is a deletion task, stopping water level filling and updating the task state of the target task to deletion; constructing a deletion request and submitting the deletion request into an OSD message queue, and the OSD message queue is read and executed by an internal thread of the OSD.

11. The method according to any one of claims 1 to 7, characterized in that, The method further comprises the following steps: when the data filling device resumes normal work, checking whether there is an uncompleted processing task in the task bucket; if there is an uncompleted processing task, continuing to execute the processing task.

12. The method of claim 1, wherein, Before the step of writing the to-be-processed task into the task bucket of the object storage cluster, the method further comprises the following steps: based on the data filling device, acquiring a task parameter and a water level filling command; based on the task parameter and the water level filling command, constructing the to-be-processed task.

13. A data padding apparatus, characterized by comprising: The data filling device directly communicates with an object storage device (OSD), and the device comprises the following steps: A writing module, configured to write a to-be-processed task into a task bucket of an object storage cluster; A reading module, configured to read a target task based on a time attribute of each to-be-processed task in the task bucket; A constructing module, configured to construct a target object and write the OSD when the target task is a water level filling task; A filling module, configured to complete water level filling when a water level of the OSD meets a preset first threshold.

14. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-12.

15. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the method of any one of claims 1-12.

16. A computer program product, characterised in that, The non-transitory computer-readable storage medium includes or stores computer-readable code, and when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device executes the method of any one of claims 1-12.

Images