Data filling method and device, equipment, medium and product

Abstract

The invention relates to the technical field of computers, and particularly provides a data filling method and device, equipment, a medium and a product. The method comprises the following steps: writing a to-be-processed task into a task bucket of an object storage cluster; reading the target task based on the time attribute of each to-be-processed task in the task bucket; when the target task is a water level filling task, constructing a target object and writing the target object into an OSD; and when the water level of the OSD meets a preset first threshold value, completing water level filling. By setting the data filling device in the object storage cluster, the data filling device can directly communicate with the object storage device OSD, and the to-be-processed task is directly written into the task bucket of the object storage cluster, so that the process of a transmission path in the data reading and writing process is reduced, the limitation of a gateway node is avoided, and the data reading and writing efficiency is improved. The data filling efficiency and the response speed can be improved. Therefore, the data filling efficiency can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of computer technology, and particularly relates to a data filling method, device, equipment, medium and product. Background Art

[0002] Object storage is a technology that stores and manages data in unstructured formats (called objects). Object Storage Devices (OSDs) are core components of distributed storage clusters, responsible for data storage, management, and high availability. OSDs use objects as the basic unit, encapsulating data and metadata into a single entity, enabling highly scalable, reliable, and high-performance data storage services. To ensure the stability of distributed data storage clusters, cluster status is typically tested under high concurrency conditions to verify cluster stability.

[0003] Typically, you can use common population tools, such as Cosbench, to simulate actual usage. This sends client data population requests to the OSD service process, which then populates the data on the OSD. This data population method requires a longer data write path, reducing data population efficiency. Summary of the Invention

[0004] In view of the above problems, the present disclosure is proposed. The present disclosure provides a data filling method, apparatus, device, medium and product, which can improve the accuracy of data filling.

[0005] According to one aspect of the present disclosure, a data filling method is provided, which is applied to a data filling device in an object storage cluster, wherein the data filling device directly communicates with an object storage device (OSD). The method includes: Writing the pending tasks into the task bucket of the object storage cluster; Read the target task based on the time attribute of each pending task in the task bucket; When the target task is a water level filling task, construct a target object and write it into the OSD; When the water level of the OSD meets a preset first threshold, the water level filling is completed.

[0006] Optionally, when the target task is a water level filling task, constructing a target object and writing it into the OSD includes: Determining the data volume of the target object based on the task parameters of the water level filling task; constructing the target object based on the task parameters and the data volume; Start a water level filling thread; wherein, the water level filling thread is used to construct a water level filling request for the target object and submit the water level filling request to the OSD message queue, and the OSD message queue is read and executed by the internal thread of the OSD.

[0007] Optionally, the task parameters include at least one of the following: a target filling level, a filling object name, and a filling data size.

[0008] Optionally, the method further includes: A monitoring thread is started, where the monitoring thread is used to monitor the water level of the OSD.

[0009] Optionally, the method further includes: When the water level of the OSD meets a preset second threshold, the actual water level of the target task is updated; wherein the second threshold is one or more.

[0010] Optionally, the task status of each pending task in the task bucket includes at least one of the following: being processed, pending, completed, and pending deletion; the method further includes: Update the task status of each of the pending tasks.

[0011] Optionally, updating the task status of each pending task includes: Update the task status of the currently executed target task to being processed; When the target task is completed, the task status of the target task is updated to completed.

[0012] Optionally, the method further includes: When the target task is a pending task, stop filling the water level; Update the task status of the target task to suspended.

[0013] Optionally, the method further includes: When the target task is a continued task for a suspended task, updating the task status of the target task to being processed; Continue to execute the target task.

[0014] Optionally, the method further includes: When the target task is a deletion task, stop filling the water level and update the task status of the target task to be deleted; A delete request is constructed and submitted to the OSD message queue, which is read and executed by the OSD's internal thread.

[0015] Optionally, the method further includes: When the data filling device resumes normal operation, checking whether there are any unfinished tasks in progress in the task bucket; If there are unfinished tasks being processed, continue to execute the tasks being processed.

[0016] Optionally, before writing the pending task into the task bucket of the object storage cluster, the method further includes: Based on the data filling device, obtaining task parameters and water level filling commands; The task to be processed is constructed based on the task parameters and the water level filling command.

[0017] According to another aspect of the present disclosure, a data filling device is provided, wherein the data filling device directly communicates with an object storage device (OSD), and the device includes: A writing module, used to write the pending tasks into the task bucket of the object storage cluster; A reading module, configured to read a target task based on the time attributes of each pending task in the task bucket; A construction module, configured to construct a target object and write the target object into the OSD when the target task is a water level filling task; The filling module is used to complete water level filling when the water level of the OSD meets a preset first threshold.

[0018] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned data filling method.

[0019] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. The program is executed by a processor to implement the above-mentioned data filling method.

[0020] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes and implements the above-mentioned data filling method.

[0021] In the present disclosure, pending tasks are written into the task bucket of the object storage cluster; based on the time attributes of each pending task in the task bucket, the target task is read; when the target task is a water level filling task, the target object is constructed and written to the OSD; when the water level of the OSD meets a preset first threshold, the water level filling is completed. By providing a data filling device within the object storage cluster, the data filling device can communicate directly with the object storage device OSD and write the pending tasks directly into the task bucket of the object storage cluster, reducing the transmission path process during data reading and writing, avoiding the limitation of gateway nodes, and improving the data filling efficiency and response speed. Therefore, the efficiency of data filling can be improved.

[0022] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other purposes, features, and advantages of the present disclosure will become more apparent through a more detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and are not intended to limit the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 A flow chart of a data filling method provided by the present disclosure.

[0025] Figure 2 This is an architectural diagram of a data filling device provided by the present disclosure.

[0026] Figure 3 Another flow chart of a data filling method provided by the present disclosure.

[0027] Figure 4 A schematic structural diagram of a data filling device provided by the present disclosure.

[0028] Figure 5 A hardware block diagram of an electronic device provided in the present disclosure.

[0029] Figure 6 A schematic diagram of a computer program product provided by the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the application scenario of the solution of the present application is first described below.

[0031] Object storage is a technology for storing and managing data in an unstructured format (called objects). A typical object storage architecture includes components such as access interfaces, metadata storage clusters, and distributed data storage clusters. OSD (Object Storage Device) is the core component of a distributed storage cluster, responsible for data storage, management, and high availability. OSD uses objects as the basic unit, encapsulating data and metadata into a whole, achieving highly scalable, highly reliable, and high-performance data storage services. To ensure the stability of the distributed data storage cluster, the cluster status is usually tested under high watermarks and high concurrency to verify the stability of the cluster. In the above architecture, OSD is the key component for actual data storage, responsible for storing objects on physical devices and providing read and write access. Other components are responsible for providing interfaces, controlling data access, and managing metadata and data.

[0032] In object storage clusters, waterfilling refers to artificially increasing the amount of object data in the storage system to achieve a specific storage utilization level. Testing the cluster's status under high watermarks and high concurrency is a crucial step before a cluster is released. Its primary purpose is to verify stability under high load, avoid rollout issues, analyze system response time, optimize resource utilization, ensure system scalability, and enhance the user experience. Waterfilling testing helps identify and resolve potential issues, develop contingency plans, and ensure smooth operation and continuous optimization of the system under various load and watermark conditions.

[0033] Currently, common population tools, such as Cosbench, can be used to simulate real-world usage. This sends client data population requests to the OSD service process, which then populates the data on the OSD. The process works as follows: First, client traffic is load-balanced by the SLB and sent to the object storage gateway (RGW). The RGW gateway analyzes the user authentication and access information in the request and initiates identity authentication with the object storage cluster. After authentication is passed by reading information from the metadata storage cluster, the RGW connects to the data storage cluster using the token obtained. Once connected, it queries the cluster for key configuration information. Next, the RGW gateway parses the client's read and write requests and obtains the backend storage service address based on the cluster configuration. It establishes a connection to the backend storage service process using the librados interface connect, forwarding the client's read and write requests to the backend storage service process. The backend process then reads and writes data on the physical disks on the OSD and returns the results to the client. However, this data population method requires a longer data write path, which reduces data population efficiency.

[0034] In order to solve the above technical problems, the present disclosure provides an inventive concept: a data filling device is set inside the object storage cluster, and the data filling device communicates directly with the object storage device OSD, so that data can be written concurrently directly on the OSD physical disk, bypassing the gateway node, reducing the intermediate steps in the transmission path of data reading and writing, avoiding the limitations of the gateway node, and improving the filling efficiency and response speed.

[0035] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, the following will describe in detail exemplary embodiments of the present disclosure with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0036] Figure 1 A flow chart of a data filling method provided by the present disclosure. Figure 1 As shown, the method is applied to a data filling device in an object storage cluster, where the data filling device communicates directly with the object storage device OSD, and includes: S101: Writing the pending task into the task bucket of the object storage cluster.

[0037] Specifically, in the case of a received pending task, the parameter validity and legality of the task parameters of the pending task can be checked based on the task type of the pending task. If the check passes, the pending task can be written into the task bucket of the object storage cluster as a new object. Among them, when checking the task parameters, if the task type is a water level filling task, it is necessary to check whether the task parameters include the following fields: target filling water level, filling data size and filling object name, and check whether the data is legal and valid based on the type of the field; if the task type is a suspended task, a continued task of a suspended task or a deleted task, it is only necessary to check whether the task parameters include the filling object name field, and check whether the data is legal and valid based on the type of the field.

[0038] S102: Read the target task based on the time attribute of each to-be-processed task in the task bucket.

[0039] Specifically, when a pending task is written to a task bucket, the write-time attribute is updated to indicate the time of the task's most recent modification. When scheduling tasks, the task with the most recently updated time attribute in the current task bucket is read as the target task. This prioritizes the most recently updated task, ensuring timely response to changes in task status.

[0040] In addition, you can also set up a monitor on the task bucket. When the task in the task bucket changes, the OSD will respond immediately and perform corresponding actions based on the task status. Each task has a specified target OSD, and the OSDs in the object storage cluster can execute in parallel without affecting each other.

[0041] S103: When the target task is a water level filling task, construct a target object and write it to the OSD.

[0042] Specifically, when the target task is a watermark filling task, data is written directly to the OSD, bypassing the gateway node. After reading the target task, the task parameters are obtained. Based on the task parameters, the number of target objects to be built is determined, and the target objects are built. The watermark filling thread is started on the OSD, and the built target objects are written to the OSD.

[0043] S104: When the water level of the OSD meets the preset first threshold, the water level filling is completed.

[0044] Specifically, in this embodiment, the preset first threshold may be a target filling water level in the task parameters. When the water level of the OSD reaches the target filling water level, it indicates that the water level filling task has been completed.

[0045] In the present disclosure, pending tasks are written into the task bucket of the object storage cluster; based on the time attributes of each pending task in the task bucket, the target task is read; when the target task is a water level filling task, the target object is constructed and written to the OSD; when the water level of the OSD meets the preset first threshold, the water level filling is completed. By setting a data filling device in the object storage cluster, the data filling device can communicate directly with the object storage device OSD, and the pending tasks are directly written into the task bucket of the object storage cluster, which reduces the transmission path process during data reading and writing, avoids the limitation of gateway nodes, improves data filling efficiency and response speed, compresses test time, and improves the efficiency of version iteration. Therefore, the efficiency of data filling can be improved.

[0046] In a possible implementation, when the target task is a water level filling task, an exemplary method of constructing a target object and writing it to the OSD includes: Based on the task parameters of the water level filling task, determine the data volume of the target object; based on the task parameters and the data volume, build the target object; and start the water level filling thread.

[0047] Specifically, when constructing the target object, it is necessary to obtain the task parameters of the target task in the task bucket and update the task status of the target task in the task bucket to being processed. For example, the task parameters include at least one of the following: target fill water level, fill object name, and fill data size. Among them, the fill object name is used to match the target task in the task bucket when writing the target object. The current total capacity and used capacity of the OSD are obtained from the object storage cluster. Based on the total capacity, used capacity, and target fill water level, the amount of data that needs to be filled to reach the target fill water level can be calculated. The following formula is an exemplary formula for calculating the amount of data that needs to be filled to reach the target fill water level:

[0048] Afterwards, the data amount of the target object can be calculated based on the data amount to be filled and the filling data size. In this embodiment, bufferlist can be used to construct Y / X target objects obj with a filling data size of X and a filling object name of m_osdid_random.

[0049] Finally, a water level fill thread is started on the OSD. This thread constructs a water level fill request for the target object and submits the request to the OSD message queue. The OSD message queue is read and executed by an internal thread of the OSD. Specifically, in this embodiment, a pseudo-random algorithm can be used to evenly map objects to virtual nodes. A water level fill request is constructed for each target object and submitted to the OSD message queue. The OSD internal thread reads and writes the request to disk.

[0050] This water level filling method can avoid the need for users to manually configure multiple complex parameters such as concurrency and write duration in third-party filling tools. It can accurately control the usage capacity of the object storage cluster at the target filling water level, automatically calculate the amount of filling data written to the OSD, and monitor the cluster capacity in real time. It can estimate the time required to achieve the target filling water level and ensure that the object storage cluster is within a safe water level.

[0051] In one possible implementation, the method further includes: Start the listening thread.

[0052] Specifically, when starting the water level filling thread, you can also start a monitoring thread. The monitoring thread is used to monitor the water level of the OSD. In an object storage cluster, the water level is the utilization rate of the storage space. This allows users to easily view the filling progress and continuously monitor until the water level reaches the target filling level.

[0053] In one possible implementation, the method further includes: When the water level of the OSD meets the preset second threshold, the actual water level of the target task is updated.

[0054] Specifically, when writing the target object to the OSD, in order to ensure the safety of the OSD's water level, when the OSD's water level meets a preset second threshold, a listening thread can be started to monitor the OSD's current water level, update the actual water level of the target task in real time, and visualize the actual water level to facilitate viewing the filling progress. When the actual water level reaches the target filling level, the task status of the target task in the task bucket is modified to completed, indicating that the water level filling task has been completed. The second threshold can be one or more, for example, the actual water level can be updated when the water level reaches 10%, 20%, and / or 30%.

[0055] In one possible implementation, the task status of each pending task in the task bucket includes at least one of the following: processing, pending, completed, and pending deletion; the method further includes: Update the task status of each pending task.

[0056] Specifically, when a pending task is executing or completing its current task phase, the task status can be updated. For tasks with pending, completed, or pending deletion status, the task can be modified by finding the corresponding pending task in the task bucket using the object name filled in the task parameters.

[0057] Exemplarily: the task status of the currently executed target task is updated to being processed; when the target task is completed, the task status of the target task is updated to being completed.

[0058] Specifically, when the target task read from the task bucket is a water level filling task, the task status of the currently executed target task is updated to processing. At this time, the thread that writes the target object to the OSD is executed. When the target task is completed, that is, when the actual water level in the OSD reaches the target filling water level, the task status of the target task in the task bucket is updated to completed.

[0059] In addition, after the water level of the OSD has been filled, the task status of the target task in the task bucket can be updated to pending deletion, and a deletion task request can be constructed and submitted to the message queue of the OSD. When the target object matching the filled object name cannot be retrieved in the object storage cluster, it means that the deletion is completed. At this time, the target task in the task bucket can be deleted.

[0060] It is understandable that the update methods of other task states can be inferred similarly, and will not be described in detail here.

[0061] In one possible implementation, the method further includes: When the target task is a suspended task, stop filling the water level; update the task status of the target task to suspended.

[0062] Specifically, when the target task is a suspended task, it means that the water level filling thread needs to be suspended. Get the filling object name in the task parameters of the suspended task, and based on the filling object name of the suspended task, search in the task bucket to determine the water level filling task with the same filling object name as the suspended task. The suspended task and the water level filling task are target tasks that operate on the same object. Stop the water level filling thread of the OSD corresponding to the water level filling task to ensure that no more filling operations are performed, and update the task status of the target task in the task bucket to suspended.

[0063] In one possible implementation, the method further includes: When the target task is a continued task for a suspended task, the task status of the target task is updated to being processed; and the target task is continued to be executed.

[0064] Specifically, when the target task is a continued task for a suspended task, the continued task is a task that continues filling based on the suspended waterlevel filling thread, rather than a suspended filling caused by OSD reasons. Get the filling object name in the task parameters of the target task, and search in the task bucket based on the filling object name of the pending waterlevel task to determine the waterlevel filling task with the same filling object name as the continued task. The continued task and the waterlevel filling task are target tasks that operate on the same object. Update the task status of the target task to processing, and continue executing the target task.

[0065] In one possible implementation, the method further includes: When the target task is a deletion task, stop filling the watermark and update the task status of the target task to pending deletion; build a deletion request and submit it to the OSD message queue.

[0066] Specifically, when the target task is a deletion task, obtain the filling object name in the task parameters of the deletion task, and search in the task bucket based on the filling object name of the deletion task to determine the waterlevel filling task with the same filling object name as the deletion task. The deletion task and the waterlevel filling task are target tasks that operate on the same object. Stop the waterlevel filling thread of the OSD corresponding to the waterlevel filling task to ensure that no more filling operations are performed, and update the task status of the target waterlevel task to deletion. Construct a deletion request and submit it to the OSD message queue, delete the corresponding waterlevel filling request in the waterlevel filling thread, and the OSD message queue is read by the internal thread of the OSD and executes the deletion operation. In addition, if the object storage cluster cannot retrieve the target object that matches the filling object name, it means that the request corresponding to the target object has been cleared. At this time, the corresponding target task can be directly deleted from the task bucket.

[0067] In one possible implementation, the method further includes: When the data filling device resumes normal operation, it checks whether there are any unfinished processing tasks in the task bucket; if there are any unfinished processing tasks, the processing tasks are continued.

[0068] Specifically, when executing a water level filling task, an OSD anomaly may cause the water level filling task to be interrupted. After the OSD recovers, such as coming back online or restarting, the data filling device resumes normal operation. At this time, the data filling device can proactively check whether there are any unfinished tasks related to the OSD in the task bucket with the status of "in-process". In this way, the task can be automatically restored and continued after the interruption, ensuring the continuity and stability of the water level filling task, avoiding resource waste and filling interruptions, and reducing the need for manual intervention.

[0069] If it exists, and the actual water level of the OSD does not meet the preset first threshold, the task parameters of the target task are read, the data volume of the target object is recalculated, and the target object is constructed based on the task parameters and data volume, and the water level filling thread is started. As mentioned above, the execution process of the water level filling task will not be repeated here.

[0070] If the target task's task status is pending deletion, the target task's populated object name is read to execute the deletion task. As mentioned above, the execution process of the deletion task will not be repeated here.

[0071] In a possible implementation, before writing the pending task into the task bucket of the object storage cluster, the method further includes: Based on the data filling device, task parameters and water level filling commands are obtained; based on the task parameters and water level filling commands, tasks to be processed are constructed.

[0072] Specifically, the data filling device can be viewed as an internal filling interface, directly connecting to the backend cluster and the object storage cluster. The data filling device can directly call back task parameters and water level filling commands from the backend cluster. The backend cluster can directly use the admin socket mechanism (inter-process communication) to send water level filling commands to the data filling device. Based on the task parameters and water level filling commands, a pending task can be constructed.

[0073] Figure 2 This is an architectural diagram of a data filling device provided by the present disclosure, such as Figure 2 The backend cluster 210 is directly connected to the object storage cluster 230 through the data filling device 220. The data filling device 220 can be set in the object storage cluster 230, and the object storage cluster 230 includes multiple OSDs.

[0074] Figure 3 Another flow chart of a data filling method provided by the present disclosure. Figure 3 As shown, the method includes: S301: Construct tasks to be processed.

[0075] Specifically, based on the data filling device, the backend cluster and the object storage cluster are directly connected, and the task parameters and water level filling commands are obtained from the backend cluster callback; based on the task parameters and the water level filling commands, the tasks to be processed are constructed.

[0076] S302: Check the tasks to be processed.

[0077] Specifically, based on the task type of the task to be processed, the parameter validity and legality of the task parameters of the task to be processed are checked. If the check passes, the process proceeds to S303. If the check fails, it means that the task to be processed is invalid and the process ends.

[0078] S303: Writing the pending task into the task bucket of the object storage cluster.

[0079] Specifically, the pending task is written as a new object into the task bucket of the object storage cluster.

[0080] S304: Based on the time attributes of each to-be-processed task in the task bucket, read the target task and execute the target task.

[0081] Specifically, when scheduling tasks, based on the time attributes of each pending task in the task bucket, the task with the most recently updated time attribute in the current task bucket is read as the target task and executed. As mentioned above, the execution method of the target task will not be repeated here.

[0082] S305: Start the OSD water level monitoring thread.

[0083] Specifically, when the water level filling thread is started, a listening thread may also be started, wherein the listening thread is used to monitor the water level of the OSD. In the object storage cluster, the water level is the usage rate of the storage space.

[0084] Figure 4 This is a structural diagram of a data filling device provided by the present disclosure. Figure 4 As shown, the device 400 includes: a writing module 410 , a reading module 420 , a building module 430 and a filling module 440 .

[0085] A writing module 410 is configured to write the pending task into the task bucket of the object storage cluster; A reading module 420 is configured to read a target task based on the time attributes of each to-be-processed task in the task bucket; A construction module 430 is configured to construct a target object and write the target object into the OSD when the target task is a water level filling task; The filling module 440 is configured to complete water level filling when the water level of the OSD meets a preset first threshold.

[0086] Optionally, the building block is used to: Determining the data volume of the target object based on the task parameters of the water level filling task; constructing the target object based on the task parameters and the data volume; Start a water level filling thread; wherein, the water level filling thread is used to construct a water level filling request for the target object and submit the water level filling request to the OSD message queue, and the OSD message queue is read and executed by the internal thread of the OSD.

[0087] Optionally, the device further comprises: A monitoring thread is started, where the monitoring thread is used to monitor the water level of the OSD.

[0088] Optionally, the device further comprises: When the water level of the OSD meets a preset second threshold, the actual water level of the target task is updated; wherein the second threshold is one or more.

[0089] Optionally, the device further comprises: Update the task status of each of the pending tasks.

[0090] Optionally, the device further comprises: Update the task status of the currently executed target task to being processed; When the target task is completed, the task status of the target task is updated to completed.

[0091] Optionally, the device further comprises: When the target task is a pending task, stop filling the water level; Update the task status of the target task to suspended.

[0092] Optionally, the device further comprises: When the target task is a continued task for a suspended task, updating the task status of the target task to being processed; Continue to execute the target task.

[0093] Optionally, the device further comprises: When the target task is a deletion task, stop filling the water level and update the task status of the target task to be deleted; A delete request is constructed and submitted to the OSD message queue, which is read and executed by the OSD's internal thread.

[0094] Optionally, the device further comprises: When the data filling device resumes normal operation, checking whether there are any unfinished tasks in progress in the task bucket; If there are unfinished tasks being processed, continue to execute the tasks being processed.

[0095] Optionally, the device further comprises: Before writing the pending task into the task bucket of the object storage cluster, obtaining task parameters and a water level filling command based on the data filling device; The task to be processed is constructed based on the task parameters and the water level filling command.

[0096] The present application also provides an electronic device to perform the above data filling method. Figure 5 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 5 As shown, the electronic device 5 includes: a processor 500, a memory 501, a bus 502 and a communication interface 503, and the processor 500, the communication interface 503 and the memory 501 are connected via the bus 502; the memory 501 stores a computer program that can be run on the processor 500, and when the processor 500 runs the computer program, it executes the data filling method provided in any of the aforementioned embodiments of the present application.

[0097] Memory 501 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between the device network element and at least one other network element is achieved through at least one communication interface 503 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0098] The bus 502 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 501 is used to store programs. The processor 500 executes the programs after receiving execution instructions. The data filling method disclosed in any of the aforementioned embodiments of the present application may be applied to the processor 500 or implemented by the processor 500.

[0099] The processor 500 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 500 or by software instructions. The above processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 501 , and the processor 500 reads the information in the memory 501 and completes the steps of the above method in combination with its hardware.

[0100] The electronic device provided in the embodiment of the present application and the data filling method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0101] An embodiment of the present application also provides a computer-readable storage medium corresponding to the data filling method provided in the aforementioned embodiment. The computer-readable storage medium shown therein may be a CD on which a computer program is stored. When the computer program is run by a processor, it will execute the data filling method provided in any of the aforementioned embodiments.

[0102] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0103] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the data filling method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0104] The present application also provides a computer program product 600. Figure 6 The computer program product carries a computer program 601, and the program code includes instructions that can be used to execute the steps of the data filling method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0105] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0106] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0107] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0108] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0109] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0110] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0111] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0112] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A data filling method, characterized in that: A data filling device applied to an object storage cluster, wherein the data filling device directly communicates with an object storage device (OSD); the method includes: Writing the pending tasks into the task bucket of the object storage cluster; Read the target task based on the time attribute of each pending task in the task bucket; When the target task is a water level filling task, construct a target object and write it into the OSD; When the water level of the OSD meets a preset first threshold, the water level filling is completed.

2. The method according to claim 1, characterized in that When the target task is a water level filling task, constructing a target object and writing it into the OSD includes: Determining the data volume of the target object based on the task parameters of the water level filling task; constructing the target object based on the task parameters and the data volume; Start a water level filling thread; wherein, the water level filling thread is used to construct a water level filling request for the target object and submit the water level filling request to the OSD message queue, and the OSD message queue is read and executed by the internal thread of the OSD.

3. The method according to claim 2, characterized in that The task parameters include at least one of the following: a target filling level, a filling object name, and a filling data size.

4. The method according to claim 1, wherein The method further comprises: A monitoring thread is started, where the monitoring thread is used to monitor the water level of the OSD.

5. The method according to claim 1, wherein The method further comprises: When the water level of the OSD meets a preset second threshold, the actual water level of the target task is updated; wherein the second threshold is one or more.

6. The method according to claim 1, characterized in that The task status of each pending task in the task bucket includes at least one of the following: processing, pending, completed, and pending deletion; the method further includes: Update the task status of each of the pending tasks.

7. The method according to claim 6, characterized in that The updating of the task status of each pending task includes: Update the task status of the currently executed target task to being processed; When the target task is completed, the task status of the target task is updated to completed.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When the target task is a pending task, stop filling the water level; Update the task status of the target task to suspended.

9. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When the target task is a continued task for a suspended task, updating the task status of the target task to being processed; Continue to execute the target task.

10. The method according to any one of claims 1 to 7, characterized in that The method further comprises: When the target task is a deletion task, stop filling the water level and update the task status of the target task to be deleted; A delete request is constructed and submitted to the OSD message queue, which is read and executed by the OSD's internal thread.

11. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: When the data filling device resumes normal operation, checking whether there are any unfinished tasks in progress in the task bucket; If there are unfinished tasks being processed, continue to execute the tasks being processed.

12. The method according to claim 1, characterized in that Before writing the pending task into the task bucket of the object storage cluster, the method further includes: Based on the data filling device, obtaining task parameters and water level filling commands; The task to be processed is constructed based on the task parameters and the water level filling command.

13. A data filling device, characterized in that: The data filling device communicates directly with the object storage device OSD, and the device includes: The writing module is used to write pending tasks into the task bucket of the object storage cluster; A reading module, configured to read a target task based on the time attributes of each pending task in the task bucket; A construction module, configured to construct a target object and write the target object into the OSD when the target task is a water level filling task; The filling module is used to complete water level filling when the water level of the OSD meets a preset first threshold.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the method according to any one of claims 1 to 12.

Images