Data processing method, device, electronic device and storage medium
Patent Information
- Application Number
- CN202311703935.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-12
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-12-12
AI Technical Summary
[0018] It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.
Smart Images

Figure CN117909090B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to data processing methods, devices, electronic devices and storage media in the fields of distributed storage and cloud computing. Background Art
[0002] Current distributed storage systems, such as high-performance distributed storage (Ceph) systems, usually design layered functions for different hardware media. For example, frequently accessed data is stored in a cache pool, which usually uses a solid-state drive (SSD) as the storage medium, and infrequently accessed data is flushed to a backend pool, which usually uses an ordinary serial port hard drive (SATA) as the storage medium. Summary of the invention
[0003] The present disclosure provides a data processing method, device, electronic device and storage medium.
[0004] A data processing method, comprising:
[0005] Determine the priority of each placement group in the cache pool, and assign the placement groups with the same priority to the same waiting queue;
[0006] Constructing an initially empty target queue, in response to determining that the supplement trigger condition is met, determining the placement group to be removed according to the principle that the placement group in the waiting queue with a higher priority is removed first, and taking the placement group to be removed from the corresponding waiting queue and adding it to the target queue;
[0007] And, in response to determining that the target queue is not empty, each placement group therein is traversed in a loop, wherein, when traversing to each placement group, the placement group is respectively used as the target placement group, and the number of objects that can be flushed is determined as the first number, and the first number of objects taken out from the target placement group is flushed to the backend pool.
[0008] A data processing device, comprising: a preprocessing module, a queue management module and a data download module;
[0009] The preprocessing module is used to determine the priority of each placement group in the buffer pool, and divide the placement groups with the same priority into the same waiting queue;
[0010] The queue management module is used to construct an initially empty target queue, and in response to determining that the supplement trigger condition is met, determine the placement group to be taken out according to the principle that the placement group in the waiting queue with a higher priority is taken out first, and take the placement group to be taken out from the corresponding waiting queue and add it to the target queue;
[0011] The data flushing module is used for, in response to determining that the target queue is not empty, cyclically traversing each placement group therein, wherein, when traversing to each placement group, the placement group is respectively used as a target placement group, and the number of objects that can be flushed is determined as a first number, and the first number of objects taken out from the target placement group is flushed to the backend pool.
[0012] An electronic device, comprising:
[0013] at least one processor; and
[0014] a memory communicatively connected to the at least one processor; wherein,
[0015] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.
[0016] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.
[0017] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the method described above when executed by a processor.
[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0020] Figure 1 A flowchart of an embodiment of the data processing method disclosed in the present invention;
[0021] Figure 2 It is a schematic diagram of the composition structure of the data processing device embodiment 200 described in the present disclosure;
[0022] Figure 3 A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0024] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0025] Figure 1 Flow chart of an embodiment of the data processing method disclosed in the present invention. Figure 1 As shown, the following specific implementation methods are included.
[0026] In step 101, the priorities of the placement groups (PGs) in the cache pool are determined respectively, and PGs with the same priority are divided into the same waiting queue.
[0027] In step 102, an initially empty target queue is constructed, and in response to determining that the supplement trigger condition is met, the PG to be taken out is determined according to the principle that the PG in the waiting queue with a higher priority is taken out first, and the PG to be taken out is taken out from the corresponding waiting queue and added to the target queue.
[0028] In step 103, in response to determining that the target queue is not empty, each PG therein is traversed in a loop, wherein when traversing to each PG, the PG is used as the target PG, and the number of objects that can be flushed is determined as the first number, and the first number of objects taken out from the target PG is flushed to the backend pool.
[0029] By adopting the scheme described in the above method embodiment, by setting the priority and target queue, etc., objects in the high-priority PG can be flushed to the backend pool first, thereby improving the flushing effect. Moreover, by cyclically traversing and flushing each PG in the target queue, the pressure balance of the disk in the backend pool can be improved.
[0030] A common problem in distributed storage systems is that using ordinary SATA disks for data storage usually does not provide adequate performance, while using SSD disks for long-term data storage is too expensive. Therefore, when users write data, they will first write it to the SSD disk and then transfer it to the SATA disk, thereby balancing performance and cost. Frequently accessed data can be stored in the cache pool, which usually uses SSD as the storage medium, and infrequently accessed data can be flushed to the backend pool, which usually uses SATA disks as the storage medium.
[0031] Preferably, in response to determining that the amount of data in the cache pool is greater than the first threshold, it can be determined that the cache pool meets the flush trigger condition, and the following processing can be performed for each PG in the cache pool: the amount of data in the PG is compared with the reference data amount corresponding to the PG, the reference data amount is the data amount threshold corresponding to the PG determined according to the first threshold, and the priority of the PG is determined according to the comparison result, wherein the larger the ratio of the amount of data in the PG to the reference data amount, the higher the priority of the PG. The specific value of the first threshold can be determined according to actual needs.
[0032] The cache pool may include multiple object storage devices (OSDs), each OSD may include multiple PGs, and each PG may include multiple objects.
[0033] A first threshold value can be set for the cache pool, and the specific value can be determined according to actual needs. When it is determined that the amount of data in the cache pool is greater than the first threshold value, it can be determined that the cache pool meets the refresh trigger condition, and the priority of each PG in the cache pool can be determined separately. Among them, for any PG, the amount of data in the PG can be compared with the reference data amount corresponding to the PG. For example, the ratio of the amount of data in the PG to the reference data amount corresponding to the PG can be calculated. The larger the ratio, the higher the priority. In other words, the closer the amount of data in the PG is to the reference data amount corresponding to the PG, or the more it exceeds, the higher the priority will be.
[0034] There is no restriction on how to determine the reference data volume corresponding to different PGs. For example, assuming that the cache pool includes 10 OSDs, each OSD includes 100 PGs, and the value of the first threshold is 100G, then 100G can be divided by 1000, and the result can be used as the reference data volume corresponding to each PG.
[0035] Correspondingly, different PGs can be divided into corresponding waiting queues according to their priorities. For example, assuming that there are 3 priorities in total, the PG with priority 1 can be divided into the first waiting queue, the PG with priority 2 can be divided into the second waiting queue, and the PG with priority 3 can be divided into the third waiting queue. The specific number of priorities can be determined according to actual needs.
[0036] Through the above processing, different PGs can be efficiently and accurately divided into corresponding waiting queues, thus laying a good foundation for subsequent processing.
[0037] In addition, an initially empty target queue can be constructed. Each time it is determined that the supplement trigger condition is met, the PG to be taken out can be determined from the PGs that have not been taken out in each waiting queue according to the principle that the PG in the waiting queue with a higher priority will be taken out first, and the PG to be taken out can be taken out from the corresponding waiting queue and added to the target queue.
[0038] Preferably, there are at least the following two implementation methods.
[0039] 1) In response to determining that there are no unprocessed PGs in the target queue, it is determined that the supplementary trigger condition is met, and the processing is completed including: all objects therein have been taken out, and the processed PGs will be deleted from the target queue. Accordingly, each PG in the waiting queue where the PG has not been taken out and the waiting queue with the highest priority can be used as the PG to be taken out, and the PG to be taken out can be taken out from the corresponding waiting queue and added to the target queue.
[0040] For example, assuming that there are three waiting queues, they are the first waiting queue, the second waiting queue and the third waiting queue in order of priority from high to low. Since the initially constructed target queue is empty, it can be determined that the supplementary trigger condition is met, that is, it is determined that there are no unprocessed PGs in the target queue. Accordingly, each PG in the first waiting queue can be taken out and added to the target queue. Afterwards, when each PG in the target queue has been processed, it can be determined again that the supplementary trigger condition is met, and each PG in the second waiting queue can be taken out and added to the target queue. Afterwards, when each PG in the target queue has been processed, it can be determined again that the supplementary trigger condition is met, and each PG in the third waiting queue can be taken out and added to the target queue.
[0041] In other words, the PGs in each waiting queue can be processed in order from high to low priority, which is very simple and convenient in implementation.
[0042] 2) In response to determining that the number of unprocessed PGs in the target queue is less than the second threshold, it can be determined that the supplementary trigger condition is met, and the processing completion includes: all objects therein have been taken out, and the processed PGs will be deleted from the target queue. Accordingly, in response to determining that the supplementary trigger condition is met for the first time, each PG in the waiting queue with the highest priority can be used as a PG to be taken out. In response to determining that it is not the first time that the supplementary trigger condition is met, the number of PGs to be taken out can be determined as the second number, and the second number of PGs determined from the untaken PGs in each waiting queue according to the principle can be used as the PGs to be taken out. Further, the PGs to be taken out can be taken out from the corresponding waiting queue and added to the target queue. The specific value of the second threshold can be determined according to actual needs.
[0043] In addition, preferably, determining the number of PGs to be taken out as the second number may include: obtaining the difference between the second threshold and the number of unprocessed PGs in the target queue, and using the difference as the second number.
[0044] For example, assuming that there are three waiting queues, they are the first waiting queue, the second waiting queue and the third waiting queue in order of priority from high to low. Since the initially constructed target queue is empty, it can be determined that the supplementary trigger condition is met, that is, it is determined that the number of unprocessed PGs in the target queue is less than the second threshold. Accordingly, each PG in the first waiting queue can be taken out and added to the target queue. Afterwards, when the number of unprocessed PGs in the target queue is less than the second threshold, it can be determined again that the supplementary trigger condition is met, and the difference between the second threshold and the number of unprocessed PGs in the target queue can be obtained as the second number. Accordingly, the second number of PGs can be taken out from the PGs that have not been taken out in the second waiting queue and added to the target queue. If the number of PGs that have not been taken out in the second waiting queue is less than the second number, all the PGs that have not been taken out in the second waiting queue can be taken out, and then some PGs can be taken out from the third waiting queue, so that the sum of the number of PGs taken out is the second number, and then the second number of PGs taken out can be added to the target queue, and so on.
[0045] There is no restriction on the order in which the PGs in each waiting queue are sorted.
[0046] In actual applications, the disks in the backend pool corresponding to each PG are usually fixed. If one or several PGs are processed all the time, it is equivalent to writing to the corresponding one or several disks all the time, which causes the problem of uneven disk pressure. After adopting the processing method described in the present disclosure, when the number of unprocessed PGs in the target queue is less than the second threshold, the target queue can be supplemented, so that the number of PGs in the target queue is maintained at least at the second threshold, thereby avoiding the problem of uneven disk pressure caused by too few PGs in the target queue as much as possible.
[0047] For the target queue, as long as it is not empty, each PG therein can be traversed in a loop. When traversing each PG, the PG can be used as the target PG, and the number of objects that can be flushed can be determined as the first number, and then the first number of objects taken out from the target PG can be flushed to the backend pool.
[0048] Preferably, the method of taking out the first number of objects from the target PG may include: calling a list thread, using the list thread to take out the first number of objects from the target PG, and adding them to the flush queue as objects to be flushed; the method of flushing them to the backend pool may include: in response to determining that the flush queue is not empty, flushing each object to be processed in the flush queue to the backend pool in the order of the time of joining the flush queue from first to last.
[0049] In the traditional method, the PGs in each waiting queue are usually processed in order from high to low priority, and for each waiting queue, the next PG will be processed only after the current PG is processed. Accordingly, the list thread is frequently used to obtain objects from the current PG for flushing operations, which leads to excessive pressure on the disk in the backend pool corresponding to the current PG, increasing the problem of uneven disk pressure. After adopting the processing method described in the present disclosure, each PG in the target queue can be traversed in a loop, and a certain number of objects can be taken from each traversed PG and flushed to the backend pool, thereby improving the pressure balance of the disk in the backend pool.
[0050] Preferably, the number of unprocessed objects in the flush queue that have not been flushed can be monitored, and in response to determining that the number of unprocessed objects that have not been flushed is less than or equal to a third threshold value according to the monitoring result, the next PG can be traversed, and the third threshold value is less than a fourth threshold value, and the fourth threshold value is the maximum number of unprocessed objects allowed to be included in the flush queue. The specific values of the third threshold value and the fourth threshold value can be determined according to actual needs, for example, the fourth threshold value can be 20, and the third threshold value can be 5.
[0051] That is to say, once it is determined that the number of pending objects that have not been flushed in the flush queue is less than or equal to the third threshold, the next PG can be traversed, and accordingly, the number of objects that can be flushed can be determined as the first number, and the first number of objects can be taken out from the objects that have not been taken out in the traversed PG and added to the flush queue.
[0052] Preferably, a difference between the fourth threshold and the number of unprocessed objects in the refresh queue that have not been refreshed can be obtained, and the difference is used as the first number.
[0053] For example, assuming that the value of the fourth threshold is 20, and the number of unprocessed objects in the flush queue is 4, then the difference between the two, that is, 16, can be obtained, and 16 can be used as the first number determined, and then the list thread can be used to take out 16 objects from the target PG and add them to the flush queue.
[0054] It can be seen that by adopting the above processing method, the list thread can be used to take out objects from the target PG and add them to the flush queue without waiting until the flush queue is empty, thereby realizing the concurrent execution of the list thread and the flush operation, thereby improving the processing efficiency.
[0055] In addition, preferably, when the list thread is used to take out the first number of objects from the target PG, the list thread can be used to take out the top M objects from the objects that have not been taken out in the target PG, and the objects in the target PG are arranged in a predetermined order (the sorting method is not limited), M is a positive integer greater than or equal to the first number, and the M objects taken out need to meet the following conditions: after filtering out the objects whose time difference between the write time and the current time is less than the fifth threshold from the M objects, the number of remaining objects is equal to the first number. The write time refers to the time of writing to the cache pool.
[0056] The specific value of the fifth threshold may be determined according to actual needs, such as 600 seconds.
[0057] Assuming that the first number is 16 and the number of objects that have not been taken out in the target PG is 35, then if it is assumed that the objects ranked in the top 16 do not include objects whose time difference between the write time and the current time is less than the fifth threshold, the objects ranked in the top 16 can be directly taken as the first number of objects to be taken out and added to the flush queue. Assuming that the objects ranked in the top 16 include 2 objects whose time difference between the write time and the current time is less than the fifth threshold, and assuming that the objects ranked in the top 18 also only include these 2 objects whose time difference between the write time and the current time is less than the fifth threshold, then the objects ranked in the top 18 can be taken out, and the 2 objects whose time difference between the write time and the current time is less than the fifth threshold can be filtered out, and the remaining 16 objects can be taken as the first number of objects to be taken out and added to the flush queue.
[0058] It can be seen that through the above processing, objects whose time difference between the write time and the current time is less than the fifth threshold can be retained in the cache pool instead of being flushed to the backend pool. Objects with shorter write times are usually more likely to be accessed again. Therefore, retaining such objects in the cache pool can improve data access efficiency, etc.
[0059] In addition, preferably, in response to determining that the M objects cannot be retrieved, all objects that have not been retrieved in the target PG can be retrieved, and objects whose time difference between the write time and the current time is less than a fifth threshold can be filtered out, and the remaining objects can be added to the flush queue as objects to be flushed.
[0060] Assuming that the first number is 16 and the number of objects that have not been taken out in the target PG is 10, then all these 10 objects can be taken out, and the objects whose time difference between the write time and the current time is less than the fifth threshold can be filtered out, and the remaining objects can be added to the flush queue as objects to be flushed.
[0061] Furthermore, preferably, after the list thread is used to take out the object from the target PG, the list thread can be controlled to enter a sleep state, and the lock corresponding to the target PG occupied by the list thread when taking out the object from the target PG can be released.
[0062] When the list thread retrieves an object from the target PG, it needs to occupy the lock corresponding to the target PG. As mentioned above, in the traditional method, for each waiting queue, the next PG will be processed only after the current PG is processed. Accordingly, the list thread is frequently used to obtain objects from the current PG for flushing operations, which will cause the lock corresponding to the current PG to be occupied by the list thread for a long time, and thus the user cannot perform normal read and write operations on the current PG.
[0063] After adopting the processing method disclosed in the present invention, a loop traversal method is used to perform object extraction processing on each PG in the target queue, and the lock occupied by the list thread is released in time after each object is extracted, thereby avoiding the problem of the lock corresponding to the same PG being occupied for a long time, and thus allowing enough time to be used for ordinary read and write operations, thereby avoiding as much as possible the problem of slow requests (block ops) caused by the failure to process read and write requests in a timely manner.
[0064] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.
[0065] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.
[0066] Figure 2 FIG. 2 is a schematic diagram of the structure of the data processing device embodiment 200 described in the present disclosure. Figure 2 As shown, it includes: a pre-processing module 201, a queue management module 202 and a data flushing module 203.
[0067] The pre-processing module 201 is used to determine the priority of each PG in the cache pool respectively, and divide the PGs with the same priority into the same waiting queue.
[0068] The queue management module 202 is used to construct an initially empty target queue. In response to determining that the supplement trigger condition is met, the PG to be taken out is determined according to the principle that the PG in the waiting queue with higher priority is taken out first, and the PG to be taken out is taken out from the corresponding waiting queue and added to the target queue.
[0069] The data flush module 203 is used to, in response to determining that the target queue is not empty, cyclically traverse each PG therein, wherein, when traversing each PG, the PG is respectively used as the target PG, and the number of objects that can be flushed is determined as the first number, and the first number of objects taken out from the target PG is flushed to the backend pool.
[0070] By adopting the scheme described in the above-mentioned device embodiment, by setting the priority and target queue, etc., objects in the high-priority PG can be flushed to the backend pool first, thereby improving the flushing effect. Moreover, by cyclically traversing and flushing each PG in the target queue, the pressure balance of the disk in the backend pool can be improved.
[0071] Preferably, in response to determining that the amount of data in the cache pool is greater than the first threshold, the preprocessing module 201 can determine that the cache pool meets the refresh trigger condition, and can perform the following processing for each PG in the cache pool: compare the amount of data in the PG with a reference data amount corresponding to the PG, the reference data amount being the data amount threshold corresponding to the PG determined according to the first threshold, and determine the priority of the PG based on the comparison result, wherein the larger the ratio of the amount of data in the PG to the reference data amount, the higher the priority of the PG.
[0072] In addition, the queue management module 202 can construct an initially empty target queue. Each time it is determined that the supplement trigger condition is met, the PG to be taken out can be determined from the PGs that have not been taken out in each waiting queue according to the principle that the PG in the waiting queue with a higher priority will be taken out first, and the PG to be taken out can be taken out from the corresponding waiting queue and added to the target queue.
[0073] Preferably, the queue management module 202 can determine that the supplement trigger condition is met in response to determining that there is no unprocessed PG in the target queue, and the processing completion includes: all objects therein have been taken out, and the processed PG will be deleted from the target queue. Accordingly, each PG in the waiting queue where the PG has not been taken out and the waiting queue with the highest priority can be used as the PG to be taken out, and the PG to be taken out can be taken out from the corresponding waiting queue and added to the target queue.
[0074] Alternatively, preferably, the queue management module 202 can determine that the supplement trigger condition is met in response to determining that the number of unprocessed PGs in the target queue is less than a second threshold, and the processing completion includes: all objects therein have been taken out, and accordingly, in response to determining that the supplement trigger condition is met for the first time, each PG in the waiting queue with the highest priority can be used as a PG to be taken out, and in response to determining that it is not the first time that the supplement trigger condition is met, the number of PGs to be taken out can be determined as a second number, and the second number of PGs determined according to the principle from the PGs that have not been taken out in each waiting queue can be used as the PGs to be taken out. Furthermore, the PGs to be taken out can be taken out from the corresponding waiting queues and added to the target queue.
[0075] In addition, preferably, the queue management module 202 determines the number of PGs to be taken out as the second number, which may include: obtaining the difference between the second threshold and the number of unprocessed PGs in the target queue, and using the difference as the second number.
[0076] For the target queue, as long as it is not empty, the data flush module 203 can loop through each PG therein. When traversing each PG, the PG can be used as the target PG respectively, and the number of objects that can be flushed can be determined as the first number, and then the first number of objects taken out from the target PG can be flushed to the backend pool.
[0077] Preferably, the method in which the data flush module 203 takes out the first number of objects from the target PG may include: calling the list thread, using the list thread to take out the first number of objects from the target PG, and adding them to the flush queue as objects to be flushed; the method in which they are flushed to the backend pool may include: in response to determining that the flush queue is not empty, flushing each object to be processed in the flush queue to the backend pool in the order of the time of joining the flush queue from first to last.
[0078] Preferably, the data flush module 203 can monitor the number of unprocessed objects that have not been flushed in the flush queue. In response to determining according to the monitoring results that the number of unprocessed objects that have not been flushed is less than or equal to a third threshold, the next PG can be traversed, and the third threshold is less than the fourth threshold, and the fourth threshold is the maximum number of unprocessed objects allowed to be included in the flush queue.
[0079] Preferably, the data flushing module 203 may obtain a difference between the fourth threshold and the number of to-be-processed objects in the flushing queue that have not been flushed, and use the difference as the first number.
[0080] In addition, preferably, when the data flush module 203 uses the list thread to take out the first number of objects from the target PG, it can use the list thread to take out the first M objects from the objects that have not been taken out in the target PG, and the objects in the target PG are arranged in a predetermined order, M is a positive integer greater than or equal to the first number, and the M objects taken out need to meet the following conditions: after filtering out the objects whose time difference between the write time and the current time is less than the fifth threshold from the M objects, the number of remaining objects is equal to the first number. Preferably, in response to determining that the M objects cannot be taken out, the data flush module 203 can take out all the objects that have not been taken out in the target PG, and can filter out the objects whose time difference between the write time and the current time is less than the fifth threshold, and add the remaining objects as objects to be flushed to the flush queue.
[0081] In addition, preferably, after the data flushing module 203 uses the list thread to take out the object from the target PG, it can control the list thread to enter a sleep state and release the lock corresponding to the target PG occupied by the list thread when taking out the object from the target PG.
[0082] Figure 2The specific working process of the illustrated device embodiment can refer to the relevant description in the aforementioned method embodiment and will not be described in detail.
[0083] The solution disclosed in the present disclosure can be applied to the field of artificial intelligence, especially to the fields of distributed storage and cloud computing. Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0084] In addition, in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0085] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0086] Figure 3 A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0087] like Figure 3 As shown, the device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0088] A number of components in the device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0089] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the method described in the present disclosure may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the method described in the present disclosure in any other appropriate manner (e.g., by means of firmware).
[0090] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0091] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0092] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0094] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0095] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0096] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0097] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A data processing method, comprising: Determining the priority of each placement group in the buffer pool respectively, including: in response to determining that the amount of data in the buffer pool is greater than a first threshold, determining that the buffer pool meets the down-flush trigger condition, and performing the following processing for each placement group in the buffer pool: comparing the amount of data in the placement group with a reference amount of data corresponding to the placement group, the reference amount of data being a data amount threshold corresponding to the placement group determined according to the first threshold, and determining the priority of the placement group according to the comparison result, wherein the larger the ratio of the amount of data in the placement group to the reference amount of data, the higher the priority of the placement group; and placing the placement groups with the same priority into the same waiting queue; Constructing an initially empty target queue, in response to determining that the supplement trigger condition is met, determining the placement group to be removed according to the principle that the placement group in the waiting queue with a higher priority is removed first, and taking the placement group to be removed from the corresponding waiting queue and adding it to the target queue; And, in response to determining that the target queue is not empty, each placement group therein is traversed in a loop, wherein, when traversing to each placement group, the placement group is respectively used as the target placement group, and the number of objects that can be flushed is determined as the first number, and the first number of objects taken out from the target placement group is flushed to the backend pool.
2. The method according to claim 1, wherein: The response to determining that the supplement trigger condition is met includes: in response to determining that there is no unprocessed placement group in the target queue, determining that the supplement trigger condition is met, and the processing is completed includes: all objects therein have been taken out, and the processed placement group will be deleted from the target queue; The determining of the placement groups to be taken out includes: taking each placement group in the waiting queue where the placement groups have not been taken out and in the waiting queue with the highest priority as the placement groups to be taken out.
3. The method according to claim 1, wherein: The response to determining that the supplement trigger condition is met includes: in response to determining that the number of unprocessed placement groups in the target queue is less than a second threshold, determining that the supplement trigger condition is met, and the processing is completed includes: all objects therein have been taken out, and the processed placement groups will be deleted from the target queue; The determining of the placement groups to be removed includes: in response to determining that the supplementary trigger condition is met for the first time, taking each placement group in the waiting queue with the highest priority as the placement group to be removed; in response to determining that the supplementary trigger condition is not met for the first time, determining the number of the placement groups to be removed as a second number, and taking the second number of placement groups determined according to the principle from the placement groups that have not been taken out in the waiting queues as the placement groups to be removed.
4. The method according to claim 3, wherein: The determining the number of the to-be-removed placement groups as the second number includes: obtaining a difference between the second threshold and the number of unprocessed placement groups in the target queue, and using the difference as the second number.
5. The method according to any one of claims 1 to 4, wherein: Taking out the first number of objects from the target placement group includes: calling an enumeration thread, taking out the first number of objects from the target placement group by using the enumeration thread, and adding the objects to be flushed into a flush queue as objects to be flushed; The flushing to the backend pool includes: in response to determining that the flush queue is not empty, flushing each to-be-processed object in the flush queue to the backend pool in order of time of joining the flush queue.
6. The method according to claim 5, wherein: The looping through each placement group includes: The number of pending objects that have not been flushed in the flush queue is monitored, and in response to determining according to the monitoring result that the number of pending objects that have not been flushed is less than or equal to a third threshold, traverse the next placement group, the third threshold is less than a fourth threshold, and the fourth threshold is the maximum number of pending objects allowed to be included in the flush queue.
7. The method according to claim 6, wherein: The determining the number of objects that can be swiped down as a first number includes: A difference between the fourth threshold and the number of objects to be processed that have not been refreshed is obtained, and the difference is used as the first number.
8. The method according to claim 5, wherein: The taking out the first number of objects from the target placement group by using the enumeration thread comprises: taking out the top M objects from the objects that have not been taken out in the target placement group by using the enumeration thread, the objects in the target placement group are arranged in a predetermined order, M is a positive integer greater than or equal to the first number, and the taken out M objects need to meet the following conditions: after filtering out objects whose time difference between the write time and the current time is less than a fifth threshold from the M objects, the number of remaining objects is equal to the first number; It also includes: in response to determining that the M objects cannot be taken out, all objects that have not been taken out in the target placement group are taken out, and objects whose time difference between the write time and the current time is less than the fifth threshold are filtered out, and the remaining objects are added to the flush queue as the objects to be flushed.
9. The method according to claim 8, further comprising: After the enumeration thread is used to take out the object from the target placement group, the enumeration thread is controlled to enter a sleep state, and a lock corresponding to the target placement group occupied by the enumeration thread when taking out the object from the target placement group is released.
10. A data processing device, comprising: Preprocessing module, queue management module and data flushing module; The preprocessing module is used to respectively determine the priority of each placement group in the cache pool, including: in response to determining that the amount of data in the cache pool is greater than a first threshold, determining that the cache pool meets the down-flush trigger condition, and performing the following processing for each placement group in the cache pool: comparing the amount of data in the placement group with a reference amount of data corresponding to the placement group, the reference amount of data being a data amount threshold corresponding to the placement group determined according to the first threshold, and determining the priority of the placement group according to the comparison result, wherein the larger the ratio of the amount of data in the placement group to the reference amount of data, the higher the priority of the placement group; and placing the placement groups with the same priority into the same waiting queue; The queue management module is used to construct an initially empty target queue, and in response to determining that the supplement trigger condition is met, determine the placement group to be taken out according to the principle that the placement group in the waiting queue with a higher priority is taken out first, and take the placement group to be taken out from the corresponding waiting queue and add it to the target queue; The data flushing module is used for, in response to determining that the target queue is not empty, cyclically traversing each placement group therein, wherein, when traversing to each placement group, the placement group is respectively used as a target placement group, and the number of objects that can be flushed is determined as a first number, and the first number of objects taken out from the target placement group is flushed to the backend pool.
11. The device according to claim 10, wherein: The queue management module determines that the supplement trigger condition is met in response to determining that there is no unprocessed placement group in the target queue, wherein the processing is completed including that all objects therein have been taken out, and the processed placement group will be deleted from the target queue; The queue management module uses each placement group in the waiting queue whose placement group has not been taken out and in the waiting queue with the highest priority as the placement group to be taken out.
12. The device according to claim 10, wherein: The queue management module determines that the supplement trigger condition is met in response to determining that the number of unprocessed placement groups in the target queue is less than a second threshold, wherein the processing is completed including: all objects therein have been taken out, and the processed placement groups will be deleted from the target queue; In response to determining that the supplementary trigger condition is met for the first time, the queue management module uses each placement group in the waiting queue with the highest priority as the placement groups to be removed; in response to determining that the supplementary trigger condition is not met for the first time, the queue management module determines the number of the placement groups to be removed as the second number, and uses the second number of placement groups determined according to the principle from the placement groups that have not been removed in the waiting queues as the placement groups to be removed.
13. The device according to claim 12, wherein: The queue management module obtains a difference between the second threshold and the number of unprocessed placement groups in the target queue, and uses the difference as the second number.
14. The device according to any one of claims 10 to 13, wherein: The data flushing module calls the enumeration thread, uses the enumeration thread to take out the first number of objects from the target placement group, and adds the objects to be flushed into the flushing queue as objects to be flushed; In response to determining that the flush queue is not empty, the data flush module flushes each to-be-processed object in the flush queue to the backend pool in the order of the time of joining the flush queue.
15. The device according to claim 14, wherein: The data flushing module monitors the number of pending objects that have not been flushed in the flushing queue, and in response to determining that the number of pending objects that have not been flushed is less than or equal to a third threshold according to the monitoring result, traverses the next placement group, the third threshold is less than a fourth threshold, and the fourth threshold is the maximum number of pending objects allowed to be included in the flushing queue.
16. The device according to claim 15, wherein: The data flushing module obtains a difference between the fourth threshold and the number of unflushed objects to be processed, and uses the difference as the first number.
17. The device according to claim 14, wherein: The data flushing module uses the enumeration thread to extract the top M objects from the objects that have not been taken out in the target placement group, the objects in the target placement group are arranged in a predetermined order, M is a positive integer greater than or equal to the first number, and the M objects taken out need to meet the following conditions: after filtering out objects whose time difference between the write time and the current time is less than a fifth threshold from the M objects, the number of remaining objects is equal to the first number; The data flushing module is further used to, in response to determining that the M objects cannot be taken out, take out all the objects that have not been taken out in the target placement group, and filter out the objects whose time difference between the write time and the current time is less than the fifth threshold, and add the remaining objects as the objects to be flushed to the flushing queue.
18. The device according to claim 17, wherein: The data flushing module is further used to control the enumeration thread to enter a sleep state after taking out the object from the target placement group using the enumeration thread, and release the lock corresponding to the target placement group occupied by the enumeration thread when taking out the object from the target placement group.
19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.
20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 9.
21. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Data recovery method and device, electronic equipment and storage medium
CN113553216A
Rapid capacity balancing method, system and device and storage medium
CN114546265A
Data object down-refreshing method and device, equipment and medium
CN115113824A
Data storage method and apparatus, and related device
WO2023070935A1
