Object storage service batch processing task processing method and device, electronic equipment, medium and product
By splitting and prioritizing the batch task of the bucket list of object storage services, the problem of uneven resource occupancy in batch tasks for different data volumes is solved, and the fairness and efficiency of task distribution is achieved, ensuring that each Bucket list batch tasks can have sufficient resources.
Patent Information
- Application Number
- CN202411155249.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-21
AI Technical Summary
In batch processing tasks, buckets with larger data volumes occupy a large amount of resources, making it difficult to guarantee the timeliness and fairness of bucket list services for buckets with smaller data volumes.
By splitting each bucket inventory batch task, dividing it into multiple batch subtasks according to the preset data volume, and allocating and executing these subtasks in sequence according to the order of the bucket and execution priority, ensuring that each bucket inventory is in sequence. All batch tasks can obtain execution resources.
The fairness and efficiency of task distribution strategies between object buckets is realized, ensuring that each bucket list batch task has sufficient resources, and improving the efficiency and fairness of bucket list services between buckets.
Smart Images

Figure CN119025243A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a method, device, electronic device, medium, and product for processing batch processing tasks of an object storage service. Background Art
[0002] The bucket list function of object storage can process the data in the bucket according to the bucket list rules corresponding to the bucket to generate a list file and store it in the specified bucket, helping users to list massive data.
[0003] However, when batch processing the object data of a bucket, the next batch of tasks will not start until one batch of tasks is completed. The amount of data to be processed in the batch processing tasks corresponding to the bucket list rules of different buckets is different. The bucket list service provided for buckets with large data volumes will occupy a large amount of task execution resources. For buckets with relatively small data volumes, it is difficult to ensure the timeliness and fairness of the bucket list service provided to them. Summary of the invention
[0004] The present disclosure provides a method, device, electronic device, medium and product for processing batch processing tasks of an object storage service. By splitting and limiting the bucket list batch processing tasks corresponding to a single object storage bucket, a fair task distribution strategy is implemented for object storage buckets of different data sizes, ensuring that each bucket list batch processing task has execution resources, thereby achieving efficiency and fairness among object storage buckets.
[0005] In a first aspect, an embodiment of the present disclosure provides a method for processing a batch task of an object storage service, the method comprising:
[0006] Obtain the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage, wherein each bucket list rule corresponds to a bucket list batch processing task, and the processing object of each bucket list batch processing task includes multiple object data;
[0007] For each bucket list batch processing task, split the task according to the preset data volume to obtain multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task;
[0008] According to the bucket order of the storage buckets corresponding to each bucket list batch task and according to the execution priority order, multiple batch subtasks corresponding to each bucket list batch task are selected in turn, and the allocation and execution of the selected batch subtasks are triggered.
[0009] In a second aspect, an embodiment of the present disclosure further provides a processing device for a batch processing task of an object storage service, the device comprising:
[0010] A task acquisition module, used to acquire a bucket list batch processing task corresponding to each bucket list rule through a bucket list service of an object storage, wherein each bucket list rule corresponds to a bucket list batch processing task, and a processing object of each bucket list batch processing task includes multiple object data;
[0011] A task splitting module is used to split each bucket list batch processing task according to a preset data volume to obtain a plurality of batch processing subtasks with execution priority order corresponding to each bucket list batch processing task;
[0012] The task allocation and processing module is used to select multiple batch processing subtasks corresponding to each bucket list batch processing task in sequence according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and in the order of execution priority, and trigger the allocation and execution of the selected batch processing subtasks.
[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:
[0014] one or more processors;
[0015] a storage device for storing one or more programs,
[0016] When one or more programs are executed by one or more processors, the one or more processors implement a method for processing a batch task of an object storage service as described in any one of the embodiments of the present disclosure.
[0017] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute a processing method for an object storage service batch processing task as described in any of the embodiments of the present disclosure.
[0018] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, implements a processing method for a batch processing task of an object storage service as described in any one of the embodiments of the present invention.
[0019] In the disclosed embodiment, a bucket list batch processing task corresponding to each bucket list rule is obtained through a bucket list service of an object storage, wherein each bucket list rule corresponds to a bucket list batch processing task, and the processing object of each bucket list batch processing task includes multiple object data; for each bucket list batch processing task, the task is split according to the preset data volume, and multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task are obtained; according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and the execution priority order, multiple batch processing subtasks corresponding to each bucket list batch processing task are selected in sequence, and the allocation and execution of the selected batch processing subtasks are triggered. The technical solution of the disclosed embodiment solves the problem that the fairness and efficiency between the bucket list batch processing tasks corresponding to the bucket list rules of each object storage bucket cannot be taken into account in the bucket list service. By splitting and limiting the bucket list batch processing tasks corresponding to a single bucket list rule, a fair task distribution and processing strategy is implemented for storage buckets of different orders of magnitude, ensuring that each bucket list batch processing task can have execution resources, and achieving efficiency and fairness between the bucket list batch processing tasks corresponding to the bucket list rules. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0021] Figure 1 It is a flowchart of a method for processing a batch processing task of an object storage service provided by an embodiment of the present disclosure;
[0022] Figure 2 It is a flowchart of a method for processing a batch processing task of an object storage service provided by an embodiment of the present disclosure;
[0023] Figure 3 It is a flowchart of a method for processing a batch processing task of an object storage service provided by an embodiment of the present disclosure;
[0024] Figure 4 It is a flowchart of a method for processing a batch processing task of an object storage service provided by an embodiment of the present disclosure;
[0025] Figure 5 It is a schematic diagram of an implementation process of an application example of a processing method for a batch processing task of an object storage service provided by an embodiment of the present disclosure; Figure 6 This is a schematic diagram of task splitting in an application example provided by an embodiment of the present disclosure;
[0026] Figure 7This is a schematic diagram of multi-round distribution of tasks in an application example provided by an embodiment of the present disclosure;
[0027] Figure 8 It is a structural schematic diagram of a processing device for a batch processing task of an object storage service provided by an embodiment of the present disclosure;
[0028] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0031] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0032] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0034] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0035] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
[0036] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0037] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0038] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0039] Figure 1 A flow chart of a method for processing a batch task of an object storage service provided in an embodiment of the present disclosure is applicable to a scenario in which data processing tasks corresponding to a storage bucket are scheduled and managed in an object storage. The method can be executed by a processing device for a batch task of an object storage service, which can be implemented in the form of software and / or hardware. Optionally, the processing device for a batch task of an object storage service can be implemented by an electronic device, which can be a mobile terminal, a PC, a server, etc.
[0040] like Figure 1 As shown, the method for processing a batch task of the object storage service includes:
[0041] S110 . Obtain the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage.
[0042] The bucket list batch processing task may be a task corresponding to a bucket list rule of an object data storage bucket. Each bucket list rule corresponds to a bucket list batch processing task, and the processing object of each bucket list batch processing task includes multiple object data.
[0043] In the field of object storage, file data is stored in a key-value pair format, where the file name is the key and the file data is the value. This key-value pair is called an object. A bucket is a container for storing object data. All object data must belong to a bucket, which can be considered a namespace.
[0044] Each bucket can be customized by the user to configure one or more bucket list rules. Each bucket list rule can correspond to a bucket list batch task. In the bucket list rule, you can define whether each bucket list batch task is executed on a daily or weekly basis; you can define the range of object data to be listed, such as object data with a specified prefix, or only list specific metadata information of the object, such as object size, storage type, last updated date information, etc. When the object data in the bucket meets the conditions such as the specified prefix, these object data can be filtered out, a list can be generated, and stored in the specified bucket.
[0045] Bucket list service is a service provided by the object storage server to process object data in the bucket based on bucket list rules. The object storage server can provide operation interfaces such as object upload, deletion, and storage type conversion to implement specific services.
[0046] The bucket list batch processing tasks corresponding to each bucket list rule are obtained through the bucket list service of the object storage, which can be tasks obtained from the corresponding bucket list batch processing task queue consumer queue message, or tasks obtained from a preset task information list or task acquisition interface. The obtained bucket list batch processing tasks can include multiple bucket list batch processing tasks corresponding to multiple bucket list rules of a storage bucket, and can also include multiple bucket list batch processing tasks corresponding to multiple bucket list rules corresponding to multiple storage buckets.
[0047] S120 , for each bucket list batch processing task, split the task according to the preset data volume to obtain a plurality of batch processing subtasks with execution priority order corresponding to each bucket list batch processing task.
[0048] Normally, when processing bucket list batch tasks, each bucket list rule corresponds to a bucket list batch task. Accordingly, the processing time of bucket list batch tasks corresponding to buckets of different magnitudes is different. Batch tasks corresponding to large-scale buckets will occupy a large amount of execution resources for a long time, thereby blocking the execution of bucket list batch tasks corresponding to bucket list rules of other buckets. This results in that execution resources cannot be fairly allocated to each bucket, especially for small-scale buckets. Bucket list batch tasks corresponding to bucket list rules of small-scale buckets may not be executed for a long time.
[0049] In this embodiment, after obtaining the bucket list batch processing task, the bucket list batch processing task can be split to obtain corresponding batch processing subtasks. That is, each bucket list batch processing task is split to obtain multiple batch processing subtasks to reduce the data level corresponding to a single assigned bucket list batch processing task and ensure the timeliness of the bucket list batch processing task.
[0050] When splitting tasks, tasks can be split according to preset data volumes to obtain multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task. That is, the data volume corresponding to each batch processing subtask is the same.
[0051] The execution priority order of the batch processing subtasks may be determined according to the order of data in the storage bucket, or may be determined randomly or according to certain rules after the batch processing subtasks are split.
[0052] For example, the execution priority order of the batch processing subtasks in the corresponding bucket list batch processing task is determined according to the data order of the batch processing subtasks in the corresponding bucket list batch processing task.
[0053] Among them, the order in which the batch processing subtasks are split in the corresponding bucket list batch processing task can be the order in which the corresponding subtasks are executed when the bucket list batch processing task is split. It can be understood that the data order depends on the data sequence of the object data in the corresponding storage bucket. For example, the amount of object data in a storage bucket is 10 million data, and the task can be split according to the amount of 1 million data as a unit, and the 10 million data in the storage bucket are split according to the default data sorting, and a corresponding batch processing subtask is obtained for every 1 million data split. The task can also be split according to the time when the object data in the storage bucket is generated. Based on the order of data generation time, the object data within every three consecutive hours is split into a corresponding batch processing subtask. The order of the batch processing subtasks split in sequence determines the execution priority order of the corresponding batch processing subtasks. The batch processing subtasks split first have a higher execution priority order.
[0054] S130 , selecting multiple batch processing subtasks corresponding to each bucket list batch processing task in sequence according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and in the order of execution priority, and triggering allocation and execution of the selected batch processing subtasks.
[0055] The bucket order of the buckets may be a lexicographic order corresponding to each bucket identifier, or a randomly generated order.
[0056] Specifically, according to the bucket order of the storage buckets corresponding to each bucket list batch processing task, batch processing subtasks with high execution priority can be selected from multiple batch processing subtasks corresponding to each bucket list batch processing task, until the number of selected batch processing subtasks reaches the preset maximum task distribution number.
[0057] The preset maximum task distribution number may be a preset maximum task distribution number in a task distribution, which may be greater than or equal to the value of the number of task execution resources. The task execution resource may be a preset worker thread. That is, only a fixed number of batch processing subtasks are distributed each time, and after the batch processing subtasks distributed once are processed, other batch processing subtasks will be re-screened and distributed.
[0058] Specifically, in a task distribution round, each bucket list batch task is equal, and the order of task processing will not be affected by the different corresponding bucket list rules and the different number of objects, thereby affecting the timeliness of task completion. In each bucket list batch task, the batch subtasks corresponding to the highest execution priority order are distributed to the corresponding work thread according to the execution priority order. There can be a number of batch subtasks corresponding to the highest execution priority order of multiple bucket list batch tasks with the same number as the preset maximum task distribution number to be processed in the same batch task batch. If the number of bucket list batch tasks is less than the preset maximum task distribution number, the corresponding batch subtasks can be selected from each bucket list batch task according to the execution priority order, and the tasks can be distributed and processed together.
[0059] The fair distribution strategy based on this step ensures that buckets with different numbers of objects can be processed in parallel. When the service system is expanded, the expansion amount can also be evenly distributed to the bucket list batch processing tasks of each bucket, and there will be no resource skewness restrictions. In addition, the number of tasks in a single batch can effectively control the execution time of each batch task, giving the bucket list batch processing tasks corresponding to the new bucket list rules more opportunities to participate in distribution, which can effectively improve the user experience of using the object storage service.
[0060] In this embodiment, a bucket list batch processing task corresponding to each bucket list rule is obtained through the bucket list service of the object storage, wherein each bucket list rule corresponds to a bucket list batch processing task, and the processing object of each bucket list batch processing task includes multiple object data; for each bucket list batch processing task, the task is split according to the preset data volume, and multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task are obtained; according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and according to the execution priority order, multiple batch processing subtasks corresponding to each bucket list batch processing task are selected in sequence, and the allocation and execution of the selected batch processing subtasks are triggered. The technical solution of the disclosed embodiment solves the problem that the fairness and efficiency between the bucket list batch processing tasks corresponding to the bucket list rules of each object storage bucket cannot be taken into account in the bucket list service. By splitting and limiting the bucket list batch processing tasks corresponding to a single bucket list rule, a fair task distribution and processing strategy is implemented for object storage buckets of different orders of magnitude, ensuring that each bucket list batch processing task can have execution resources, and achieving efficiency and fairness between the bucket list batch processing tasks corresponding to the bucket list rules.
[0061] Figure 2 The present invention provides a flowchart of a method for processing a batch task of an object storage service, which further explains the process of task distribution in the process of implementing the method for processing a batch task of an object storage service based on the above embodiment. The method can be executed by a processing device for a batch task of an object storage service, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC, or a server.
[0062] like Figure 2 As shown, the method for processing a batch task of the object storage service includes:
[0063] S210: Obtain the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage.
[0064] Each bucket list rule corresponds to a bucket list batch processing task, and the processing object of each bucket list batch processing task includes multiple object data.
[0065] S220 , for each bucket list batch processing task, split the task according to the preset data volume to obtain a plurality of batch processing subtasks with execution priority order corresponding to each bucket list batch processing task.
[0066] The task data volume corresponding to each batch processing subtask is less than or equal to the preset data volume. That is, in this embodiment, the bucket list batch processing task is split according to a fixed data volume. In the result after task splitting, the data volume corresponding to each batch processing subtask will not exceed the preset data volume.
[0067] In the current bucket list service, there is no control over the task processing time after a task distribution round, that is, the working time of a task execution resource. During a batch task processing execution, the newly created bucket list batch task can only participate in the next task distribution round after the current batch task is completed. The waiting time is uncontrollable, which seriously affects the timeliness of the bucket list service.
[0068] In this embodiment, since the maximum amount of data processed by each batch processing subtask is defined, that is, the amount of data processed by each batch processing subtask has an upper limit, the task processing time is also controlled, further ensuring the processing timeliness of the bucket list batch processing task. According to the quantitative relationship between the number of tasks in the bucket list batch processing task and the preset maximum task distribution number, S230 or S240 can be executed.
[0069] S230. When the number of tasks in the bucket list batch processing task is greater than or equal to the preset maximum task distribution number, the batch processing subtask corresponding to the highest priority in the execution priority order is selected in each bucket list batch processing task according to the bucket order of the storage buckets corresponding to each bucket list batch processing task.
[0070] The preset maximum number of task distributions may correspond to the amount of execution resources supporting the bucket list batch processing task. For example, if the amount of execution resources supporting the bucket list batch processing task is 10 worker threads, the preset maximum number of task distributions is 10. In one task distribution, 10 batch processing subtasks may be selected and distributed.
[0071] The bucket list batch tasks corresponding to each bucket list rule are fair. When selecting batch subtasks, the batch subtasks corresponding to the highest priority in the execution priority order will be selected from the batch subtasks corresponding to each bucket list batch task according to the bucket order of the storage buckets corresponding to each bucket list batch task. Accordingly, the number of selected batch subtasks is also greater than or equal to the preset maximum number of task distributions.
[0072] S240. When the number of tasks in a bucket list batch task is less than the preset maximum number of task distributions, the batch subtask corresponding to the highest priority in the execution priority order is repeatedly selected in each bucket list batch task according to the bucket order of the storage buckets corresponding to each bucket list batch task and according to the execution priority order, until the number of tasks corresponding to the selected batch subtask is greater than or equal to the preset maximum number of task distributions.
[0073] Based on the fairness principle of the bucket list batch tasks corresponding to each bucket list rule, when selecting batch subtasks, the batch subtasks corresponding to the highest priority in the execution priority order will be selected from the batch subtasks corresponding to each bucket list batch task according to the bucket order of the storage buckets corresponding to each bucket list batch task. When the number of tasks of the bucket list batch task is less than the preset maximum number of task distributions, the number of selected batch subtasks will be less than the preset maximum number of task distributions, indicating that if only one batch subtask is selected, the execution resources of the bucket list batch task will be idle. In order to improve the utilization of task execution resources, the batch subtask corresponding to the highest priority in the execution priority order can be repeatedly selected in each bucket list batch task until the number of tasks corresponding to the selected batch subtask is greater than or equal to the preset maximum number of task distributions.
[0074] For example, there are 3 bucket list batch tasks and 10 corresponding worker threads. The first batch subtasks corresponding to each bucket list batch task are selected from the batch subtasks corresponding to each bucket list batch task according to the bucket order of the storage buckets corresponding to each bucket list batch task. The batch subtasks with the highest execution priority order are selected from the 3 bucket list batch tasks. At this time, there are still idle execution resources, and the second, third, and fourth task selections can be performed to select batch subtasks with a task quantity greater than or equal to the preset maximum task distribution quantity.
[0075] S250. According to the preset maximum task distribution quantity, the selected batch processing subtasks are assigned to the task execution work thread and executed.
[0076] Specifically, for the task number of the batch processing subtasks selected through S230 or S240, it is greater than or equal to the preset maximum task distribution number. According to the execution priority order and bucket order corresponding to each batch processing subtask in the selected batch processing subtasks, the batch processing subtasks corresponding to the preset maximum task distribution number in the selected batch processing subtasks can be assigned to the task execution work thread and executed. If there are remaining batch processing subtasks that have not been assigned for execution, the unassigned batch processing subtasks in the selected batch processing subtasks can be used as the assignment object of the next round of task assignment until all the selected batch processing subtasks are assigned and executed.
[0077] When the number of selected batch subtasks is the same as the preset maximum number of task distributions, it can be determined that a task distribution is completed, and then each batch subtask can be distributed to the corresponding work thread to perform bucket list batch task processing.
[0078] S260: After each assigned batch processing subtask is executed, the bucket list batch processing task corresponding to each bucket list rule is obtained again through the bucket list service to update the bucket list batch processing tasks that need to be processed.
[0079] After all the selected batch subtasks in the current task distribution round are distributed and processed, the next round of task distribution can be started. During the next round of task distribution, new bucket list batch tasks will be obtained through the bucket list service. The new bucket list batch tasks that need to be processed can be added to the next round of task distribution, without waiting for the old tasks to be completed before starting new tasks.
[0080] For example, there are 3 bucket list batch tasks (A, B, and C) and corresponding 5 worker threads. The 3 bucket list batch tasks are split into batch subtasks A1, A2, A3, B1, B2, C1, C2, C3, and C4. The first batch subtask selects A1, B1, and C1. The second batch subtask selects A2, B2, and C2. The number of selected batch subtasks is 6, which is greater than the number of worker threads. The task allocation and execution process can be triggered. According to the bucket order and execution priority order, the first task distribution allocates A1, B1, C1, A2, and B2 to the corresponding worker threads for execution.
[0081] Then, the bucket list batch processing tasks corresponding to each bucket list rule are obtained again through the bucket list service to update the bucket list batch processing tasks to be processed, and obtain the bucket list batch processing task D. D is split into D1 and D2. Then, batch processing subtasks can be selected, A3, C2, D1, C3 and D2. According to the bucket order and execution priority order, the second task distribution assigns D1, C2, D2, A3 and C3 to the corresponding worker threads for execution.
[0082] The technical solution of the embodiment of the present disclosure obtains the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage; for each bucket list batch processing task, the task is split according to the preset data volume to obtain multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task; when the number of tasks of the bucket list batch processing task is greater than or equal to the preset maximum task distribution number, the batch processing subtask corresponding to the highest priority in the execution priority order is selected in each bucket list batch processing task according to the bucket order of the storage buckets corresponding to each bucket list batch processing task; when the number of tasks of the bucket list batch processing task is less than the preset maximum task distribution number, the batch processing subtask corresponding to the highest priority in the execution priority order is repeatedly selected in each bucket list batch processing task according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and according to the execution priority order, until the number of tasks corresponding to the selected batch processing subtask is greater than or equal to the preset maximum task distribution number; according to the preset maximum task distribution number, the selected batch processing subtask is assigned to the task execution worker thread and executed. The technical solution of the disclosed embodiment solves the problem that the bucket list service cannot take into account the fairness and efficiency of the bucket list batch processing tasks corresponding to the bucket list rules of each object storage bucket. By splitting the bucket list batch processing tasks corresponding to a single storage bucket and limiting the data amount of a single subtask, a fair task distribution strategy is implemented for buckets of different orders of magnitude, ensuring that the bucket list batch processing tasks corresponding to the newly created bucket list rules can have execution resources, thereby further improving the efficiency and fairness of the bucket list service between storage buckets.
[0083] Figure 3 A flowchart of a method for processing a batch task of an object storage service provided in an embodiment of the present disclosure further explains the process of managing the batch tasks of the old and new bucket lists of the same bucket list rule in the process of implementing the processing method flow of the batch task of the object storage service based on the above embodiment. The method can be executed by a processing device for the batch task of the object storage service, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC, or a server, etc.
[0084] like Figure 3 As shown, the method for processing a batch task of the object storage service includes:
[0085] S310: Obtain the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage.
[0086] S320 . Determine, based on the bucket list rule information associated with the bucket list batch processing task, whether there are multiple bucket list batch processing tasks corresponding to the same bucket list rule in the bucket list batch processing task.
[0087] Normally, in the Bucket List Service, the bucket list batch tasks corresponding to the bucket list rules of each bucket are considered to have the same priority, and one bucket list rule corresponds to one bucket list batch task. When the task spans across days (a bucket list batch task has not been processed after 00:00), a bucket list rule of the bucket will correspond to both yesterday's bucket list batch task (old) and today's bucket list batch task (new), that is, one bucket corresponds to two bucket list batch tasks. From the bucket dimension, this bucket occupies two execution resources, which is also unfair to other buckets. When this phenomenon occurs in buckets of large magnitude, the unfairness to buckets of small magnitude will be further aggravated.
[0088] In this embodiment, after obtaining the bucket list batch processing task, it is first determined whether there are more than one bucket list batch processing tasks associated with the same bucket list rule according to the bucket list rule information corresponding to the bucket list batch processing task. That is, it is found whether there is a storage bucket corresponding to two or more old and new bucket list batch processing tasks of a bucket list rule. When two or more bucket list batch processing tasks associated with a bucket list rule identification information are found, it can be determined that there are associated pending list tasks corresponding to the same bucket list rule in the bucket list batch processing task.
[0089] S330: When there are multiple bucket list batch processing tasks corresponding to the same bucket list rule, a bucket list batch processing task with the earliest task time is selected from the multiple bucket list batch processing tasks according to the single bucket old rule priority strategy and is executed first.
[0090] In the case of multiple bucket list batch tasks corresponding to the same bucket list rule, in order to ensure that each bucket list rule corresponds to a bucket list batch task, the bucket list batch task with the earliest task time will be selected from the multiple bucket list batch tasks according to the single bucket old rule priority strategy for priority execution. That is, in the scenario of bucket list batch tasks that span multiple days, in order to ensure that all bucket list batch tasks can be scheduled fairly, the bucket list batch task splitting and task distribution management operations of multiple bucket list batch tasks corresponding to the same bucket list rule will be suspended. That is, when a bucket list rule has a corresponding unfinished old task, only the unfinished old task can participate in the task distribution scheduling and be processed first, and the corresponding new bucket list batch task does not participate in the task splitting and distribution.
[0091] For example, the bucket list batch processing task corresponding to bucket list rule A generates a multi-day task. Rule A corresponds to 3 bucket list batch processing tasks, of which 2 are old tasks, one is not completed and the other is not started; and 1 new task is left. At this time, the task that participates in task over-division and distribution corresponding to rule A is only the unfinished old task, and the old tasks and new tasks that have not started to be processed are suspended.
[0092] S340 , for each bucket list batch processing task, split the task according to the preset data volume to obtain a plurality of batch processing subtasks with execution priority order corresponding to each bucket list batch processing task.
[0093] S350 , selecting multiple batch processing subtasks corresponding to each bucket list batch processing task in sequence according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and in the order of execution priority, and triggering the allocation and execution of the selected batch processing subtasks.
[0094] The technical solution of the embodiment of the present disclosure obtains the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage; determines whether there are multiple bucket list batch processing tasks corresponding to the same bucket list rule in the bucket list batch processing task according to the bucket list rule information associated with the bucket list batch processing task; in the case of multiple bucket list batch processing tasks corresponding to the same bucket list rule, selects the bucket list batch processing task with the earliest task time from the multiple bucket list batch processing tasks according to the single bucket old rule priority strategy for priority execution; for each bucket list batch processing task, splits the task according to the preset data volume, and obtains multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task; selects multiple batch processing subtasks corresponding to each bucket list batch processing task in turn according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and the execution priority order, and triggers the allocation and execution of the selected batch processing subtasks. The technical solution of the embodiment of the present disclosure solves the problem of unfairness caused by one storage bucket occupying multiple task execution resources when a bucket list rule in the bucket list service has a cross-day task, and can further improve the efficiency and fairness of the bucket list service between storage buckets through the single storage bucket old task priority strategy.
[0095] Figure 4A flowchart of a method for processing a batch task of an object storage service provided in an embodiment of the present disclosure further explains the process of managing the batch tasks of the old and new bucket lists of the same bucket list rule in the process of implementing the processing method flow of the batch task of the object storage service based on the above embodiment. The method can be executed by a processing device for the batch task of the object storage service, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC, or a server, etc.
[0096] like Figure 4 As shown, the method for processing a batch task of the object storage service includes:
[0097] S410: Determine a preset execution time threshold of a single bucket list batch processing task and a preset concurrency threshold of the bucket list batch processing task, and determine an execution resource amount supporting the bucket list batch processing task.
[0098] In the Bucket List service, the service administrator can pre-configure the execution time threshold of a single Bucket List batch task, the concurrency threshold of the Bucket List batch task, and the execution resource amount supporting the Bucket List batch task to manage the processing process of the Bucket List batch task.
[0099] Among them, the execution time threshold of a single bucket list batch task limits the time it takes a worker thread to execute a task to a certain extent. If a task's execution time reaches the execution time threshold and is not completed, the thread will also end its life cycle to prevent the thread from being occupied for a long time. Correspondingly, the execution time threshold is also used to limit the preset data volume used to guide task splitting in the subsequent bucket list batch task splitting process, ensuring that each batch subtask can be completed within the execution time threshold under normal operation.
[0100] The concurrency threshold of the bucket list batch processing task is the number of tasks corresponding to the bucket list batch processing tasks that are processed simultaneously, that is, the number of bucket list rules that are processed simultaneously.
[0101] The amount of execution resources supporting the bucket list batch processing task may correspond to a preset maximum task distribution quantity.
[0102] S420: Determine a screening period for the bucket list batch processing task according to an execution time threshold of a single bucket list batch processing task, a concurrency threshold of the bucket list batch processing task, and an execution resource amount of the bucket list batch processing task. The screening period is used to control an execution period for the bucket list batch processing task.
[0103] In each task distribution round, the number of target distribution tasks corresponds to the execution resource amount of the bucket list batch processing tasks. The execution time of each task processed by the execution resource is limited by the execution time threshold, and the processing time of each round of tasks is also controlled accordingly.
[0104] Assume that each batch subtask is defined to execute for m minutes, and n batch subtasks are distributed each time. When there are t execution resources, the maximum execution time of all batch subtasks in the task distribution round is m*n / t minutes. In other words, with m*n / t minutes as the longest execution cycle, the batch subtasks of the next task distribution round will be re-screened and distributed every m*n / t minutes, and the newly created bucket list batch task can be executed after m*n / t minutes at most. At the same time, this time can be shortened by increasing the value of t. The timeliness of the bucket list batch tasks can be controlled at the hourly level, which greatly improves the user experience of the bucket list service.
[0105] After the task flow management parameters of the bucket list service are determined, the bucket list batch tasks that need to be processed can be executed according to the screening cycle of the corresponding bucket list batch tasks.
[0106] S430: Obtain the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage.
[0107] S440 . Determine, based on the bucket list rule information associated with the bucket list batch processing task, whether there are multiple bucket list batch processing tasks corresponding to the same bucket list rule in the bucket list batch processing task.
[0108] S450: When there are multiple bucket list batch processing tasks corresponding to the same bucket list rule, a bucket list batch processing task with the earliest task time is selected from the multiple bucket list batch processing tasks according to the single bucket old rule priority strategy and is executed first.
[0109] S460 , for each bucket list batch processing task, split the task according to the preset data volume to obtain a plurality of batch processing subtasks with execution priority order corresponding to each bucket list batch processing task.
[0110] S470 , selecting multiple batch processing subtasks corresponding to each bucket list batch processing task in sequence according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and in the order of execution priority, and triggering the allocation and execution of the selected batch processing subtasks.
[0111] S480: After each assigned batch processing subtask is executed, the bucket list batch processing task corresponding to each bucket list rule is obtained again through the bucket list service to update the bucket list batch processing tasks that need to be processed.
[0112] S490. According to the screening cycle, from the batch processing subtasks corresponding to the updated bucket list batch processing tasks that need to be processed, multiple batch processing subtasks are selected in sequence according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and according to the execution priority order, and the allocation and execution of the selected batch processing subtasks are triggered.
[0113] In this embodiment, a pre-configuration scheme for the execution time threshold, the concurrency threshold of the bucket list batch processing task, and the execution resource amount is provided. The execution time threshold of a preset single bucket list batch processing task and the concurrency threshold of the preset bucket list batch processing task can be determined according to the configuration result, and the execution resource amount supporting the bucket list batch processing task can be determined; and then, according to the execution time threshold of a single bucket list batch processing task, the concurrency threshold of the bucket list batch processing task, and the execution resource amount of the bucket list batch processing task, the screening cycle of the bucket list batch processing task can be determined. The screening cycle is used to control the execution cycle of the bucket list batch processing task to manage the process of the bucket list service. The fairness and efficiency of the bucket list batch processing tasks corresponding to the bucket list rules of each object storage bucket are achieved.
[0114] In a specific application example of the object storage service batch task processing method, the object storage service batch task processing process can be referred to Figure 5 The specific process includes the following:
[0115] First, after the object storage service batch task processing starts, the bucket list rule of the bucket is obtained periodically or according to the processing progress of each batch task, that is, the bucket list batch task corresponding to the bucket list rule of the bucket is obtained.
[0116] Then, split the single bucket list rule task into multiple batch processing tasks (corresponding to the batch processing subtasks in the above embodiment), and set the priority in order.
[0117] Before distributing tasks (batch subtasks), confirm whether there are any unfinished old tasks. That is, confirm whether there is a bucket with a bucket list rule for cross-day tasks. If so, the old tasks in the cross-day tasks will participate in the task distribution. If not, the new tasks will participate in the task distribution directly.
[0118] When allocating tasks, the tasks participating in the current task allocation round are selected in order of priority from the batch task list corresponding to each bucket rule.
[0119] Determine whether the number of tasks participating in the allocation of a single batch reaches the batch distribution upper limit. If so, distribute the tasks and then execute them. If not, return to the batch processing task list corresponding to each bucket rule and select tasks in order of priority.
[0120] After a single batch of tasks is completed, the bucket list rules can be obtained again.
[0121] Further, when splitting a single bucket list rule task into multiple batch tasks, the splitting process can refer to Figure 6 Schematic diagram of the process shown.
[0122] exist Figure 6 In the example, when obtaining the bucket list rules, two bucket list rules are obtained, rule A and rule B. The batch tasks are split according to the data processing volume, where the bucket a corresponding to rule A has 2 million data, and the bucket b corresponding to rule B has 3 million data. If the batch tasks are split according to the 1 million data volume, rule A corresponds to 2 batch tasks, with priorities 1 and 2, respectively, and the tasks can be represented as Ap1 and Ap2 respectively; rule B corresponds to 3 batch tasks, with priorities 1, 2, and 3, respectively, and the tasks can be represented as Bp1, Bp2, and Bp3 respectively. Furthermore, tasks can be distributed according to priority, and priorities 1, 2, 3, etc. are selected in turn. The distributed batch execution order is Ap1, Bp1, Ap2, Bp2, Bp3.
[0123] exist Figure 7 In the figure, the maximum number of tasks to be distributed in a single batch of tasks is set to 4. Tasks can be distributed until the maximum number of tasks is reached. If there are no tasks to be distributed before the maximum number of tasks is reached, all tasks will be distributed. For example, if there are both rules A and B at this time, rule A corresponds to 3 tasks, and rule B corresponds to 3 tasks, then according to the priority, the first distribution can take 2 tasks of rule A and 2 tasks of rule A for distribution, and the remaining tasks will wait for the next distribution. Wait for all tasks of the first distribution to be executed, and re-pull all bucket list rules. If there are newly created bucket list rules at this time, they can be obtained in this step. For example, the new rule C corresponds to 1 task. At this time, the tasks to be distributed will include 1 task of rule A, 1 task of rule B, and 1 task of rule C. Then, the second distribution can select tasks from the tasks to be distributed for distribution until all tasks are distributed. Starting from the second distribution, the tasks of the new rule (rule C) will participate in the task allocation, and there is no need to wait until all tasks corresponding to all rules are processed. Timeliness is guaranteed.
[0124] Figure 8 A processing device for a batch processing task of an object storage service provided in an embodiment of the present disclosure is suitable for a scenario in which data processing tasks in an object storage bucket are scheduled and managed in an object storage. The processing device for a batch processing task of an object storage service can be implemented in the form of software and / or hardware and can be configured on an electronic device, which can be a mobile terminal, a PC, a server, etc.
[0125] like Figure 8 As shown, the processing device of the object storage service batch processing task includes: a task acquisition module 510, a task splitting module 520 and a task allocation and processing module 530.
[0126] Among them, the task acquisition module 510 is used to obtain the bucket list batch processing tasks corresponding to each bucket list rule through the bucket list service of the object storage, wherein each bucket list rule corresponds to a bucket list batch processing task, and the processing object of each bucket list batch processing task includes multiple object data; the task splitting module 520 is used to split the tasks according to the preset data volume for each bucket list batch processing task, and obtain multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task; the task allocation and processing module 530 is used to select multiple batch processing subtasks corresponding to each bucket list batch processing task in turn according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and according to the execution priority order, and trigger the allocation and execution of the selected batch processing subtasks.
[0127] The technical solution of the disclosed embodiment is to obtain the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage, wherein each bucket list rule corresponds to a bucket list batch processing task, and the processing object of each bucket list batch processing task includes multiple object data; for each bucket list batch processing task, the task is split according to the preset data volume, and multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task are obtained; according to the bucket order of the storage bucket corresponding to each bucket list batch processing task and the execution priority order, the multiple batch processing subtasks corresponding to each bucket list batch processing task are selected in sequence, and the allocation and execution of the selected batch processing subtasks are triggered. The technical solution of the disclosed embodiment solves the problem that the fairness and efficiency between the bucket list batch processing tasks corresponding to the bucket list rules of each object storage bucket cannot be taken into account in the bucket list service. By splitting and limiting the bucket list batch processing tasks corresponding to a single bucket list rule, a fair task distribution and processing strategy is implemented for object storage buckets of different orders of magnitude, ensuring that each bucket list batch processing task can have execution resources, and achieving efficiency and fairness between the bucket list batch processing tasks corresponding to the bucket list rules.
[0128] In an optional implementation, the task allocation and processing module 530 may be specifically used for:
[0129] When the number of tasks in the bucket list batch task is greater than or equal to the preset maximum number of tasks to be distributed, the batch subtask corresponding to the highest priority in the execution priority order is selected in each bucket list batch task according to the bucket order of the storage buckets corresponding to each bucket list batch task;
[0130] According to the preset maximum task distribution quantity, the selected batch subtasks are assigned to the task execution worker thread and executed.
[0131] In an optional implementation, the task allocation and processing module 530 may be further specifically configured to:
[0132] According to the execution priority order and bucket order of each batch processing subtask in the selected batch processing subtasks, the batch processing subtasks corresponding to the preset maximum task distribution quantity in the selected batch processing subtasks are assigned to the task execution work thread and executed;
[0133] The unassigned batch processing subtasks among the selected batch processing subtasks are used as the assignment objects of the next round of task assignment until all the selected batch processing subtasks are assigned and executed.
[0134] In an optional implementation, the task allocation and processing module 530 may also be specifically used for:
[0135] When the number of tasks in a bucket list batch task is less than the preset maximum number of tasks to be distributed, the batch subtask corresponding to the highest priority in the execution priority order is repeatedly selected in each bucket list batch task according to the bucket order of the storage buckets corresponding to each bucket list batch task and according to the execution priority order, until the number of tasks corresponding to the selected batch subtask is greater than or equal to the preset maximum number of tasks to be distributed.
[0136] In an optional implementation, the task acquisition module 510 may also be used to:
[0137] After obtaining the bucket list batch processing tasks corresponding to each bucket list rule, determining whether there are multiple bucket list batch processing tasks corresponding to the same bucket list rule in the bucket list batch processing tasks according to the bucket list rule information associated with the bucket list batch processing tasks;
[0138] In the case where there are multiple bucket list batch processing tasks corresponding to the same bucket list rule, a bucket list batch processing task with the earliest task time is selected from the multiple bucket list batch processing tasks according to the single bucket old rule priority strategy and is executed first.
[0139] In an optional implementation, the task acquisition module 510 may also be used to:
[0140] After each assigned batch processing subtask is executed, the bucket list batch processing task corresponding to each bucket list rule is obtained again through the bucket list service to update the bucket list batch processing tasks that need to be processed.
[0141] In an optional implementation, the processing device for the object storage service batch processing task further includes a task execution management module, which is used to:
[0142] Determine a preset execution time threshold for a single bucket list batch task and a preset concurrency threshold for the bucket list batch task, and determine the amount of execution resources to support the bucket list batch task;
[0143] The screening period of the bucket list batch processing task is determined according to the execution time threshold of a single bucket list batch processing task, the concurrency threshold of the bucket list batch processing task and the execution resource amount of the bucket list batch processing task. The screening period is used to control the execution period of the bucket list batch processing task.
[0144] The processing device for batch processing tasks of the object storage service provided in the embodiments of the present disclosure can execute the processing method for batch processing tasks of the object storage service provided in any embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method.
[0145] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.
[0146] Fig. 9 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig. 9 , which shows an electronic device (eg, Fig. 9 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0147] like Fig. 9 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An edit / output (I / O) interface 605 is also connected to the bus 605.
[0148] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0149] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0150] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0151] The electronic device provided in the embodiment of the present disclosure and the method for processing batch tasks of the object storage service provided in the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0152] The embodiments of the present disclosure also provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the method for processing a batch task of the object storage service provided in the above embodiments is implemented.
[0153] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0154] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0155] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0156] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:
[0157] Obtain the bucket list batch processing task corresponding to each bucket list rule through the bucket list service of the object storage, wherein each bucket list rule corresponds to a bucket list batch processing task, and the processing object of each bucket list batch processing task includes multiple object data;
[0158] For each bucket list batch processing task, split the task according to the preset data volume to obtain multiple batch processing subtasks with execution priority order corresponding to each bucket list batch processing task;
[0159] According to the bucket order of the storage buckets corresponding to each bucket list batch task and according to the execution priority order, multiple batch subtasks corresponding to each bucket list batch task are selected in turn, and the allocation and execution of the selected batch subtasks are triggered.
[0160] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0161] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0162] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".
[0163] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0164] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0165] An embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements a method for processing a batch task of an object storage service as provided in any embodiment of the present disclosure.
[0166] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code for the disclosed operation, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).
[0167] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0168] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0169] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A method for processing a batch task of an object storage service, characterized in that: include: Obtaining a bucket list batch processing task corresponding to each bucket list rule through a bucket list service of an object storage, wherein each bucket list rule corresponds to one bucket list batch processing task, and a processing object of each bucket list batch processing task includes multiple object data; For each of the bucket list batch processing tasks, split the tasks according to the preset data volume to obtain a plurality of batch processing subtasks with execution priority order corresponding to each of the bucket list batch processing tasks; According to the bucket order of the storage buckets corresponding to each of the bucket list batch processing tasks and according to the execution priority order, multiple batch processing subtasks corresponding to each of the bucket list batch processing tasks are selected in turn, and the allocation and execution of the selected batch processing subtasks are triggered.
2. The method according to claim 1, characterized in that The step of selecting, in accordance with the bucket order of the storage buckets corresponding to each of the bucket list batch processing tasks and in accordance with the execution priority order, a plurality of batch processing subtasks corresponding to each of the bucket list batch processing tasks, and triggering the allocation and execution of the selected batch processing subtasks, comprises: When the number of tasks of the bucket list batch processing task is greater than or equal to the preset maximum task distribution number, selecting a batch processing subtask corresponding to the highest priority in the execution priority order in each bucket list batch processing task according to the bucket order of the storage buckets corresponding to each bucket list batch processing task; According to the preset maximum task distribution quantity, the selected batch processing subtasks are assigned to the task execution work thread and executed.
3. The method according to claim 2, characterized in that The method of allocating the selected batch processing subtasks to the task execution work thread and executing them according to the preset maximum task distribution quantity includes: According to the execution priority order and the bucket order corresponding to each batch processing subtask in the selected batch processing subtasks, the batch processing subtasks corresponding to the preset maximum task distribution quantity in the selected batch processing subtasks are allocated to the task execution work thread and executed; The unassigned batch processing subtasks among the selected batch processing subtasks are used as the assignment objects of the next round of task assignment until all the selected batch processing subtasks are assigned and executed.
4. The method according to claim 2, characterized in that: The method further comprises selecting, in accordance with the bucket order of the storage buckets corresponding to each of the bucket list batch processing tasks and in accordance with the execution priority order, a plurality of batch processing subtasks corresponding to each of the bucket list batch processing tasks, and triggering the allocation and execution of the selected batch processing subtasks. When the number of tasks in the bucket list batch processing task is less than the preset maximum task distribution number, according to the bucket order of the storage buckets corresponding to each of the bucket list batch processing tasks and according to the execution priority order, repeatedly select the batch processing subtask corresponding to the highest priority in the execution priority order in each of the bucket list batch processing tasks until the number of tasks corresponding to the selected batch processing subtasks is greater than or equal to the preset maximum task distribution number.
5. The method according to claim 1, characterized in that After obtaining the bucket list batch processing task corresponding to each bucket list rule, the method further includes: Determining, according to the bucket list rule information associated with the bucket list batch processing task, whether there are multiple bucket list batch processing tasks corresponding to the same bucket list rule in the bucket list batch processing task; In the case where there are multiple bucket list batch processing tasks corresponding to the same bucket list rule, a bucket list batch processing task with the earliest task time is selected from the multiple bucket list batch processing tasks according to the single bucket old rule priority strategy and is executed first.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: After each assigned batch processing subtask is executed, the bucket list batch processing task corresponding to each bucket list rule is obtained again through the bucket list service to update the bucket list batch processing tasks that need to be processed.
7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Determine a preset execution time threshold for a single bucket list batch task and a preset concurrency threshold for the bucket list batch task, and determine the amount of execution resources to support the bucket list batch task; A screening period of the bucket list batch processing task is determined according to the execution time threshold of the single bucket list batch processing task, the concurrency threshold of the bucket list batch processing task, and the execution resource amount of the bucket list batch processing task, and the screening period is used to control the execution period of the bucket list batch processing task.
8. A processing device for batch processing tasks of an object storage service, characterized in that: include: A task acquisition module, used to acquire a bucket list batch processing task corresponding to each bucket list rule through a bucket list service of an object storage, wherein each bucket list rule corresponds to one bucket list batch processing task, and a processing object of each bucket list batch processing task includes multiple object data; A task splitting module, used for splitting each of the bucket list batch processing tasks according to a preset data volume, to obtain a plurality of batch processing subtasks with execution priority order corresponding to each of the bucket list batch processing tasks; The task allocation and processing module is used to select multiple batch processing subtasks corresponding to each bucket list batch processing task in sequence according to the bucket order of the storage buckets corresponding to each bucket list batch processing task and according to the execution priority order, and trigger the allocation and execution of the selected batch processing subtasks.
9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the processing method for batch processing tasks of the object storage service as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the program implements the method for processing a batch task of an object storage service as described in any one of claims 1 to 7.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements a method for processing a batch task of an object storage service as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Batch task processing method and device and service system
CN108829506A
Task state management method and device
CN109558237A
Hotspot data batch scheduling method and device
CN110347513A
Data parallel processing method, device and equipment and storage medium
CN111400012A
Batch processing task execution method and system and computer room deployment system
CN111666144A
