Data processing task scheduling method, device, storage medium and electronic device

By sending data processing tasks to the scheduler in the energy-saving archive system, and obtaining the scheduling tasks and operation attributes of the logical storage unit, and generating control instructions to optimize the use of disk resources, the problem that the application cannot perceive the disk power scheduling information, realizing the It reduces scheduling costs and resource waste.

CN114356514BActive Publication Date: 2024-11-05ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111470968.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-03
Publication Date
2024-11-05
Estimated Expiration
2041-12-03

AI Technical Summary

Technical Problem

In existing energy-saving archive systems, since applications cannot perceive the up and down scheduling information of the disk, scheduling costs and waste of resources are increased.

Method used

By sending data processing tasks to the scheduler, the scheduling tasks and operation attributes of the logical storage unit are obtained, and control instructions are generated based on this information, and the scheduler is controlled to aggregate disk resources to process data processing tasks, so that the application can sense the up and down scheduling information of the disk , thereby optimizing the scheduling rate and resource usage of data processing tasks.

Benefits of technology

It realizes the reduction of scheduling costs and resource waste, improves the scheduling rate of data processing tasks, and solves the problem of resource waste caused by the inability to perceive disk power scheduling information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114356514B_ABST
    Figure CN114356514B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, device, storage medium and electronic device for scheduling data processing tasks. The method includes: sending data processing tasks to a scheduler, wherein the data processing tasks include at least archiving tasks and readback tasks; obtaining scheduling tasks and operation attributes of a logical storage unit; generating control instructions based on the scheduling tasks and the operation attributes; sending control instructions to the scheduler, wherein the control instructions are used to control the scheduler to process the data processing tasks by scheduling the logical storage unit. The present invention solves the technical problem that the energy-saving archiving system in the related art has increased scheduling costs and wasted resources because the application APP cannot perceive the power-on and power-off scheduling information of the disk.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and electronic device for scheduling data processing tasks. Background Art

[0002] In the related art, for energy-saving archiving systems, energy saving is usually achieved by periodically powering on and off a portion of the disks, and this part is implemented by a separate scheduling logic. The disk can only be read and written when it is powered on. If the application cannot perceive the scheduling logic, it cannot predict and judge the operability, resulting in the introduction of the transfer area system and temporarily storing the data in the transfer area for background migration, resulting in increased costs and waste of resources. For example, in the cold data processing scenario, in order to effectively reduce costs, the application in the energy-saving archiving system will import data into the energy-saving archiving system. The speed at which cold data is written to the energy-saving archiving system will affect the idle cost of the system, and the performance of the readback will affect the user's direct usage experience.

[0003] For example, in an energy-saving archiving system based on power on and off, since the application APP cannot perceive the power on and off scheduling of the disk, it is necessary to put the data in a temporary dump area, which brings additional overhead and the performance and latency are subject to the constraints of the dump area, resulting in increased costs and waste of resources.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a scheduling method, device, storage medium and electronic device for data processing tasks, so as to at least solve the technical problems of energy-saving archiving systems in related technologies, that is, the application program APP cannot perceive the power-on and power-off scheduling information of the disk, resulting in increased scheduling costs and waste of resources.

[0006] According to one aspect of an embodiment of the present invention, a scheduling method for data processing tasks is provided, comprising: sending data processing tasks to a scheduler, wherein the data processing tasks include at least archiving tasks and readback tasks; obtaining scheduling tasks and operating attributes of logical storage units; generating control instructions based on the scheduling tasks and the operating attributes; and sending control instructions to the scheduler, wherein the control instructions are used to control the scheduler to process the data processing tasks by scheduling the logical storage units.

[0007] According to another aspect of an embodiment of the present invention, a scheduling device for data processing tasks is also provided, including: a sending module, used to send data processing tasks to a scheduler, wherein the above-mentioned data processing tasks include at least: archiving tasks and readback tasks; an acquisition module, used to obtain scheduling tasks and operation attributes of logical storage units, wherein the above-mentioned logical storage units are obtained by pre-dividing disk resources in multiple disk expansion cabinets; a generation module, used to generate control instructions based on the above-mentioned scheduling tasks and the above-mentioned operation attributes; a scheduling module, used to control the sending of control instructions to the above-mentioned scheduler, wherein the above-mentioned control instructions are used to control the above-mentioned scheduler to process the above-mentioned data processing tasks by scheduling the above-mentioned logical storage units.

[0008] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, a scheduling method is used to control the device where the computer-readable storage medium is located to perform any one of the above-mentioned data processing tasks.

[0009] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the program executes any one of the above-mentioned scheduling methods for data processing tasks when running.

[0010] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory, connected to the above-mentioned processor, for providing the above-mentioned processor with instructions for processing the following processing steps: Step 1, sending a data processing task to a scheduler, wherein the above-mentioned data processing task includes at least: an archiving task and a read-back task; Step 2, obtaining a scheduling task and an operation attribute of a logical storage unit, wherein the above-mentioned logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; Step 3, generating a control instruction based on the above-mentioned scheduling task and the above-mentioned operation attribute; Step 4, sending a control instruction to the above-mentioned scheduler, wherein the above-mentioned control instruction is used to control the above-mentioned scheduler to process the above-mentioned data processing task by scheduling the above-mentioned logical storage unit.

[0011] In an embodiment of the present invention, by sending a data processing task to a scheduler, wherein the data processing task includes at least: an archiving task and a readback task; obtaining the scheduling task and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing the disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attributes; sending the control instruction to the scheduler, wherein the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit, thereby achieving the purpose of aggregating data by dividing the disk so that the application program perceives the power-on and power-off scheduling information of the disk, thereby achieving the technical effect of improving the scheduling rate of data processing tasks and reducing scheduling costs and resource waste, thereby solving the technical problem of increased scheduling costs and resource waste in the energy-saving archiving system in the related technology due to the inability of the application program APP to perceive the power-on and power-off scheduling information of the disk. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0013] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a scheduling method for data processing tasks is shown;

[0014] Figure 2 is a flow chart of a method for scheduling data processing tasks according to an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of an optional division of disk resources into logical storage units according to an embodiment of the present invention;

[0016] Figure 4 is a scenario schematic diagram of a method for scheduling data processing tasks according to an embodiment of the present invention;

[0017] Figure 5 A schematic diagram of the structure of a scheduling device for data processing tasks according to an embodiment of the present invention;

[0018] Figure 6 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0022] JBOD (Just a Bunch Of Disks): disk expansion cabinet;

[0023] Erasure Coding (EC): Erasure Coding. A data protection method with coding error tolerance;

[0024] DataContainer (DC): A logical storage unit, for example, a data container, which is a combination of a group of disks that meet certain location constraints selected from JBOD. It is also a power scheduling and storage unit. Data is stored in the DC in multiple copies or EC mode to ensure reliability. The cost of moving data within the DC is lower than between DCs.

[0025] Application-aware scheduling: Supports application-aware DC and scheduling logic to maximize resource utilization efficiency;

[0026] Energy-saving archiving system: Divide the disks in multiple groups of JBOD into multiple groups of DCs according to constraints, and perform power on and off scheduling based on DCs to achieve the goal of saving energy.

[0027] Example 1

[0028] According to an embodiment of the present invention, an embodiment of a scheduling method for data processing tasks is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] The method embodiment provided in Embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a scheduling method for data processing tasks is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0030] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0031] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method for scheduling data processing tasks in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the scheduling of the above-mentioned data processing tasks is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0034] Under the above operating environment, the present invention provides Figure 2 A scheduling method for data processing tasks is shown in FIG. Figure 2 is a flow chart of a method for scheduling data processing tasks according to an embodiment of the present invention. Figure 2 As shown, the method includes:

[0035] Step S202, sending a data processing task to a scheduler, wherein the data processing task at least includes: an archiving task and a read-back task;

[0036] Step S204, obtaining the scheduling tasks and operation attributes of the logical storage unit, wherein the above-mentioned logical storage unit is obtained by pre-dividing the disk resources in a plurality of disk expansion cabinets;

[0037] Step S206, generating a control instruction based on the above scheduling task and the above operation attributes;

[0038] Step S208, sending a control instruction to the above-mentioned scheduler, wherein the above-mentioned control instruction is used to control the above-mentioned scheduler to process the above-mentioned data processing task by scheduling the above-mentioned logical storage unit.

[0039] Optionally, the scheduling method for the above-mentioned data processing tasks provided in the embodiments of the present application can be but is not limited to application in archive storage scenarios, for example, energy-saving archive systems, cold archive storage systems. As an optional embodiment, the method can be implemented in the following products, for example: deep archive storage series products, cloud archive storage products, search engine archive storage products, and the like.

[0040] In an optional embodiment, taking the application of the above method to an energy-saving archiving system as an example, the energy-saving archiving system uses the above logical storage unit (DC, for example, data container) as a scheduling unit, and periodically powers on and off a predetermined portion of the above logical storage units in all disk resources to achieve an energy-saving goal.

[0041] Optional, such as Figure 3 As shown, the client of the energy-saving archiving system sends a data processing task to the scheduler, and the above data processing task includes at least: an archiving task and a readback task; wherein the archiving task is a processing task for archiving data, and the readback task is a processing task for reading back data, for example Figure 3 As shown, DC0: read file set, write file set; DC1: read file set, write file set.

[0042] Optionally, the above scheduler is as follows Figure 3 The DC scheduler shown is Figure 3 As shown, multiple logical storage units DC0, DC1, DC2, DC5, DC7, etc. are obtained by pre-dividing the disk resources in multiple disk expansion cabinets. The above scheduling tasks include: power-on and power-off planning tasks. The above operation attributes include at least one of the following: read attribute, write attribute, internal repair attribute, wherein, Figure 3 As shown, according to the upper and lower point planning tasks, when scheduling, you can but are not limited to the following Figure 3 As shown in the diagram, the operation attribute corresponding to the logical storage units DC0 and DC1 is a read attribute, the operation attribute corresponding to the logical storage units DC5 and DC7 is a write attribute, and the attribute corresponding to the logical storage unit DC2 is an internal repair attribute.

[0043] As an optional embodiment, in order to improve the end-to-end writing and reading back processing performance, and eliminate the overhead of the dump area (that is, there is no need to introduce a temporary dump area for application reading and writing data, and the archiving background logic does not need to move out or move in user data from the dump area), due to the particularity of the energy-saving archiving system, the application can obtain the archiving or reading back plan in advance, and send it to the archiving backend scheduler, and convert the archiving and reading back into priority read and write operation tasks of the DC. The DC scheduler obtains the DC read and write operation tasks sent by all clients, merges them, and schedules the DC according to the priority. The scheduling plan is determined within a certain period of time, and the scheduling result is the DC of the next power-on cycle, which is divided into 3 categories: application reading, application writing, and DC internal repair, and is updated to the DC status table. The application can periodically obtain the scheduling plan from the DC status table. Because the scheduling plan is determined, the application can arrange the workload of the next power-on time according to the scheduling plan, such as resource allocation and scheduling of archiving and reading back, etc. In addition, the application will aggregate and write the data that needs to be read back at the same time in the future into the same DC. By perceiving DC and scheduling planning, the application can achieve the purpose of directly and efficiently using hard disk resources.

[0044] The embodiment of the present application proposes a scheduling method for data processing tasks, including a new disk resource division method, which divides disk resources into logical storage units, namely, scheduling and storage units DC, so that applications can perceive DC information and scheduling plans. The scheduling system can optimize resource utilization efficiency, and applications can cooperate with scheduling to achieve optimal use of resources. In addition, applications can aggregate data and write related data on the same DC, thereby reducing the latency under small-scale readback.

[0045] In an embodiment of the present invention, by sending a data processing task to a scheduler, wherein the data processing task includes at least: an archiving task and a readback task; obtaining the scheduling task and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing the disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attributes; sending the control instruction to the scheduler, wherein the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit, thereby achieving the purpose of aggregating data by dividing the disk so that the application program perceives the power-on and power-off scheduling information of the disk, thereby achieving the technical effect of improving the scheduling rate of data processing tasks and reducing scheduling costs and resource waste, thereby solving the technical problem of increased scheduling costs and resource waste in the energy-saving archiving system in the related technology due to the inability of the application program APP to perceive the power-on and power-off scheduling information of the disk.

[0046] In an optional embodiment, before sending the data processing task to the scheduler, the method further includes:

[0047] Step S302 , dividing disk resources in a plurality of disk expansion cabinets into a plurality of the above-mentioned logical storage units, wherein the above-mentioned plurality of disk expansion cabinets are distributed in rack servers under a plurality of switches of the energy-saving archiving system.

[0048] like Figure 3 As shown, since multiple disk expansion cabinets are distributed in rack servers under multiple switches of the energy-saving archiving system, before sending the data processing task to the scheduler, the disk resources in the multiple disk expansion cabinets can be divided into multiple of the above-mentioned logical storage units, for example, logical storage unit 1, logical storage unit 2 and logical storage unit 3, and so on.

[0049] An optional division process of dividing the disk resources in the plurality of disk expansion cabinets into the plurality of the above-mentioned logical storage units may be:

[0050] disk j : The jth disk;

[0051] rack j :jth rack server;

[0052] dc k : kth DC;

[0053] repeat

[0054] From the remaining racks, select the rack that is under the same switch. j0 and rack j1 ;

[0055] repeat

[0056] From the rack under the same switch j0 and rack j1 Among all JBODs, for the remaining unselected disk resources;

[0057] Join to dc in turn k , until data security requirements cannot be met;

[0058] Until dc k Contains n disks,

[0059] Until all the rack disks are added to a DC.

[0060] In an optional embodiment, the above-mentioned dividing the disk resources in the plurality of disk expansion cabinets into the plurality of the above-mentioned logical storage units includes:

[0061] Step S402: dividing the disk resources into a plurality of logical storage units according to preset constraints.

[0062] Optionally, the preset constraints include at least one of the following: a first constraint, a second constraint and a third constraint, wherein the first constraint is used to describe that the disk resources under the same logical storage unit correspond to a predetermined number of the rack servers, the second constraint is used to describe that each logical storage unit contains multiple disk resources, and the third constraint is used to describe that each logical storage unit selects disk resources in each of the disk expansion cabinets.

[0063] As an optional embodiment, in the archive cluster, there are several disk expansion cabinets JBOD distributed in rack servers Rack under multiple switches, which can be divided into DC groups according to the following constraints: the first constraint is that the disks under the same DC are located in two rack servers Rack under the same switch. Optionally, the first constraint is to reduce the cost of data movement within the DC; the second constraint is that the number of disks contained in each DC is n, ensuring that there are enough DCs in the cluster for scheduling; the third constraint is that each DC only selects a few block disks in each JBOD to ensure data security in the JBOD maintenance scenario.

[0064] In addition, it should be noted that after the disk resources are divided into DC groups, the background scheduler can periodically schedule power on and off based on DC units, and provide an interface to expose the scheduling plan to the application.

[0065] In an optional embodiment, obtaining the scheduling task and operation attribute of the logical storage unit includes:

[0066] Step S502, obtaining the power-on and power-off planning tasks corresponding to each of the plurality of logical storage units, and obtaining the operation attributes corresponding to each of the logical storage units, wherein the power-on and power-off planning tasks are used to plan the next power-on time interval of the logical storage unit.

[0067] In an optional embodiment, according to the scheduling task and the operation attribute, controlling the scheduler to execute scheduling of the logical storage unit to process the data processing task includes:

[0068] Step S602, obtaining the next power-on time interval of the above-mentioned logical storage unit determined by the above-mentioned power-on and power-off planning task;

[0069] Step S604 , according to the next power-on time interval of the above-mentioned logical storage unit and the above-mentioned operation attribute corresponding to each of the above-mentioned logical storage units, correspondingly controlling the above-mentioned scheduler to execute scheduling of the above-mentioned logical storage unit to process the above-mentioned data processing task.

[0070] In an optional embodiment, the power-on and power-off planning tasks corresponding to each of the plurality of logic storage units are obtained, and the operation attributes corresponding to each of the logic storage units are obtained, and then according to the next power-on time interval of the logic storage unit and the operation attributes corresponding to each of the logic storage units, the scheduler is controlled to execute the scheduling of the logic storage unit to process the data processing tasks; Figure 4 As shown, the content of the above data processing tasks includes: application read processing, application write processing, and internal repair processing of the above logic storage unit. It should be noted that since the scheduling plan is determined within a certain period of time, the scheduling result is the DC of the next power-on cycle, which can be roughly divided into three categories, but not limited to, application read, application write, and DC internal repair.

[0071] In addition, Figure 4 As shown, the application can also periodically obtain the scheduler plan. Since the schedule plan is fixed, the application can arrange the load of the next power-on time according to the schedule plan, for example, resource allocation and scheduling for archiving and readback, etc. In addition, the application will aggregate and write the data that needs to be read back at the same time in the future to the same DC. By sensing the DC and the schedule plan, the application in the embodiment of the present application can achieve the purpose of directly and efficiently using hard disk resources.

[0072] In an optional embodiment, controlling the scheduler to execute scheduling of the logical storage unit to process the data processing task includes:

[0073] Step S702, determining the priority of the above data processing task;

[0074] Step S704: scheduling the above-mentioned logical storage unit to process the above-mentioned data processing task according to the above-mentioned priority.

[0075] In an embodiment of the present application, the above-mentioned scheduler is controlled to execute scheduling of the above-mentioned logical storage unit to process the above-mentioned data processing task, that is, according to the determined priority of the data processing task, the scheduler is controlled to schedule the logical storage unit according to the priority to process the above-mentioned data processing task.

[0076] Through the embodiments of the present application, the concepts of scheduling and storage units are introduced, and the scheduling plans are exposed to applications. Applications can periodically obtain the scheduling plans of DCs, and perform IO planning based on the operating attributes of DCs to achieve the effect of direct and efficient use. In addition, the concepts of scheduling and storage units are introduced, and DC grouping and scheduling plans are exposed to applications. The use of temporary transfer areas is avoided, and applications can use disk resources more transparently and efficiently in the case of power on and off by perceiving DCs and scheduling.

[0077] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0079] Example 2

[0080] According to an embodiment of the present invention, there is also provided an apparatus embodiment for implementing the scheduling method for the above data processing tasks. Figure 5 A schematic diagram of the structure of a data processing task scheduling device according to an embodiment of the present invention is shown in Figure 5. The device includes: a sending module 500, an acquisition module 502, a generation module 504 and a scheduling module 506, wherein:

[0081] A sending module 500 is used to send data processing tasks to a scheduler, wherein the data processing tasks include at least: archiving tasks and read-back tasks; an acquiring module 502 is used to acquire scheduling tasks and operating attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; a generating module 504 is used to generate control instructions based on the scheduling tasks and the operating attributes; a scheduling module 506 is used to control sending control instructions to the scheduler, wherein the control instructions are used to control the scheduler to process the data processing tasks by scheduling the logical storage unit.

[0082] It should be noted that the sending module 500, the acquiring module 502, the generating module 504 and the scheduling module 506 correspond to steps S202 to S208 in Example 1, and the examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0083] In an embodiment of the present invention, a sending module 500 is used to send data processing tasks to a scheduler, wherein the data processing tasks include at least: archiving tasks and readback tasks; an obtaining module 502 is used to obtain scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in multiple disk expansion cabinets; a generating module 504 is used to generate control instructions based on the scheduling tasks and the operation attributes; a scheduling module 506 is used to control the sending of control instructions to the scheduler, wherein the control instructions are used to control the scheduler to process the data processing tasks by scheduling the logical storage unit, thereby achieving the purpose of aggregating data by dividing the disk, so that the application program perceives the power-on and power-off scheduling information of the disk, thereby achieving the technical effect of improving the scheduling rate of data processing tasks and reducing scheduling costs and resource waste, thereby solving the technical problem of increased scheduling costs and resource waste in the energy-saving archiving system in the related technology due to the inability of the application program APP to perceive the power-on and power-off scheduling information of the disk.

[0084] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0085] Example 3

[0086] According to an embodiment of the present invention, an embodiment of an electronic device is also provided, and the electronic device can be any computing device in a computing device group. The electronic device includes: a processor and a memory, wherein:

[0087] A processor; and a memory, connected to the processor, for providing instructions for the processor to process the following processing steps: Step 1, sending a data processing task to a scheduler, wherein the data processing task includes at least an archiving task and a read-back task; Step 2, obtaining a scheduling task and an operation attribute of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; Step 3, generating a control instruction based on the scheduling task and the operation attribute; Step 4, sending a control instruction to the scheduler, wherein the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit.

[0088] In an embodiment of the present invention, by sending a data processing task to a scheduler, wherein the data processing task includes at least: an archiving task and a readback task; obtaining the scheduling task and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing the disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attributes; sending the control instruction to the scheduler, wherein the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit, thereby achieving the purpose of aggregating data by dividing the disk so that the application program perceives the power-on and power-off scheduling information of the disk, thereby achieving the technical effect of improving the scheduling rate of data processing tasks and reducing scheduling costs and resource waste, thereby solving the technical problem of increased scheduling costs and resource waste in the energy-saving archiving system in the related technology due to the inability of the application program APP to perceive the power-on and power-off scheduling information of the disk.

[0089] It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0090] Example 4

[0091] According to an embodiment of the present invention, an embodiment of a computer terminal can also be provided, and the computer terminal can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0092] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0093] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the scheduling method of data processing tasks: sending data processing tasks to the scheduler, wherein the above-mentioned data processing tasks include at least: archiving tasks and read-back tasks; obtaining scheduling tasks and operation attributes of logical storage units, wherein the above-mentioned logical storage units are obtained by pre-dividing disk resources in multiple disk expansion cabinets; generating control instructions based on the above-mentioned scheduling tasks and the above-mentioned operation attributes; sending control instructions to the above-mentioned scheduler, wherein the above-mentioned control instructions are used to control the above-mentioned scheduler to process the above-mentioned data processing tasks by scheduling the above-mentioned logical storage units.

[0094] Optionally, Figure 6 is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 6 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 602 , a memory 604 , and a peripheral interface 606 .

[0095] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the scheduling method and device for data processing tasks in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the scheduling method for the above-mentioned data processing tasks. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0096] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: sending a data processing task to the scheduler, wherein the data processing task includes at least: an archiving task and a read-back task; obtaining the scheduling task and operation attributes of the logical storage unit, wherein the logical storage unit is obtained by pre-dividing the disk resources in multiple disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attributes; sending the control instruction to the scheduler, wherein the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit.

[0097] Optionally, the processor may also execute program code of the following steps: dividing disk resources in a plurality of disk expansion cabinets into a plurality of the logical storage units, wherein the plurality of disk expansion cabinets are distributed in rack servers under a plurality of switches of the energy-saving archiving system.

[0098] Optionally, the processor may also execute program code for the following steps: dividing the disk resources into a plurality of the logical storage units according to preset constraints, wherein the preset constraints include at least one of the following: a first constraint, a second constraint, and a third constraint, wherein the first constraint is used to describe that the disk resources under the same logical storage unit correspond to the same switch; the second constraint is used to describe that the disk resources under the same logical storage unit correspond to a predetermined number of the rack servers; and the third constraint is used to describe that each logical storage unit contains multiple disks.

[0099] Optionally, the processor may also execute program code for the following steps: obtaining the power-on and power-off planning tasks corresponding to each of the multiple logical storage units, and obtaining the operation attributes corresponding to each of the logical storage units, wherein the power-on and power-off planning tasks are used to plan the next power-on time interval of the logical storage unit.

[0100] Optionally, the processor may also execute program code for the following steps: obtaining the next power-on time interval of the logical storage unit determined by the power-on and power-off planning task; and controlling the scheduler to schedule the logical storage unit to process the data processing task according to the next power-on time interval of the logical storage unit and the operation attributes corresponding to each of the logical storage units. The content of the data processing task includes: application read processing, application write processing, and internal repair processing of the logical storage unit.

[0101] Optionally, the processor may also execute program codes of the following steps: determining the priority of the data processing task; and scheduling the logic storage unit to process the data processing task according to the priority.

[0102] By adopting the embodiment of the present invention, a scheme for scheduling data processing tasks is provided. By sending data processing tasks to a scheduler, wherein the data processing tasks at least include: archiving tasks and readback tasks; obtaining the scheduling tasks and operation attributes of the logical storage unit, wherein the logical storage unit is obtained by pre-dividing the disk resources in multiple disk expansion cabinets; generating control instructions based on the scheduling tasks and the operation attributes; sending control instructions to the scheduler, wherein the control instructions are used to control the scheduler to process the data processing tasks by scheduling the logical storage unit, thereby achieving the purpose of aggregating data by dividing the disk, so that the application program perceives the power-on and power-off scheduling information of the disk, thereby achieving the technical effect of improving the scheduling rate of data processing tasks, reducing scheduling costs and resource waste, and thus solving the technical problem of increased scheduling costs and resource waste in the energy-saving archiving system in the related technology due to the inability of the application program APP to perceive the power-on and power-off scheduling information of the disk.

[0103] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 6 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown, or having Figure 6 Different configurations are shown.

[0104] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0105] Example 5

[0106] According to an embodiment of the present invention, an embodiment of a computer-readable storage medium is also provided. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method for scheduling data processing tasks provided in the first embodiment.

[0107] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0108] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: sending a data processing task to a scheduler, wherein the data processing task includes at least: an archiving task and a readback task; obtaining a scheduling task and an operation attribute of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit.

[0109] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: dividing disk resources in multiple disk expansion cabinets into multiple of the above-mentioned logical storage units, wherein the above-mentioned multiple disk expansion cabinets are distributed in rack servers under multiple switches of the energy-saving archiving system.

[0110] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: dividing the above-mentioned disk resources into multiple above-mentioned logical storage units according to preset constraints, and the above-mentioned preset constraints include at least one of the following: a first constraint, a second constraint and a third constraint, wherein the first constraint is used to describe that the disk resources under the same logical storage unit correspond to the same switch; the second constraint is used to describe that the disk resources under the same logical storage unit correspond to a predetermined number of the above-mentioned rack servers; and the third constraint is used to describe that the number of disks contained in each logical storage unit is multiple.

[0111] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining the power-on and power-off planning tasks corresponding to each of the multiple logical storage units, and obtaining the operation attributes corresponding to each of the logical storage units, wherein the power-on and power-off planning tasks are used to plan the next power-on time interval of the logical storage unit.

[0112] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining the next power-on time interval of the above-mentioned logical storage unit determined by the above-mentioned power-on and power-off planning task; according to the next power-on time interval of the above-mentioned logical storage unit and the above-mentioned operation attributes corresponding to each of the above-mentioned logical storage units, correspondingly controlling the above-mentioned scheduler to execute scheduling of the above-mentioned logical storage unit to process the above-mentioned data processing task, and the content of the above-mentioned data processing task includes: application read processing, application write processing, and internal repair processing of the above-mentioned logical storage unit.

[0113] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: determining the priority of the above-mentioned data processing task; and scheduling the above-mentioned logical storage unit to process the above-mentioned data processing task according to the above-mentioned priority.

[0114] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0115] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or a predetermined number or more than a predetermined number of units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0120] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for scheduling a data processing task, comprising: Sending a data processing task to a scheduler, wherein the data processing task at least includes: an archiving task and a read-back task; Obtaining a scheduling task and an operation attribute of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets, the scheduling task includes a power-on and power-off planning task, and the operation attribute includes at least one of the following attributes: a read attribute, a write attribute, and an internal repair attribute; generating a control instruction based on the scheduling task and the operation attribute; A control instruction is sent to the scheduler, wherein the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit.

2. The method according to claim 1, wherein: The method is applied to an energy-saving archiving system. The energy-saving archiving system uses the logical storage unit as a scheduling unit and periodically powers on and off a predetermined portion of the logical storage units in all disk resources to achieve an energy-saving target.

3. The method according to claim 1, wherein: Before sending the data processing task to the scheduler, the method further includes: Disk resources in a plurality of disk expansion cabinets are divided into a plurality of the logical storage units, wherein the plurality of disk expansion cabinets are distributed in rack servers under a plurality of switches of the energy-saving archiving system.

4. The method according to claim 3, wherein: The dividing of the disk resources in the plurality of disk expansion cabinets into the plurality of the logical storage units comprises: The disk resources are divided into a plurality of the logical storage units according to preset constraints, and the preset constraints include at least one of the following: a first constraint, a second constraint, and a third constraint, wherein the first constraint is used to describe that the disk resources under the same logical storage unit correspond to a predetermined number of the rack servers, the second constraint is used to describe that each logical storage unit contains a plurality of disk resources, and the third constraint is used to describe that each of the logical storage units selects disk resources in each of the disk expansion cabinets.

5. The method according to claim 1, obtaining the scheduling tasks and operation attributes of the logical storage unit, comprising: The power-on and power-off planning tasks corresponding to each of the plurality of logic storage units are obtained, and the operation attributes corresponding to each of the logic storage units are obtained, wherein the power-on and power-off planning tasks are used to plan the next power-on time interval of the logic storage unit.

6. The method according to claim 1, controlling the scheduler to execute scheduling of the logical storage unit to process the data processing task according to the scheduling task and the operation attribute, comprises: Obtaining the next power-on time interval of the logical storage unit determined by the power-on and power-off planning task; According to the next power-on time interval of the logical storage unit and the operation attribute corresponding to each of the logical storage units, the scheduler is controlled to execute scheduling of the logical storage unit to process the data processing task, and the content of the data processing task includes: application read processing, application write processing, and internal repair processing of the logical storage unit.

7. The method according to claim 1, wherein: Controlling the scheduler to execute scheduling of the logical storage unit to process the data processing task includes: Determining the priority of the data processing task; According to the priority of the data processing task, the logical storage unit is scheduled to process the data processing task.

8. A data processing task scheduling device, comprising: A sending module, used for sending data processing tasks to the scheduler, wherein the data processing tasks at least include: an archiving task and a read-back task; An acquisition module is used to acquire a scheduling task and an operation attribute of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets, the scheduling task includes a power-on and power-off planning task, and the operation attribute includes at least one of the following attributes: a read attribute, a write attribute, and an internal repair attribute; A generating module, used for generating a control instruction based on the scheduling task and the operation attribute; A scheduling module is used to control the sending of control instructions to the scheduler, wherein the control instructions are used to control the scheduler to process the data processing task by scheduling the logical storage unit.

9. A computer-readable storage medium, the computer-readable storage medium comprising a stored program, wherein: A scheduling method for controlling the device where the computer-readable storage medium is located to execute the data processing task described in any one of claims 1 to 7 when the program is running.

10. A processor for running a program, wherein: The method for scheduling data processing tasks described in any one of claims 1 to 7 is executed when the program is running.

11. An electronic device comprising: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Step 1, sending a data processing task to a scheduler, wherein the data processing task at least includes: an archiving task and a read-back task; Step 2, obtaining a scheduling task and an operation attribute of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets, the scheduling task includes a power-on and power-off planning task, and the operation attribute includes at least one of the following: a read attribute, a write attribute, and an internal repair attribute; Step 3, generating a control instruction based on the scheduling task and the operation attribute; Step 4: Send a control instruction to the scheduler, wherein the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit.