Distributed task scheduling method, device, system and electronic equipment

Abstract

The present disclosure discloses a distributed task scheduling method, device and system and electronic equipment, relates to the technical field of cloud computing, and particularly relates to the technical field of resource scheduling. The technical problem of the distributed computing needing to access data from a database each time, and the frequent reading and writing causing a serious performance reduction is solved. The specific implementation scheme is as follows: obtaining a task to be processed; determining a processing module corresponding to the task, and determining an instance corresponding to the processing module; adding an identifier to the task, and sending the task after the addition to the instance, so that the instance distributes the task to the processing module based on the identifier, and the identifier is used for marking the processing module corresponding to the task and the instance corresponding to the task.

Description

Technical Field

[0001] This disclosure relates to the field of cloud computing technology, and more particularly to the field of resource scheduling technology, specifically to a method, apparatus, system, and electronic device for scheduling distributed tasks. Background Technology

[0002] With the development of computing technology, some applications require enormous computing power to complete, which would take a considerable amount of time if centralized computing were used. Distributed computing breaks down the application into many smaller parts and distributes them to multiple computers for processing. This can save overall computing time and improve computing efficiency.

[0003] For the above scenario, the relevant technologies cache user data in the database and use distributed locks to ensure the security of data access. However, each calculation requires retrieving data from the database and updating the data in the database after the calculation is completed. Frequent read and write operations will cause a serious performance degradation. At the same time, the calculation depends on the database, and if the database fails, the service needs to be degraded. Summary of the Invention

[0004] This disclosure provides a method, apparatus, system, and electronic device for scheduling distributed tasks.

[0005] According to a first aspect of this disclosure, a method for scheduling distributed tasks is provided, comprising:

[0006] Get the tasks to be processed;

[0007] Identify the processing module corresponding to the task, and identify the instance corresponding to the processing module, wherein each instance is configured with at least one processing module;

[0008] Add an identifier to the task and send the task to the instance so that the instance can assign the task to the processing module based on the identifier. The identifier is used to mark the processing module and the instance corresponding to the task.

[0009] According to a second aspect of this disclosure, a method for scheduling distributed tasks is provided, comprising:

[0010] Retrieve the task containing the identifier, which is used to mark the processing module and the instance corresponding to the task.

[0011] Tasks are assigned to the corresponding processing modules based on their identifiers.

[0012] The processing module is used to handle tasks.

[0013] According to a third aspect of this disclosure, a scheduling apparatus for distributed tasks is provided, comprising:

[0014] The first acquisition unit is used to acquire tasks to be processed.

[0015] The determining unit is used to determine the processing module corresponding to the task and to determine the instance corresponding to the processing module, wherein each instance is equipped with at least one processing module;

[0016] The first allocation unit is used to add an identifier to the task and send the added task to the instance so that the instance can allocate the task to the processing module based on the identifier. The identifier is used to mark the processing module and the instance corresponding to the task.

[0017] According to a fourth aspect of this disclosure, a scheduling apparatus for distributed tasks is provided, comprising:

[0018] The second acquisition unit is used to acquire a task containing an identifier, where the identifier is used to mark the processing module corresponding to the task and the instance corresponding to the task.

[0019] The second allocation unit is used to allocate tasks to the corresponding processing modules based on the identifier.

[0020] The processing unit is used to process tasks using the processing module.

[0021] According to a fifth aspect of this disclosure, a distributed task scheduling system is provided, including a distributed task scheduling apparatus as mentioned in the third aspect and a distributed task scheduling apparatus as mentioned in the fourth aspect.

[0022] According to a sixth aspect of this disclosure, an electronic device is provided, comprising:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in either the first or second aspect.

[0026] According to a seventh aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described in any one of the first or second aspects.

[0027] According to the eighth aspect of this disclosure, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in either the first or second aspect.

[0028] In the embodiments of this disclosure, a task to be processed is obtained, the processing module corresponding to the task is determined, and the instance corresponding to the processing module is identified. Finally, an identifier is added to the task, and the task with the identifier is sent to the instance. By providing a complete distributed task scheduling method, the task to be processed can be accurately allocated to the processing module that processes the task according to demand, improving the hardware utilization of distributed computing. At the same time, since the task is allocated and distributed to the processing module with independent computing capabilities for computation, it is not necessary to reload the data required for task computation every time distributed computing is performed, reducing the amount of data in distributed computing and thus effectively improving computing efficiency.

[0029] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0030] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0031] Figure 1 This is a flowchart illustrating a distributed task scheduling method provided according to an embodiment of the present disclosure;

[0032] Figure 2 This is a flowchart illustrating a method for determining a correspondence according to an embodiment of this disclosure;

[0033] Figure 3 This is a schematic diagram of a hash ring according to an embodiment of the present disclosure;

[0034] Figure 4 This is a flowchart illustrating another distributed task scheduling method provided according to an embodiment of the present disclosure;

[0035] Figure 5 This is a flowchart illustrating an instance update method provided according to an embodiment of the present disclosure;

[0036] Figure 6 This is a block diagram of a distributed task scheduling device provided according to an embodiment of the present disclosure;

[0037] Figure 7 This is a block diagram of another distributed task scheduling apparatus provided according to embodiments of the present disclosure;

[0038] Figure 8 This is a block diagram of an electronic device used to implement the distributed task scheduling method of the embodiments of this disclosure. Detailed Implementation

[0039] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0040] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0041] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0042] In related technologies, user data is cached in a database, and distributed locks are used to ensure the security of data access. However, each computation requires retrieving data from the database, and the data is updated back into the database after the computation is complete, thus isolating computation from data. This approach has drawbacks because all computations depend on the database. If the database fails, the service must be degraded, resulting in a significant performance degradation. Furthermore, the need to cache data in the database and the frequent read / write operations place high demands on the interface. If the amount of data in a single computation is large, data transmission, serialization, and deserialization will consume significant performance. Moreover, scalability is poor; when there are dependencies between different user data, the system complexity increases, making it difficult to support more complex computations.

[0043] Based on this, in the embodiments of this disclosure, the task to be processed is obtained, the processing module corresponding to the task is determined, and the instance corresponding to the processing module is determined. Finally, an identifier is added to the task, and the task with the added identifier is sent to the instance. By providing a complete distributed task scheduling method, the task to be processed can be accurately allocated to the processing module that processes the task according to demand, improving the hardware utilization of distributed computing. At the same time, since the task is allocated and distributed to the processing module with independent computing capabilities for computation, it is not necessary to reload the data required for task computation every time distributed computing is performed, reducing the amount of data in distributed computing and thus effectively improving computing efficiency.

[0044] The technical solutions provided by the embodiments of this disclosure are described below with reference to the accompanying drawings.

[0045] Figure 1 This is a flowchart illustrating a distributed task scheduling method provided in an embodiment of this disclosure. This method can be applied to the scheduling module of a distributed task scheduling system, such as... Figure 1 As shown, the method includes:

[0046] S110, retrieve tasks to be processed.

[0047] In this embodiment of the disclosure, the task module of the distributed task scheduling system obtains the task that needs to be processed. The task may include the calculation type, calculation method and calculation data of the task.

[0048] S120, determine the processing module corresponding to the task, and determine the instance corresponding to the processing module.

[0049] In this embodiment, the processing module for handling the task is determined based on the task type, and then the instance containing that processing module is determined. Specifically, the instance corresponding to the processing module is determined based on a correspondence. This allows for rapid instance determination based on the correspondence, eliminating the need for calculation each time, simplifying the calculation process and improving computational efficiency.

[0050] It should be noted that an instance refers to any hardware device with computing capabilities, or a virtual device set up based on a hardware device, which can be a personal computer, a smartphone, or a virtual system.

[0051] In this step, the system can also receive invocation information sent by the processing module. This invocation information is used to determine the target processing module and the target instance. Based on the invocation information, the system calls the target processing module for the processing module. In this way, when a processing module needs to perform collaborative work, the instance part can determine the target processing module for collaboration and generate invocation information to send to the scheduling module. The scheduling module can then use the invocation information to determine the target processing module and the target instance for the collaborative work for that processing module, without needing to perform recalculation.

[0052] The above-described method, based on invocation information, involves a processing module calling a target processing module. In practice, the target instance corresponding to the target processing module is determined according to the correspondence, and the invocation information is sent to the target instance of the instance module, enabling the target instance to invoke the target processing module. This eliminates the need to determine the corresponding instance through calculation each time, thus simplifying the calculation process.

[0053] S130, add an identifier to the task and send the added task to the instance so that the instance can assign the task to the processing module based on the identifier.

[0054] In this embodiment, step S120 determines the processing module and instance corresponding to the task, then adds an identifier to the task. This identifier indicates the correspondence between the task, processing module, and instance in the instance module. The task and identifier are then sent to the instance module, which then progressively distributes the task to the corresponding instance and processing module. This method allocates and distributes computational tasks to processing modules with independent computing capabilities, eliminating the need to reload the data required for basic computation each time distributed computing is performed, thus effectively improving computational efficiency.

[0055] Optionally, since the instance may lack or partially lack runtime data for the processing module, the instance can be checked before adding the identifier to determine if complete runtime data for the processing module exists. If it is missing or absent, the runtime data can be loaded and updated through a server or other means. Each processing module adopts an object-oriented design. The runtime data of the processing module is the basic layout data of the processing module, used to process tasks in the corresponding way. The completeness of the runtime data makes each processing module an independent and complete object with its own attributes and methods, providing access to the outside world, and also allowing access to other processing modules.

[0056] In some embodiments of this disclosure, such as Figure 2 As shown, the correspondence is determined according to the following method:

[0057] S210, determine the hash value corresponding to each instance.

[0058] In practice, the instance's address and port information determine its corresponding hash value. That is, the instance is uniquely identified by its IP address and port. After starting, the instance registers its IP address and port with the cloud, and it is deregistered from the cloud upon destruction. The cloud records the IP addresses and ports of all currently valid instances, and the instance retrieves information about all valid instances from the cloud. The hash value of an instance node is calculated as hash(IP + "_" + port) % 2^32, meaning the hash value of IP address + "_" + port is calculated first.

[0059] S220 maps the hash value corresponding to each instance to the hash ring.

[0060] In practice, the modulo operation is performed on 2^32 to map machine nodes onto a virtual hash ring consisting of 2^32 nodes.

[0061] S230, determine the hash value for each processing module.

[0062] In practice, the hash value of each processing module is determined based on user information and the calling method identifier. User information can be user ID, user information, etc. The calling method is the algorithm or processing method required for this task. Each processing module has a globally unique key, which can be represented by user (i.e., user information), calling method name or identifier, etc. For example, key = user Id + "_" + method name, and its node mapping value = hash(key) % 2^32.

[0063] S240, for each processing module, determine the instance corresponding to the processing module on the hash ring based on the hash value of the processing module.

[0064] In this step, based on the hash value of each processing module, a first mapping position of each processing module on the hash ring is determined. Then, based on the first mapping position, a second mapping position of the instance corresponding to that processing module on the hash ring is determined. Finally, based on the second mapping position, the instance corresponding to that processing module is determined. Specifically, when determining the second mapping position based on the first mapping position, the mapping position closest to the first mapping position can be used as the second mapping position, or the first mapping position encountered from any direction (such as clockwise or counterclockwise) starting from the first mapping position can be used as the second mapping position. This embodiment of the disclosure does not limit this.

[0065] The following example illustrates how the first instance in a clockwise direction is identified as the instance corresponding to the processing module. Figure 3 As shown, the hash ring has 2^32 virtual nodes, with the first node being 0 and the last node being 2^32-1, distributed clockwise. Instance nodes are mapped to nodes on the hash ring based on their mapping values. The process moves clockwise from the node position on the hash ring according to the mapping value of the processing module, and the first instance node encountered is its corresponding instance. This ensures that each processing module corresponds to an instance, simplifying the calculation process, ensuring accurate mapping, reducing the possibility of allocation errors, and improving the reliability of distributed computing.

[0066] This step also allows obtaining instance change information sent by the instance module. This information includes the address and port information of the changed instance. Based on this address and port information, the hash value corresponding to the changed instance is determined, and then each hash value is mapped to a hash ring. This provides an operational solution for changing instance information without requiring a complete replanning of all instances, improving efficiency and flexibility.

[0067] In this step, we can also obtain instance deletion information sent by the instance module. This information includes the address and port information of the deleted instance. Then, we remove the mapping of the deleted instance from the hash ring. This provides an operational solution for deleting instance information without having to re-plan all instances, improving efficiency and flexibility.

[0068] In this embodiment of the disclosure, by calculating the hash values ​​of the instance and the processing module respectively, a correspondence between the instance and the processing module is generated using a hash ring. This allows for direct matching on the hash ring when determining the instance corresponding to the processing module, eliminating the need for each lookup and calculation, thus improving the computational efficiency of distributed computing.

[0069] Figure 4 This is a flowchart illustrating another distributed task scheduling method provided in this disclosure. This method can be applied to instance modules in a distributed task scheduling system. The method includes:

[0070] S410, retrieve the task containing the identifier.

[0071] In this embodiment of the disclosure, the scheduling module of the distributed task scheduling system obtains a task containing an identifier. The task may also contain the task's calculation type, calculation method, and calculation data. The identifier is used to mark the processing module corresponding to the task and the instance corresponding to the task.

[0072] S420 assigns tasks to the corresponding processing modules based on identifiers.

[0073] In this embodiment, the instance corresponding to the processing module is determined based on the correspondence, and the task is sent to the instance. The instance performs a self-check to determine if it contains complete data for the processing module, and then the task is assigned to that processing module. If the instance does not contain complete data for the processing module, the complete data can be obtained and loaded from the server to ensure the integrity of the processing module's runtime data and its ability to process tasks. The determination of the correspondence can be handled partially by the instance module or partially by the scheduling module. This simplifies the calculation process, allowing the instance to be determined directly based on the correspondence, eliminating the need for calculation to determine the corresponding instance each time.

[0074] S430 uses a processing module to handle tasks.

[0075] In this embodiment of the disclosure, the processing module performs calculations after receiving the corresponding task. This allocates and distributes the calculation tasks to processing modules with independent computing capabilities, eliminating the need to reload the data required for basic calculations each time distributed computing is performed, thereby effectively improving computational efficiency.

[0076] In this step, when a processing module requests assistance with a task, the target processing module is determined based on the task. The target instance corresponding to the target processing module is then identified according to the corresponding relationship. If the target instance differs from the target instance, a call message is sent to the scheduling module. This call message instructs the scheduling module to call the target processing module and the target instance. The scheduling module then calls the target processing module of the target instance. Specifically, when the task side calls a processing module or a processing module calls another processing module, the corresponding instance is found using the consistent hashing algorithm described above. If it is the current instance, a local call is made; if it is a remote instance, a remote call is made based on the IP address and port obtained from the cloud. When a call hits an instance, if the processing module is not initialized, it is initialized first before processing. Initialization includes loading data from the database. This provides a solution for processing tasks requiring collaboration from other processing modules. When other processing modules need to be called for collaborative processing, the required processing module and its corresponding instance are determined directly at the instance level, simplifying the allocation and calculation process and improving efficiency. If the target instance is the same as the instance, the target processing module is called directly on the instance. In this way, if the target instance is the same as the instance, there is no need to allocate it again through the scheduling module. The allocation is performed directly on the local machine, saving the calculation process and improving the efficiency of distributed computing.

[0077] In some embodiments of this disclosure, such as Figure 5 As shown, when adding or updating instances, the following method is used to determine the process:

[0078] S510, identify instances with changed address and port information and / or newly added instances as changed instances.

[0079] Of course, when an instance needs to be deleted, the instance to be deleted is identified, and then the instance deletion information is determined based on the address and port information of the deleted instance. This information is then sent to the scheduling module so that the scheduling module can update the corresponding relationships. In this way, a data update scheme for deleting instances is provided, which does not require replanning all instances, thus improving efficiency and flexibility.

[0080] S520 determines the instance change information based on the address and port information of the changed instance, and sends the instance deletion information to ensure the change correspondence.

[0081] In this step, the running data of the corresponding processing module for each instance can also be updated periodically according to the correspondence. This ensures that the running data can be updated regularly, thus guaranteeing the accuracy of the distributed system data.

[0082] In this embodiment of the disclosure, data updates are implemented when adding / deleting / updating instances. This provides a solution for instance changes without the need to replan all instances, thereby improving the accuracy, efficiency, and flexibility of the distributed computing system.

[0083] The solution provided in this disclosure significantly improves computation speed, eliminating the performance overhead of interface, serialization, and deserialization. Furthermore, by adopting an object-oriented design, the minimum processing module encapsulates a complete object, which is an independent logical unit and globally unique, with only one instance existing at a time. The processing module also encapsulates inter-service calls; calls between objects do not need to concern themselves with whether the target object is local or remote. Calling a remote object is the same as calling a local object, allowing the processing module to focus solely on task logic processing, thus facilitating task expansion.

[0084] Based on the same inventive concept, this disclosure also provides a scheduling device for distributed tasks, such as... Figure 6 As shown, the distributed task scheduling device 600 may include:

[0085] The first acquisition unit 601 is used to acquire the task to be processed;

[0086] The determining unit 602 is used to determine the processing module corresponding to the task and to determine the instance corresponding to the processing module, wherein each instance is provided with at least one processing module;

[0087] The first allocation unit 603 is used to add an identifier to a task and send the added task to an instance so that the instance can allocate the task to a processing module based on the identifier. The identifier is used to mark the processing module and the instance corresponding to the task.

[0088] In one possible implementation, the determining unit 602 in the apparatus provided by the embodiments of the present invention is specifically used for:

[0089] The corresponding instance of the processing module is determined based on the correspondence relationship, which represents the correspondence between the processing module and the instance.

[0090] In one possible implementation, the determining unit 602 in the apparatus provided by the present invention is specifically used to determine the correspondence according to the following method:

[0091] Determine the hash value corresponding to each instance;

[0092] Map the hash value corresponding to each instance to a hash ring;

[0093] Determine the hash value for each processing module;

[0094] For each processing module, the corresponding instance of the processing module is determined on the hash ring based on the hash value of the processing module.

[0095] In one possible implementation, the determining unit 602 in the apparatus provided by the embodiments of the present invention is further configured to:

[0096] For each instance, based on its address and port information, determine the hash value corresponding to that instance; and / or,

[0097] Determining the hash value for each processing module includes:

[0098] The hash value of each processing module is determined based on user information and the method call identifier.

[0099] In one possible implementation, the determining unit 602 in the apparatus provided by the embodiments of the present invention is further configured to:

[0100] Retrieve instance change information, which includes the address of the changed instance and the port information of the changed instance;

[0101] The hash value corresponding to the changed instance is determined based on the address and port information of the changed instance;

[0102] Map the hash value corresponding to the changed instance to the hash ring.

[0103] In one possible implementation, the determining unit 602 in the apparatus provided by the embodiments of the present invention is further configured to:

[0104] Retrieve instance deletion information, which includes the address and port information of the deleted instance.

[0105] Remove the mapping of the deleted instance from the hash ring.

[0106] In one possible implementation, the determining unit 602 in the apparatus provided by the embodiments of the present invention is further configured to:

[0107] For each processing module, determine the first mapping position of the processing module on the hash ring based on the hash value of the processing module;

[0108] The second mapping position of the instance corresponding to the processing module on the hash ring is determined based on the first mapping position.

[0109] The instance corresponding to the processing module is determined based on the second mapping position.

[0110] In one possible implementation, the apparatus provided in this embodiment of the invention further includes:

[0111] The receiving unit is used to receive the call information sent by the processing module. The call information is used to determine the target processing module.

[0112] The determining unit 602 is also used to call the target processing module for the processing module based on the calling information.

[0113] In one possible implementation, the determining unit 602 in the apparatus provided by the embodiments of the present invention is further configured to:

[0114] Determine the target instance corresponding to the target processing module based on the correspondence relationship;

[0115] The call information is sent to the target instance so that the processing module can call the target processing module through the target instance.

[0116] Based on the same inventive concept, this disclosure also provides another scheduling device for distributed tasks, such as... Figure 7 As shown, the distributed task scheduling device 700 may include:

[0117] The second acquisition unit 701 is used to acquire a task containing an identifier, wherein the identifier is used to mark the processing module corresponding to the task and the instance corresponding to the task.

[0118] The second allocation unit 702 is used to allocate tasks to the corresponding processing modules based on the identifier.

[0119] Processing unit 703 is used to process tasks using the processing module.

[0120] In one possible implementation, the second allocation unit 702 in the apparatus provided by the present invention is specifically used for:

[0121] The identifier determines the processing module and the instance corresponding to the task.

[0122] The task is sent to the instance, and the instance assigns it to the processing module.

[0123] In one possible implementation, the processing unit 703 in the apparatus provided by the embodiments of the present invention is further configured to:

[0124] Identify instances whose address and port information have changed and / or newly added instances as changed instances;

[0125] The instance change information is determined based on the address and port information of the changed instance and sent to the scheduling module so that the scheduling module can change the corresponding relationship.

[0126] In one possible implementation, the processing unit 703 in the apparatus provided by the embodiments of the present invention is further configured to:

[0127] The instance to be deleted is confirmed to be the instance to be deleted;

[0128] Determine instance deletion information based on the address and port information of the deleted instance;

[0129] Send instance deletion information to change the corresponding relationship.

[0130] In one possible implementation, the processing unit 703 in the apparatus provided by the embodiments of the present invention is further configured to:

[0131] When a processing module requests assistance in processing a task, the target processing module for the assistance module is determined based on the task.

[0132] Determine the target instance corresponding to the target processing module based on the correspondence relationship;

[0133] When the target instance is different from the instance, a call information is sent to the scheduling module. The call information is used to indicate that the processing module should call the target processing module.

[0134] In one possible implementation, the processing unit 703 in the apparatus provided by the embodiments of the present invention is further configured to:

[0135] When the target instance is the same as the instance, the target processing module is called on the instance.

[0136] In one possible implementation, the processing unit 703 in the apparatus provided by the embodiments of the present invention is further configured to:

[0137] Update the runtime data of the corresponding processing module for each instance based on the correspondence.

[0138] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0139] Based on the same inventive concept, this disclosure also provides a distributed task scheduling system, which includes, for example, a distributed task scheduling system. Figure 6 The distributed task scheduling device shown, and as Figure 7 The distributed task scheduling device shown is illustrated.

[0140] According to embodiments of this disclosure, this disclosure also provides an electronic device, a non-transitory computer-readable storage medium, and a computer program product.

[0141] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0142] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0143] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0144] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as distributed task scheduling methods. For example, in some embodiments, the distributed task scheduling methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the deep learning compiler running method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform distributed task scheduling methods by any other suitable means (e.g., by means of firmware).

[0145] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0149] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0150] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") in terms of management difficulty and weak task scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0151] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0152] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for scheduling distributed tasks, characterized in that, The method is applied to the scheduling module of a distributed task scheduling system, including: Get the tasks to be processed; The processing module corresponding to the task is determined, and the instance corresponding to the processing module is determined, wherein each instance is equipped with at least one processing module; An identifier is added to the task, and the task is sent to the instance so that the instance can assign the task to the processing module based on the identifier. The identifier is used to mark the processing module corresponding to the task and the instance corresponding to the task. The processing module adopts an object-oriented design, and the instance contains the running data of the processing module. The running data is used to process the task in a corresponding way. The completeness of the running data makes the processing module an independent and complete object with its own attributes and methods.

2. The method according to claim 1, characterized in that, Determining the instance corresponding to the processing module includes: The instance corresponding to the processing module is determined based on the correspondence relationship, which represents the correspondence between the processing module and the instance.

3. The method according to claim 2, characterized in that, The correspondence is determined according to the following method: Determine the hash value corresponding to each instance; Map the hash value corresponding to each instance to a hash ring; Determine the hash value for each processing module; For each processing module, the instance corresponding to that processing module is determined on the hash ring based on the hash value of that processing module.

4. The method according to claim 3, characterized in that, Determining the hash value corresponding to each instance includes: For each instance, based on its address and port information, determine the hash value corresponding to that instance; and / or, Determining the hash value of each processing module includes: The hash value of each processing module is determined based on user information and the method call identifier.

5. The method according to claim 4, characterized in that, Before determining the instance corresponding to each processing module on the hash ring based on its hash value, the method further includes: Obtain instance change information, which includes the address of the changed instance and the port information of the changed instance; The hash value corresponding to the change instance is determined based on the address of the change instance and the port information of the change instance; Map the hash value corresponding to the changed instance to the hash ring.

6. The method according to claim 4, characterized in that, Before determining the instance corresponding to each processing module on the hash ring based on its hash value, the method further includes: Obtain instance deletion information, which includes the address of the deleted instance and the port information of the deleted instance; Delete the mapping associated with the deleted instance in the hash ring.

7. The method according to any one of claims 3-6, characterized in that, For each processing module, determining the corresponding instance of that processing module on the hash ring based on its hash value includes: For each processing module, the first mapping position of the processing module on the hash ring is determined based on the hash value of the processing module; Based on the first mapping position, determine the second mapping position of the instance corresponding to the processing module on the hash ring; The instance corresponding to the processing module is determined based on the second mapping position.

8. The method according to claim 2, characterized in that, The method further includes: Receive the call information sent by the processing module, the call information being used to determine the target processing module; Based on the invocation information, the target processing module is invoked by the processing module.

9. The method according to claim 8, characterized in that, The step of calling the target processing module for the processing module based on the calling information includes: The target instance corresponding to the target processing module is determined based on the correspondence. The invocation information is sent to the target instance so that the processing module can invoke the target processing module through the target instance.

10. A method for scheduling distributed tasks, characterized in that, The method is applied to instances of distributed task scheduling systems, including: The task containing an identifier is obtained from the scheduling module of the distributed task scheduling system. The identifier is used to mark the processing module corresponding to the task and the instance corresponding to the task. The identifier is added by the scheduling module. The task is assigned to the corresponding processing module based on the identifier; wherein the processing module adopts an object-oriented design, the instance contains the running data of the processing module, the running data is used to process the task in a corresponding manner, and the completeness of the running data makes the processing module an independent and complete object with its own attributes and methods; The task is processed using the processing module.

11. The method according to claim 10, characterized in that, The step of assigning the task to the corresponding processing module based on the identifier includes: The processing module corresponding to the task and the instance corresponding to the task are determined based on the identifier; The task is sent to the instance, and the instance assigns it to the processing module.

12. The method according to claim 10, characterized in that, The method further includes: Identify instances whose address and port information have changed and / or newly added instances as changed instances; The instance change information is determined based on the address and port information of the changed instance, and then sent to the scheduling module so that the scheduling module can change the corresponding relationship.

13. The method according to claim 10, characterized in that, The method further includes: The instance to be deleted is confirmed to be the instance to be deleted; The instance deletion information is determined based on the address and port information of the deleted instance; Send the instance deletion information to change the corresponding relationship.

14. The method according to claim 11, characterized in that, The method further includes: When the processing module requests assistance in processing a task, a target processing module is determined to assist the processing module based on the task. The target instance corresponding to the target processing module is determined based on the correspondence relationship; When the target instance is different from the instance, a call message is sent to the scheduling module, and the call message is used to indicate that the processing module calls the target processing module.

15. The method according to claim 14, characterized in that, The method further includes: When the target instance is the same as the instance, the target processing module is invoked on the instance.

16. The method according to claim 11, characterized in that, The method further includes: Update the runtime data of the corresponding processing module for each instance based on the correspondence.

17. A distributed task scheduling device, characterized in that, The device is applied to the scheduling module of a distributed task scheduling system, and includes: The first acquisition unit is used to acquire tasks to be processed. A determining unit is used to determine the processing module corresponding to the task and to determine the instance corresponding to the processing module, wherein each instance is provided with at least one processing module; The first allocation unit is used to add an identifier to the task and send the added task to the instance, so that the instance allocates the task to the processing module based on the identifier. The identifier is used to mark the processing module corresponding to the task and the instance corresponding to the task. The processing module adopts an object-oriented design. The instance contains the running data of the processing module. The running data is used to process the task in a corresponding way. The completeness of the running data makes the processing module an independent and complete object with its own attributes and methods.

18. A distributed task scheduling device, characterized in that, An example of the device being applied in a distributed task scheduling system includes: The second acquisition unit is used to acquire a task containing an identifier from the scheduling module of the distributed task scheduling system. The identifier is used to mark the processing module corresponding to the task and the instance corresponding to the task. The identifier is added by the scheduling module. The second allocation unit is used to allocate the task to the processing module corresponding to the task based on the identifier; wherein the processing module adopts an object-oriented design, the instance contains the running data of the processing module, the running data is used to process the task in a corresponding manner, and the completeness of the running data makes the processing module an independent and complete object with its own attributes and methods; A processing unit is used to process the task using the processing module.

19. A distributed task scheduling system, characterized in that, The system includes the distributed task scheduling device as described in claim 17, and the distributed task scheduling device as described in claim 18.

20. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the distributed task scheduling method as described in any one of claims 1 to 9 or 10 to 16.

21. A computer storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a scheduling method for a distributed task as described in any one of claims 1 to 9 or 10 to 16.

22. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9 or 10 to 16.

Images