Distributed resource scheduling method, device and system

Abstract

The present disclosure provides a distributed resource scheduling method, device and system, which relate to autonomous driving technology in artificial intelligence and can be applied to high-precision maps. The method is applied to electronic devices in a cluster, and the cluster to which the electronic devices belong is an online cluster. The method includes: obtaining a directed acyclic graph of tasks to be processed, and determining the resource requirements for executing the processing flow indicated by the directed acyclic graph; if the current available resources of the electronic device do not meet the resource requirements, resources are retrieved from offline devices, and the processing flow indicated by the directed acyclic graph is executed based on at least the retrieved resources, thereby realizing resource calls between online electronic devices and offline devices, thereby improving the efficiency of online electronic devices in processing online tasks, and achieving the technical effect of large-scale task scheduling.

Description

Technical Field

[0001] The present disclosure relates to autonomous driving technology in artificial intelligence (AI), which can be applied to high-precision maps, and in particular to a distributed resource scheduling method, device and system. Background Art

[0002] Airflow is an open-source distributed task scheduling framework that can be deployed in device clusters (such as Kubernetes (also known as K8s) clusters) to complete data processing tasks (such as completing electronic map creation tasks) by scheduling resources across the cluster's devices. The distributed task scheduling framework includes the following components: a scheduler, a task queue, and a worker process.

[0003] In the prior art, a scheduler obtains a data processing request, such as a request to generate an electronic map, determines the tasks for generating the electronic map based on a directed acyclic graph in the data processing request, adds each task to a task queue, creates and calls a work process to execute the tasks in the task queue, and thus generates the electronic map.

[0004] However, the above method has low efficiency in completing data processing requests and is unable to handle large-scale task scheduling. Summary of the Invention

[0005] The present disclosure provides a method, device, and system for distributed resource scheduling.

[0006] According to a first aspect of the present disclosure, a distributed resource scheduling method is provided. The method is applied to electronic devices in a cluster, where the cluster to which the electronic devices belong is an online cluster. The method includes:

[0007] Obtaining a directed acyclic graph of tasks to be processed, and determining resource requirements for executing a processing flow indicated by the directed acyclic graph;

[0008] If the current available resources of the electronic device do not reach the resource requirement, resources are retrieved from the offline device, and the processing flow indicated by the directed acyclic graph is executed at least based on the retrieved resources.

[0009] According to a second aspect of the present disclosure, a distributed resource scheduling apparatus is provided. The apparatus is applied to an electronic device in a cluster, where the cluster to which the electronic device belongs is an online cluster. The apparatus includes:

[0010] An acquisition unit, used to acquire a directed acyclic graph of tasks to be processed;

[0011] a determining unit, configured to determine a resource requirement for executing the processing flow indicated by the directed acyclic graph;

[0012] a retrieving unit, configured to retrieve resources from an offline device if the current available resources of the electronic device do not meet the resource requirement;

[0013] An execution unit is configured to execute the processing flow indicated by the directed acyclic graph at least according to the retrieved resources.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0018] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0019] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect.

[0020] According to the sixth aspect of the present disclosure, a distributed resource scheduling system is provided, comprising: an electronic device and an offline device, wherein the electronic device is a device in a cluster, and the cluster to which the electronic device belongs is an online cluster, and the electronic device includes the apparatus as described in the second aspect.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0023] Figure 1This is a scenario diagram of a distributed resource scheduling method that can implement an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;

[0025] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0026] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0028] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0029] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0030] Figure 8 It is a block diagram of an electronic device used to implement the distributed resource scheduling method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] For example, the distributed resource scheduling method of this embodiment can be applied to Figure 1 The application scenario shown in Figure 2 is as follows. Figure 1 As shown, the computer device is a device in the device cluster, and the computer device deploys a distributed task scheduling framework (airflow).

[0033] like Figure 1 As shown in the figure, the distributed task scheduling framework includes:

[0034] The scheduler is used to poll a directed acyclic graph (a directed acyclic graph for completing pending tasks such as those initiated by the client) from a database (not shown in the figure, the database can be a database in a computer device or data in other devices, which is not limited in this embodiment), generate tasks corresponding to the directed acyclic graph, and add each task to the task queue.

[0035] The task queue is used to store tasks transmitted by the scheduler.

[0036] Worker process, used to execute tasks in the task queue, such as Figure 1 Worker process 1 and worker process 2 are shown in the figure. Figure 1 The two working processes are only used for demonstration purposes and should not be understood as limiting the number of working processes.

[0037] It is understandable that this embodiment does not limit the specific content of the task to be processed. For example, the task to be processed may be used to instruct the generation of an electronic map, or may be used to instruct the training of a network model, etc., which are not listed here one by one.

[0038] For example, if the pending task is to generate an electronic map, the directed acyclic graph is used to indicate the processing flow of generating the electronic map. Correspondingly, if the pending task is to train a network model, the directed acyclic graph is used to indicate the processing flow of training the network model.

[0039] In related technologies, each task is performed by a work process in a computer device, but the resources of the computer device are limited. Especially when there are multiple directed acyclic graphs and many tasks need to be processed, there is a technical problem of low efficiency.

[0040] In order to avoid the above technical problems, the inventors of the present disclosure have obtained the inventive concept of the present disclosure through creative work: if an electronic device (such as Figure 1 When the current available resources of the computer device described in the embodiment do not reach the resource requirements for processing each task, resources are transferred from the offline device to execute the tasks in the task queue based on the transferred resources.

[0041] Based on the above-mentioned inventive concept, the present disclosure provides a distributed resource scheduling method, device and system, which are applied to autonomous driving technology in artificial intelligence, and can be specifically applied to high-precision maps to achieve resource utilization and support large-scale tasks to be processed.

[0042] Figure 2 The distributed resource scheduling method of the embodiment of the present disclosure is applied to electronic devices in a cluster, and the cluster to which the electronic devices belong is an online cluster, such as Figure 2 As shown, the method includes:

[0043] S201: Obtain a directed acyclic graph of tasks to be processed, and determine resource requirements for executing a processing flow indicated by the directed acyclic graph.

[0044] Exemplarily, the execution subject of this embodiment may be a distributed resource scheduling device, which may be an electronic device in an online cluster, and the electronic device may be a computer, a server, a processor, a chip, etc., which is not limited in this embodiment.

[0045] Based on the above analysis, a directed acyclic graph is a graph used to indicate a processing flow, that is, the various processing flows that need to be executed in order to complete a task to be processed, such as the various processing flows that need to be executed in order to generate an electronic map.

[0046] Specifically, the directed acyclic graph may refer to a series of processing flows for generating a base map based on map data, rendering the base map, and so on, thereby generating an electronic map.

[0047] Resource requirements refer to the amount of resources needed to complete the processing flow indicated by the directed acyclic graph, such as how many resources are needed to execute the series of processing flows to generate an electronic map, thereby obtaining the electronic map.

[0048] S202: If the current available resources of the electronic device do not meet the resource requirements, resources are retrieved from the offline device.

[0049] The available resource amount refers to the difference between the total resources of the electronic device and the currently occupied resources, that is, the amount of resources currently remaining and available for use.

[0050] In related technologies, considering the timeliness and currentness of online cluster processing tasks, the work process generally executes the tasks in the task queue in sequence until the pending tasks are completed, such as generating an electronic map. Therefore, calling resources from other electronic devices will not be considered to avoid the disadvantages of reducing the resources of other electronic devices to process pending tasks.

[0051] However, in this embodiment, a technical feature of retrieving resources from offline devices is introduced, which can avoid the impact of resource retrieval on other electronic devices in the online cluster and realize flexible resource calling.

[0052] S203: Execute the processing flow indicated by the directed acyclic graph at least according to the retrieved resources.

[0053] That is to say, after the resource call operation through S202, the processing flow indicated by the directed acyclic graph can be executed only based on the called resources, or the processing flow indicated by the directed acyclic graph can be executed in combination with the called resources, such as executing the processing flow indicated by the directed acyclic graph in combination with the called resources and available resources.

[0054] For example, tasks in the task queue can be executed based on work process 3 and process 4 (i.e., the resources called), or tasks in the task queue can be executed based on work process 1 and work process 2 (resources in the computer device), and work process 3 and process 4 (i.e., the resources called).

[0055] Based on the above analysis, an embodiment of the present disclosure provides a distributed resource scheduling method, which is applied to electronic devices in a cluster, and the cluster to which the electronic devices belong is an online cluster. The method includes: obtaining a directed acyclic graph of the tasks to be processed, and determining the resource requirements for executing the processing flow indicated by the directed acyclic graph. If the current available resources of the electronic device do not meet the resource requirements, resources are called from offline devices, and the processing flow indicated by the directed acyclic graph is executed at least based on the called resources. In this embodiment, a technical feature is introduced: when the available resources of the online electronic device are insufficient, resources are called from offline devices to execute the processing flow indicated by the directed acyclic graph at least based on the called resources. Resource calls between online electronic devices and offline devices are realized, thereby improving the efficiency of online electronic devices in processing online tasks, and achieving the technical effect of large-scale task scheduling.

[0056] Figure 3 The distributed resource scheduling method of the embodiment of the present disclosure is applied to electronic devices in a cluster, and the cluster to which the electronic devices belong is an online cluster, such as Figure 3 As shown, the method includes:

[0057] S301: In response to monitoring a task processing request, obtaining a processing flow for completing the task to be processed according to the task to be processed carried in the task processing request.

[0058] The present embodiment of the present disclosure does not elaborate on the technical features that are the same as those in the above-mentioned embodiments. Based on the above analysis, it can be seen that the present embodiment can be applied to scenarios such as electronic map generation and network model training. To facilitate the reader's understanding of the principles of the present embodiment of the present disclosure, the present embodiment uses electronic map generation as an example for illustrative description. Other scenarios (such as network model training) are not listed here because they are similar in principle.

[0059] In this embodiment, the electronic device can monitor task processing requests, such as monitoring whether the client has a request to initiate a task to be processed, such as a request to generate an electronic map, and obtain the processing flow of generating the electronic map when monitoring the task processing request.

[0060] In some embodiments, the electronic device is provided with an interface service (Api Server); S301 may include: monitoring according to the interface service, and if a task processing request is monitored, the interface service is used to obtain the processing flow for completing the pending task according to the pending task carried in the task processing request, and convert the processing flow for completing the pending task into a processing flow recognizable by the electronic device, and generate a directed acyclic graph according to the converted processing flow.

[0061] For example, Figure 1 As shown, an interface service is set on the computer device, and the interface service listens to the task processing request initiated by the client. When receiving the task of generating an electronic map, the processing flow of generating the electronic map is obtained, the flow is converted into a processing flow that can be recognized by the computer device, and a directed acyclic graph is generated according to the processing flow that can be recognized by the computer device.

[0062] It is worth noting that, in this embodiment, by setting an interface service to generate a directed acyclic graph based on the interface service, the technical effect of reducing the access cost when obtaining the directed acyclic graph from the outside is achieved.

[0063] S302: Converting a processing flow for completing the task to be processed into a processing flow recognizable by the electronic device, and generating a directed acyclic graph according to the converted processing flow.

[0064] For example, the electronic device converts the processing flow of generating an electronic map according to the recognition capability of the electronic device, such as the language that the electronic device can recognize, to convert it into a processing flow of a language that the electronic device can recognize, and generates a directed acyclic graph, which can be directly and effectively recognized by the electronic device.

[0065] It is worth noting that in related technologies, the client generally carries a directed acyclic graph in the task processing request when initiating the task processing request. In this embodiment, the electronic device generates a directed acyclic graph based on its recognition capability. Compared with related technologies, the access cost when accessing the directed acyclic graph can be reduced, and the flexibility and diversity of generating electronic maps can be improved. Technical effects.

[0066] In some embodiments, after generating the directed acyclic graph, the interface service may transmit the directed acyclic graph to a database, which may be a database in the electronic device or a database in other electronic devices, and this embodiment does not limit this.

[0067] Accordingly, the electronic device can poll the database. If a directed acyclic graph is found during the poll, the electronic device obtains the directed acyclic graph and executes the distributed resource scheduling method described in the first embodiment to implement resource scheduling and generate an electronic map.

[0068] S303: Generate a directed acyclic graph task according to the processing flow indicated by the directed acyclic graph.

[0069] S304: Mark the identifier of the directed acyclic graph as the identifier of the directed acyclic graph task, and add the task marked with the identifier of the directed acyclic graph to a preset task queue.

[0070] Among them, the directed acyclic graph has an identity.

[0071] For example, combined Figure 1 According to the above analysis, the scheduler polls the database. If a directed acyclic graph is obtained through polling, a task of generating the directed acyclic graph is generated, such as a task of generating an electronic map. The identifier of the directed acyclic graph is marked as the identifier of the task, and the task marked with the identifier is added to the task queue.

[0072] It should be noted that an electronic device may need to process multiple pending tasks. For example, when the electronic device polls a new directed acyclic graph when one or more previous pending tasks have not been completed, an identifier can be assigned to each directed acyclic graph in order to distinguish different directed acyclic graphs.

[0073] Accordingly, in order to distinguish tasks corresponding to different directed acyclic graphs, each task may be identified, and each task may be labeled based on its corresponding directed acyclic graph, thereby obtaining an identification of each task.

[0074] In this embodiment, tasks corresponding to the directed acyclic graph are marked by an identifier based on the directed acyclic graph so as to distinguish the directed acyclic graphs corresponding to each task in the task queue, thereby achieving the technical effect of improving the effectiveness, accuracy and reliability of task execution when the tasks in the task queue are subsequently executed based on the called resources.

[0075] S305: Determine the type attribute of the directed acyclic graph. The type attribute is used to indicate whether to retrieve resources from the offline device to execute the processing flow indicated by the directed acyclic graph.

[0076] S306: If the type attribute is used to indicate not retrieving resources from the offline device to execute the processing flow indicated by the directed acyclic graph, determine the resource requirement for executing the processing flow indicated by the directed acyclic graph based on the directed acyclic graph.

[0077] It should be understood that different directed acyclic graphs have different type attributes. For example, the directed acyclic graph of an electronic map has a type attribute, and the directed acyclic graph of a training network model also has a type attribute. The type attribute of the directed acyclic graph of an electronic map may be different from the type attribute of the directed acyclic graph of a training network model.

[0078] The type attribute may be determined based on the category of the directed acyclic graph, and is used to indicate whether to retrieve resource information from the offline device. The category of the directed acyclic graph may be determined based on the size, field, etc. of the directed acyclic graph.

[0079] In this embodiment, combined with the above analysis and Figure 1 When the scheduler polls the directed acyclic graph for generating the electronic map, on the one hand, it can generate a task for the directed acyclic graph and add the generated task to the task queue (see the above description); on the other hand, it can transmit the directed acyclic graph to the application management unit (APP master, AM) so that the application management unit can determine the type attribute of the directed acyclic graph and, when the type attribute of the directed acyclic graph indicates not to call resources from offline devices, determine the resource requirement for executing the processing flow indicated by the directed acyclic graph.

[0080] In some embodiments, the application management unit also determines the type attribute of the directed acyclic graph for the created work process.

[0081] It should be noted that one application management unit corresponds to one directed acyclic graph. To achieve reasonable utilization of resources, after the pending tasks corresponding to the directed acyclic graph, such as electronic map generation, are completed, the resources of the application management unit are released.

[0082] The type attribute may be used to indicate that resources are not to be retrieved from offline devices, and the way this embodiment handles this situation is to determine the resource demand based on a directed acyclic graph. In other embodiments, if the type attribute is used to indicate that resources are not to be retrieved from offline devices, then resources may not be retrieved from offline devices, and the resources of the resource electronic device will execute the tasks in the task queue in sequence to complete different tasks to be processed.

[0083] Correspondingly, in other embodiments, the type attribute may also be used to indicate that resources are to be retrieved from an offline device, and then the resources may be retrieved from the offline device to execute the tasks in the task queue.

[0084] It is worth noting that in this embodiment, by determining whether to retrieve resources from offline devices based on type attributes, the flexibility of resource retrieval is achieved, and the actual combination of resource retrieval and application scenarios is met, thereby achieving the technical effect of the effectiveness and diversity of resource retrieval.

[0085] S307: Send a resource retrieval request to the offline device, wherein the resource retrieval request is used to retrieve resources from the offline device, and the resource retrieval request carries an identifier of a directed acyclic graph.

[0086] S308: Execute the directed acyclic graph task based on the resources provided by the offline device according to the identifier of the directed acyclic graph to complete the task to be processed.

[0087] Combining the above analysis and Figure 1 , the application management unit sends a resource acquisition request to the offline electronic device, specifically, it can send the resource acquisition request to the resource manager (RM) of the offline device.

[0088] Accordingly, after receiving the resource call request, the resource manager can create a worker thread and transmit the identifier of the directed acyclic graph to the created worker thread, so that the created worker thread executes the task in the task queue that has the same identifier as the identifier of the directed acyclic graph based on the identifier of the directed acyclic graph. The number of worker threads created by the resource manager is not limited in this embodiment.

[0089] It is worth noting that the embodiments of the present disclosure do not limit "resources", such as there is no limitation on the types of resources called by the electronic device. In some embodiments, the called resources may be process resources, in other embodiments, the called resources may be processor resources, in still other embodiments, the called resources may be network resources, and so on, which are not listed one by one here.

[0090] For example, combining the above analysis and Figure 1 , the application management unit can send a resource call request to the resource manager to call process resources from the offline device. For another example, the application management unit can send a resource call request to its deep learning platform (PaddleCloud) to call processor (GPU or CPU) resources from the deep learning platform.

[0091] In this embodiment, the electronic device can access different resources, thereby achieving the technical effect of diversity and flexibility in resource access.

[0092] In this embodiment, by combining the retrieved resources and the identifier of the directed acyclic graph to complete the execution of the tasks corresponding to the directed acyclic graph in the task queue, the technical effect of accuracy and reliability of executing tasks using the retrieved resources can be achieved while achieving rational resource utilization.

[0093] Figure 4 The distributed resource scheduling method of the embodiment of the present disclosure is applied to electronic devices in a cluster, and the cluster to which the electronic devices belong is an online cluster, such as Figure 4 As shown, the method includes:

[0094] S401: Obtain a directed acyclic graph of tasks to be processed, and determine resource requirements for executing a processing flow indicated by the directed acyclic graph.

[0095] Similarly, regarding the technical features that are the same as those in the above embodiments, the embodiments of the present disclosure will not be repeated.

[0096] S402: If the current available resources of the electronic device do not meet the resource requirement, resources are retrieved from the offline device.

[0097] S403: Execute the processing flow indicated by the directed acyclic graph at least according to the retrieved resources.

[0098] In some embodiments, S403 may include the following two situations:

[0099] Case 1: If the available resources do not reach the preset resource threshold, the processing flow indicated by the directed acyclic graph is executed according to the resources called.

[0100] Case 2: If the available resource amount reaches the resource threshold, the processing flow indicated by the directed acyclic graph is executed according to the retrieved resources and the available resources corresponding to the available resource amount.

[0101] The resource threshold is determined based on the amount of resources required when the tasks in the task queue are executed.

[0102] In other words, different scenarios are distinguished. For scenarios where the available resources are less than the resource threshold, since the available resources are relatively small, the pending task, such as generating an electronic map, can be completed directly based on the retrieved resources. For scenarios where the available resources are greater than or equal to the resource threshold, since the available resources are limited, in order to improve the efficiency of electronic map generation, the electronic map can be generated based on both the available resources and the retrieved resources.

[0103] In this embodiment, by adopting different processing solutions for different scenarios, it is possible to achieve the technical effect of rational utilization of resources and improving the efficiency of completing tasks to be processed.

[0104] S404: In response to receiving a request for accessing resources sent by other electronic devices, a work process is created.

[0105] S405: Providing resources to other electronic devices based on the created work process.

[0106] That is to say, in some embodiments, electronic devices can retrieve resources from offline devices to complete tasks to be processed. In other embodiments, they can also provide resources to other electronic devices to assist other electronic devices in completing tasks to be processed, so as to improve the flexibility and diversity of resource retrieval and achieve the technical effect of rational utilization of resources.

[0107] In some embodiments, the request for resource retrieval sent by other electronic devices carries an identifier of the task corresponding to the request for resource retrieval sent by other electronic devices; S404 includes: according to the identifier of the task corresponding to the request for resource retrieval sent by other electronic devices, using the created work process to execute the task corresponding to the request for resource retrieval sent by other electronic devices, and the task corresponding to the request for resource retrieval sent by other electronic devices is stored in the task queue of the other electronic device.

[0108] Similarly, in order to achieve targeted, accurate and reliable execution of tasks, the electronic device can increase resources for tasks stored in the task queues of other electronic devices based on corresponding identifiers to complete the operation of executing the tasks.

[0109] Figure 5 Schematic diagram of the fourth embodiment of the present disclosure. The distributed resource scheduling device of the embodiment of the present disclosure is applied to electronic devices in a cluster, and the cluster to which the electronic devices belong is an online cluster, such as Figure 5 As shown, the apparatus 500 includes:

[0110] The acquisition unit 501 is configured to acquire a directed acyclic graph of tasks to be processed.

[0111] The determining unit 502 is configured to determine the resource requirements for executing the processing flow indicated by the directed acyclic graph.

[0112] The retrieving unit 503 is configured to retrieve resources from an offline device if the current available resources of the electronic device do not meet the required resources.

[0113] The execution unit 504 is configured to execute the processing flow indicated by the directed acyclic graph at least according to the retrieved resources.

[0114] Figure 6 Schematic diagram of the fifth embodiment of the present disclosure. The distributed resource scheduling device of the embodiment of the present disclosure is applied to electronic devices in a cluster, and the cluster to which the electronic devices belong is an online cluster, such as Figure 6 As shown, the apparatus 600 includes:

[0115] The acquisition unit 601 is used to acquire a directed acyclic graph of tasks to be processed.

[0116] Combine Figure 6 It can be seen that, in some embodiments, the acquisition unit 601 includes:

[0117] The acquisition subunit 6011 is configured to, in response to monitoring a task processing request, acquire a processing flow for completing the task to be processed according to the task to be processed carried in the task processing request.

[0118] In some embodiments, the electronic device is provided with an interface service; the acquisition subunit 6011 includes:

[0119] The monitoring module is used to monitor according to the interface service monitoring.

[0120] The acquisition module is used to use the interface service to obtain the processing flow for completing the pending tasks according to the pending tasks carried in the task processing request if a task processing request is monitored.

[0121] The conversion module is used to convert the processing flow of completing the task to be processed into a processing flow that can be recognized by the electronic device.

[0122] The generation module is used to generate a directed acyclic graph according to the converted processing flow.

[0123] The conversion subunit 6012 is used to convert the processing flow of completing the task to be processed into a processing flow that can be recognized by the electronic device.

[0124] The generating subunit 6013 is used to generate a directed acyclic graph according to the converted processing flow.

[0125] The determining unit 602 is configured to determine the resource requirements for executing the processing flow indicated by the directed acyclic graph.

[0126] Combine Figure 6 It can be seen that, in some embodiments, the determining unit 602 includes:

[0127] The first determining subunit 6021 is used to determine a type attribute of the directed acyclic graph, where the type attribute is used to indicate whether to retrieve resources from an offline device to execute a processing flow indicated by the directed acyclic graph.

[0128] In some embodiments, the first determining subunit 6021 includes:

[0129] Create a module to create a worker process.

[0130] A determination module is used to determine the type attribute of the directed acyclic graph based on the worker process.

[0131] The release module is used to release the work process if the processing flow indicated by the directed acyclic graph is completed.

[0132] The second determining subunit 6022 is configured to determine the resource requirement for executing the processing flow indicated by the directed acyclic graph based on the directed acyclic graph if the type attribute is used to indicate not retrieving resources from the offline device to execute the processing flow indicated by the directed acyclic graph.

[0133] The calling subunit 6023 is configured to call resources from the offline device if the type attribute indicates that resources are to be called from the offline device to execute the processing flow indicated by the directed acyclic graph.

[0134] The retrieving unit 603 is configured to retrieve resources from an offline device if the current available resources of the electronic device do not meet the resource requirement.

[0135] In some embodiments, the directed acyclic graph has an identifier; the retrieval unit 603 is used to send a resource retrieval request to the offline device, the resource retrieval request is used to retrieve resources from the offline device, the resource retrieval request carries the identifier of the directed acyclic graph, and the retrieved resources are provided by the offline device according to the identifier of the directed acyclic graph.

[0136] The execution unit 604 is configured to execute the processing flow indicated by the directed acyclic graph at least according to the retrieved resources.

[0137] In some embodiments, the execution unit 604 is used to execute the processing flow indicated by the directed acyclic graph based on the called resources if the amount of available resources does not reach a preset resource threshold; if the amount of available resources reaches the resource threshold, execute the processing flow indicated by the directed acyclic graph based on the called resources and the available resources corresponding to the available resource amount, wherein the resource threshold is determined based on the amount of resources required when the tasks in the task queue are executed.

[0138] The generating unit 605 is configured to generate a task of the directed acyclic graph according to the processing flow indicated by the directed acyclic graph.

[0139] The labeling unit 606 is configured to label the identifier of the directed acyclic graph as the identifier of the task of the directed acyclic graph.

[0140] The adding unit 607 is configured to add the task marked with the directed acyclic graph identifier to a preset task queue.

[0141] The to-be-processed task is completed by at least using resources retrieved from an offline device and executing tasks marked with the identifier of the directed acyclic graph according to the identifier of the directed acyclic graph.

[0142] The creating unit 608 is configured to create a work process in response to receiving a request for accessing resources sent by other electronic devices.

[0143] The providing unit 609 is configured to provide resources to other electronic devices based on the created work process.

[0144] In some embodiments, the request for resource retrieval sent by other electronic devices carries an identifier of a task corresponding to the request for resource retrieval sent by other electronic devices; the providing unit 609 is used to use the created work process to execute the task corresponding to the request for resource retrieval sent by other electronic devices based on the identifier of the task corresponding to the request for resource retrieval sent by other electronic devices, wherein the task corresponding to the request for resource retrieval sent by other electronic devices is stored in the task queue of the other electronic device.

[0145] In some embodiments, if the resource requirement is a process resource requirement, the available resource amount is the available process resource amount; if the resource requirement is a processor resource requirement, the available resource amount is the available processor resource amount; if the resource requirement is a network resource requirement, the available resource amount is the network resource requirement.

[0146] According to another aspect of the embodiments of the present disclosure, the embodiments of the present disclosure also provide a distributed resource scheduling system, including: an electronic device and an offline device, the electronic device is a device in a cluster, and the cluster to which the electronic device belongs is an online cluster, and the electronic device includes the device described in any of the above embodiments.

[0147] For example, combined Figure 1 As shown, the distributed resource scheduling system may include Figure 1 The computer equipment and offline equipment shown in may also include Figure 1 The computer equipment and deep learning platform shown in , of course, can also include Figure 1 The computer equipment, offline equipment, and deep learning platform shown in .

[0148] A distributed resource scheduling system may include Figure 1 The computer device and offline device shown in the example are used as an example for exemplary description. The computer device includes an interface service, a scheduler, and a task queue. Based on the above method embodiment, it can be seen that:

[0149] The interface service is used to monitor task processing requests. When a task processing request is monitored, the processing flow of the task to be processed carried in the task processing request is obtained, the obtained processing flow is converted into a processing flow that can be recognized by the electronic device, a directed acyclic graph is generated according to the converted processing flow, and the directed acyclic graph is added to the database (not shown in the figure).

[0150] The scheduler is used to poll the database, such as confirming whether the database includes a directed acyclic graph at preset intervals. When there is a directed acyclic graph in the database, it obtains the directed acyclic graph, splits the directed acyclic graph into multiple tasks, and adds each split task to the task queue.

[0151] Among them, the directed acyclic graph has an identifier. After the scheduler splits the directed acyclic graph into multiple tasks, it marks the identifier of the directed acyclic graph as the identifier of each task, that is, each task added to the task queue has an identifier, and the identifier of each task is the same as the identifier of the directed acyclic graph corresponding to the task.

[0152] The scheduler is also used to transmit the directed acyclic graph to the application management unit, wherein the application management unit is essentially a created working process.

[0153] The task queue is used to store tasks added by the scheduler.

[0154] The application management unit is used to determine whether resources need to be retrieved from the offline device according to the directed acyclic graph, and if so, send a resource scheduling request to the resource manager of the offline device.

[0155] Among them, one application management unit corresponds to one directed acyclic graph.

[0156] Resource Manager, used to create work processes based on resource scheduling requests, such as Figure 1 Worker 3 and Worker 4 are shown in the figure.

[0157] Worker process 3 and worker process 4 are used to execute tasks in the task queue that have the same identifier as the identifier of the directed acyclic graph.

[0158] Accordingly, worker process 1 and worker process 2 are created based on the resources of the computing device to execute tasks in the task queue.

[0159] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure, as shown in Figure 7 As shown, the electronic device 700 in the present disclosure may include: a processor 701 and a memory 702 .

[0160] Memory 702 is used to store programs. Memory 702 may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. Memory may also include non-volatile memory, such as flash memory. Memory 702 is used to store computer programs (such as applications and functional modules that implement the above-mentioned methods), computer instructions, etc. The above-mentioned computer programs and computer instructions may be partitioned and stored in one or more memories 702. Furthermore, the above-mentioned computer programs, computer instructions, data, etc. may be called by processor 701.

[0161] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories 702 . Furthermore, the aforementioned computer programs, computer instructions, etc. may be called by the processor 701 .

[0162] The processor 701 is configured to execute the computer program stored in the memory 702 to implement the various steps in the method involved in the above embodiment.

[0163] For details, please refer to the relevant description in the previous method embodiment.

[0164] The processor 701 and the memory 702 may be independent structures or integrated structures. When the processor 701 and the memory 702 are independent structures, the memory 702 and the processor 701 may be coupled via a bus 703 .

[0165] The electronic device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0166] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0167] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0168] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0169] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0170] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0171] The computing unit 801 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the distributed resource scheduling method. For example, in some embodiments, the distributed resource scheduling method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the distributed resource scheduling method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the distributed resource scheduling method by any other appropriate means (e.g., by means of firmware).

[0172] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0176] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0177] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0178] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0179] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A distributed resource scheduling method, the method being applied to electronic devices in a cluster, wherein the cluster to which the electronic devices belong is an online cluster, the method comprising: Obtaining a directed acyclic graph of tasks to be processed, and determining resource requirements for executing a processing flow indicated by the directed acyclic graph; If the current available resources of the electronic device do not reach the resource demand, resources are retrieved from the offline device, and the processing flow indicated by the directed acyclic graph is executed at least based on the retrieved resources; The directed acyclic graph has an identifier; resources are retrieved from offline devices, including: A resource retrieval request is sent to the offline device, where the resource retrieval request is used to retrieve resources from the offline device. The resource retrieval request carries an identifier of the directed acyclic graph, and the retrieved resources are provided by the offline device according to the identifier of the directed acyclic graph.

2. The method according to claim 1, wherein Determining resource requirements for executing the processing flow indicated by the directed acyclic graph includes: Determining a type attribute of the directed acyclic graph, where the type attribute is used to indicate whether to retrieve resources from the offline device to execute a processing flow indicated by the directed acyclic graph; If the type attribute is used to indicate that resources are not retrieved from the offline device to execute the processing flow indicated by the directed acyclic graph, the resource requirement for executing the processing flow indicated by the directed acyclic graph is determined based on the directed acyclic graph.

3. The method according to claim 2, further comprising: If the type attribute is used to indicate that resources are to be retrieved from the offline device to execute the processing flow indicated by the directed acyclic graph, resources are retrieved from the offline device.

4. The method according to claim 1, further comprising: Generating a task of the directed acyclic graph according to the processing flow indicated by the directed acyclic graph; Marking the identifier of the directed acyclic graph as the identifier of the task of the directed acyclic graph, and adding the task marked with the identifier of the directed acyclic graph to a preset task queue; The to-be-processed task is completed by at least using resources retrieved from the offline device and executing the task marked with the identifier of the directed acyclic graph according to the identifier of the directed acyclic graph.

5. The method according to any one of claims 1 to 4, further comprising: In response to receiving a resource-retrieving request sent by another electronic device, a work process is created, and resources are provided to the other electronic device based on the created work process.

6. The method according to claim 5, wherein: The resource-retrieving request sent by the other electronic device carries an identifier of the task corresponding to the resource-retrieving request sent by the other electronic device; Providing resources to the other electronic devices based on the created work process includes: According to the identifier of the task corresponding to the request for resource access sent by the other electronic device, the created work process is used to execute the task corresponding to the request for resource access sent by the other electronic device, wherein the task corresponding to the request for resource access sent by the other electronic device is stored in the task queue of the other electronic device.

7. The method according to any one of claims 1 to 4 and 6, wherein Get a directed acyclic graph of pending tasks, including: In response to monitoring a task processing request, obtaining a processing flow for completing the task to be processed according to the task to be processed carried in the task processing request; The processing flow for completing the task to be processed is converted into a processing flow recognizable by the electronic device, and the directed acyclic graph is generated according to the converted processing flow.

8. The method according to claim 7, wherein: The electronic device is provided with an interface service; in response to monitoring a task processing request, obtaining a processing flow for completing the task to be processed according to the task to be processed carried in the task processing request, including: Monitor according to the interface service. If the task processing request is monitored, the interface service is used to obtain the processing flow for completing the pending task based on the pending task carried in the task processing request, and convert the processing flow for completing the pending task into a processing flow recognizable by the electronic device, and generate the directed acyclic graph based on the converted processing flow.

9. The method according to any one of claims 1 to 4, 6 and 8, wherein At least according to the retrieved resources, executing the processing flow indicated by the directed acyclic graph includes: If the available resource amount does not reach the preset resource threshold, executing the processing flow indicated by the directed acyclic graph according to the retrieved resources; If the available resource amount reaches the resource threshold, executing the processing flow indicated by the directed acyclic graph according to the retrieved resources and the available resources corresponding to the available resource amount; The resource threshold is determined based on the amount of resources required when the tasks in the task queue are executed.

10. The method according to claim 2 or 3, wherein: Determining the type attribute of the directed acyclic graph includes: A work process is created, and a type attribute of the directed acyclic graph is determined based on the work process.

11. The method according to claim 10, further comprising: If the processing flow indicated by the directed acyclic graph is completed, the working process is released.

12. The method according to any one of claims 1 to 4, 6, 8, and 11, wherein: If the resource requirement is a process resource requirement, the available resource amount is the available process resource amount; if the resource requirement is a processor resource requirement, the available resource amount is the available processor resource amount; If the resource requirement is a network resource requirement, then the available resource amount is the network resource requirement.

13. A distributed resource scheduling device, the device being applied to electronic devices in a cluster, wherein the cluster to which the electronic devices belong is an online cluster, the device comprising: An acquisition unit, used to acquire a directed acyclic graph of tasks to be processed; a determining unit, configured to determine a resource requirement for executing the processing flow indicated by the directed acyclic graph; a retrieving unit, configured to retrieve resources from an offline device if the current available resources of the electronic device do not meet the resource requirement; an execution unit, configured to execute the processing flow indicated by the directed acyclic graph at least according to the retrieved resources; The directed acyclic graph has an identifier; the retrieval unit is used to send a resource retrieval request to the offline device, the resource retrieval request is used to retrieve resources from the offline device, the resource retrieval request carries the identifier of the directed acyclic graph, and the retrieved resources are provided by the offline device according to the identifier of the directed acyclic graph.

14. The device according to claim 13, wherein The determining unit includes: a first determining subunit, configured to determine a type attribute of the directed acyclic graph, wherein the type attribute is configured to indicate whether to retrieve resources from the offline device to execute a processing flow indicated by the directed acyclic graph; The second determining subunit is used to determine the resource requirement for executing the processing flow indicated by the directed acyclic graph based on the directed acyclic graph if the type attribute is used to indicate not retrieving resources from the offline device to execute the processing flow indicated by the directed acyclic graph.

15. The apparatus according to claim 14, further comprising: The calling subunit is configured to call resources from the offline device if the type attribute is used to indicate that resources are to be called from the offline device to execute the processing flow indicated by the directed acyclic graph.

16. The apparatus according to claim 13, further comprising: A generating unit, configured to generate a task of the directed acyclic graph according to a processing flow indicated by the directed acyclic graph; a labeling unit, configured to label the identifier of the directed acyclic graph as an identifier of a task of the directed acyclic graph; An adding unit, configured to add a task marked with the identifier of the directed acyclic graph to a preset task queue; The to-be-processed task is completed by at least using resources retrieved from the offline device and executing the task marked with the identifier of the directed acyclic graph according to the identifier of the directed acyclic graph.

17. The apparatus according to any one of claims 13 to 16, further comprising: A creating unit, configured to create a work process in response to receiving a request for accessing a resource sent by another electronic device; A providing unit is used to provide resources for the other electronic devices based on the created work process.

18. The device according to claim 17, wherein The request for resource retrieval sent by the other electronic device carries the identifier of the task corresponding to the request for resource retrieval sent by the other electronic device; the providing unit is used to use the created work process to execute the task corresponding to the request for resource retrieval sent by the other electronic device according to the identifier of the task corresponding to the request for resource retrieval sent by the other electronic device, wherein the task corresponding to the request for resource retrieval sent by the other electronic device is stored in the task queue of the other electronic device.

19. The device according to any one of claims 13 to 16 or 18, wherein: The acquisition unit includes: an acquiring subunit, configured to, in response to monitoring a task processing request, acquire a processing flow for completing the task to be processed according to the task to be processed carried in the task processing request; a conversion subunit, configured to convert a processing flow for completing the task to be processed into a processing flow recognizable by the electronic device; A generating subunit is used to generate the directed acyclic graph according to the converted processing flow.

20. The device according to claim 19, wherein The electronic device is provided with an interface service; the acquisition subunit includes: A monitoring module, configured to monitor the interface service; an acquisition module configured to, if the task processing request is monitored, adopt the interface service to acquire a processing flow for completing the task to be processed according to the task to be processed carried in the task processing request; a conversion module, configured to convert a processing flow for completing the task to be processed into a processing flow recognizable by the electronic device; A generating module is used to generate the directed acyclic graph according to the converted processing flow.

21. The device according to any one of claims 13 to 16, 18 and 20, wherein: The execution unit is used to execute the processing flow indicated by the directed acyclic graph based on the retrieved resources if the available resource amount does not reach a preset resource threshold; if the available resource amount reaches the resource threshold, execute the processing flow indicated by the directed acyclic graph based on the retrieved resources and the available resources corresponding to the available resource amount, wherein the resource threshold is determined based on the amount of resources required when the tasks in the task queue are executed.

22. The device according to claim 14 or 15, wherein The first determining subunit includes: Create a module to create a working process; A determination module is used to determine a type attribute of the directed acyclic graph based on the work process.

23. The apparatus according to claim 22, further comprising: A release module is used to release the work process if the processing flow indicated by the directed acyclic graph is completed.

24. The device according to any one of claims 13 to 16, 18, 20, and 23, wherein: If the resource requirement is a process resource requirement, the available resource amount is the available process resource amount; if the resource requirement is a processor resource requirement, the available resource amount is the available processor resource amount; If the resource requirement is a network resource requirement, then the available resource amount is the network resource requirement.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

28. A distributed resource scheduling system comprising: An electronic device and an offline device, wherein the electronic device is a device in a cluster, and the cluster to which the electronic device belongs is an online cluster, and the electronic device includes the apparatus according to any one of claims 13 to 24.

Images