Resource management method, device and equipment and computer storage medium

Abstract

The embodiment of the invention provides a resource management method and device, equipment and a computer storage medium, and the method comprises the steps: firstly, determining a task communication port corresponding to a task publishing end; and then, determining a task state corresponding to the target task by monitoring whether the task issuing end sends task communication information to the task communication port or not. And when the task state is determined to be an exit state, determining task occupation resources of the target task, and releasing the task occupation resources. Through the technical scheme provided by the embodiment of the invention, the connection condition of the task issuing end and the port and the task state of the target task can be accurately detected by monitoring the task communication condition on the communication port, so that when the target task is determined to be in the quit state, the task resources occupied by the target task are timely released, and the task resource utilization rate is improved. And invalid occupation and waste of resources are avoided, and the released resources can be used for executing other tasks, so that the execution efficiency of other tasks is improved to a certain extent.

Description

Technical Field

[0001] This application belongs to the field of resource management, and particularly relates to a resource management method, device, and computer storage medium. Background Art

[0002] Currently, when facing tasks with characteristics such as large-scale data and high-concurrency requests, the efficiency of task distribution and processing can be significantly improved through a distributed task processing system, such as Gearman.

[0003] In the current technology, the client responsible for task publishing communicates with the worker executing the task through the communication port of the distributed task processing system. Among them, the main way for the client and the worker to determine that the other party is normally connected during the task execution process is through the heartbeat mechanism, that is, the client and the worker exchange heartbeat signals regularly to ensure that the other party is in a normal online state. Generally, the time interval corresponding to the heartbeat mechanism is set to be relatively long in order to reduce resource occupation. As a result, when the client has actively exited the task communication with the worker, but since the next heartbeat cycle has not arrived, the worker cannot immediately know that the task has been cancelled or abandoned, resulting in the task still being executed, causing unnecessary resource occupation and waste. Even when resources are scarce, it will affect the normal execution of other tasks. Summary of the Invention

[0004] Embodiments of this application provide a resource management method, device, equipment, and computer storage medium to timely release task resources when the task publishing end exits the task communication.

[0005] In a first aspect, embodiments of this application provide a resource management method, and the method includes:

[0006] Determine the task communication port corresponding to the task publishing end, and determine the distributed task distribution server corresponding to the task communication port. The task communication port is used for task communication between the task publishing end and the task execution end, and the task execution end is used to execute the target task published by the task publishing end through the distributed task distribution server;

[0007] Determine the task status corresponding to the target task by monitoring the communication situation between the task communication port and the task publishing end;

[0008] In the case where the task status is determined to be the exit status, determine the task-occupied resources corresponding to the target task, and release the task-occupied resources.

[0009] In a second aspect, embodiments of this application provide a resource management device, and the device includes:

[0010] A port determination unit, configured to determine a task communication port corresponding to a task publishing end, and determine a distributed task distribution server corresponding to the task communication port, where the task communication port is used for task communication between the task publishing end and a task execution end, and the task execution end is configured to execute a target task published by the task publishing end through the distributed task distribution server;

[0011] A status determination unit, configured to determine a task status corresponding to the target task by monitoring communication between the task communication port and the task publishing end;

[0012] A resource management unit, configured to, when determining that the task status is an exit status, determine task occupied resources corresponding to the target task, and release the task occupied resources.

[0013] Thirdly, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the resource management method according to any one of the embodiments of the present application are implemented.

[0014] Fourthly, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps of the resource management method according to any one of the embodiments of the present application are implemented.

[0015] Fifthly, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device can execute the steps of the resource management method according to any one of the embodiments of the present application.

[0016] The technical solution provided by the embodiment of the present application at least brings the following beneficial effects:

[0017] A resource management method provided by an embodiment of the present application includes: First, determine a task communication port corresponding to a task publishing end. Then, determine a task status corresponding to the target task by monitoring whether the task publishing end sends task communication information to the task communication port. When determining that the task status is an exit status, determine the task occupied resources of the target task and release them.

[0018] The resource management method provided by the embodiment of the present application can accurately determine the connection status between the task publishing end and the task communication port and the task status of the target task by monitoring the communication between the task communication port and the task publishing end. When determining that the target task is in an exit status, the task resources occupied by the target task can be released in a timely manner, avoiding ineffective occupation and waste of resources. The released resources can be used to execute other tasks, thereby also improving the execution efficiency of other tasks to a certain extent. Description of the Drawings

[0019] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 A flowchart of a resource management method provided by an embodiment of the present application;

[0021] Figure 2 A flowchart of a method for determining a task communication port corresponding to a task publishing end provided by an embodiment of the present application;

[0022] Figure 3 A flowchart of a specific embodiment of a resource management method provided by an embodiment of the present application;

[0023] Figure 4 A schematic structural diagram of a resource management device provided by another embodiment of the present application;

[0024] Figure 5 A schematic hardware structure diagram of a resource management device provided by yet another embodiment of the present application. Detailed implementation manners

[0025] The following will describe in detail the features and exemplary embodiments of various aspects of the present application. To make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and not to limit the present application. For those skilled in the art, the present application can be implemented without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0026] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise", or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device including a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. Without further limitation, the elements defined by the statement "include..." do not exclude the existence of additional identical elements in the process, method, article, or device including the said elements.

[0027] It should be noted that in the technical solution of this application, the acquisition, storage, use, processing, etc. of data all comply with the relevant provisions of national laws and regulations.

[0028] It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, models, etc. may be mentioned. They should be regarded as exemplary. The purpose is only to illustrate the feasibility in the implementation of the technical solution of this application, but it does not mean that the applicant has already or necessarily used this solution.

[0029] With the rapid development of computer and network technologies, the distributed task processing technology for handling high-concurrency requests and large-data-volume tasks has become increasingly mature and popular. For example, Gearman is responsible for task communication between the client and single or multiple workers. Currently, when task communication is carried out through a distributed task processing system, the client and the worker mainly determine whether the other party is normally connected based on a heartbeat mechanism with a specific time interval.

[0030] In the current technology, in order to reduce the resource occupation caused by the heartbeat mechanism, the time interval corresponding to the heartbeat mechanism is generally set relatively long. As a result, when the client has disconnected from the distributed task processing system, the worker does not immediately learn that the task being executed has been terminated or abandoned due to the time interval of the heartbeat mechanism, but instead continues to execute the task and occupy resources. Only when the next heartbeat cycle arrives will the worker know that the task has been terminated. There is a long communication delay and ineffective occupation and waste of resources during this process.

[0031] Based on the technical problems mentioned in the above content, the embodiments of this application provide a resource management method, device, equipment, and computer storage medium. It can first determine the task communication port corresponding to the task publisher, and determine the task status of the target task by monitoring whether the task publisher sends task communication information to the task communication port. When it is determined that the task status is the exit status, determine the resources occupied by the target task and release the resources.

[0032] The technical solution provided by the embodiments of this application can accurately detect the connection status between the task publisher and the task communication port, as well as the task status of the target task by monitoring the task communication on the communication port. When it is determined that the target task is in the exit status, the task resources occupied by the target task are released in a timely manner, effectively avoiding ineffective occupation and waste of resources. The computing resources released in a timely manner can also be used to execute other tasks waiting for resource allocation, thereby also improving the execution efficiency of other tasks to a certain extent.

[0033] There are various implementation manners for the execution subject of the resource management method provided by the embodiments of the present application. For example, the execution subject may be a terminal device connected to a distributed task distribution server, such as a desktop computer, a laptop computer, etc., or may be a remote device capable of connecting to and communicating with the distributed task distribution server, such as a server, etc., or may also be the distributed task distribution server itself.

[0034] In some embodiments, the resource management method provided by the embodiments of the present application may be implemented through a monitoring process. The monitoring process may be carried on a terminal device or a server connected to the distributed task distribution server, or may also be directly deployed on the distributed task distribution server, so as to monitor the task communication between the task publishing end and the task executing end. The monitoring process may also be deployed on other feasible execution subjects.

[0035] The specific application scenarios of the resource management method, device, equipment, and computer storage medium provided by the embodiments of the present application are not strictly limited in the present application and may be flexibly applied according to actual requirements. For the convenience of understanding, the following gives examples of the specific application scenarios of the resource management method, device, equipment, and computer storage medium provided by the embodiments of the present application.

[0036] For example, taking the Gearman server as the distributed task distribution server as an example, in the scenario where the task communication port of the Gearman server is responsible for the task communication between the task publishing end (i.e., the client) and the task executing end (i.e., the worker), the monitoring process applying the technical solution provided in the present application may first determine the task communication port corresponding to the task publishing end and connected to the Gearman server. Among them, the monitoring process may be understood as a daemon process for the task occupied resources corresponding to the target task, and is used to implement the quick exit of the target task and the timely release of resources.

[0037] Then, during the process of the task executing end executing the target task published by the task publishing end, the monitoring process may monitor the task communication situation between the task publishing end and the task communication port, so as to timely determine the task status of the target task. When it is determined that the task status of the target task is the exit status, the monitoring process may timely release the occupation of the task communication port of the Gearman server by the task publishing end and the task executing end, and determine and release the task occupied resources corresponding to the target task.

[0038] Through the technical solution provided by this application, the task communication between the task publishing end and the task communication port of the Gearman server can be monitored to determine the task status of the target task. When it is determined that the target task is in the exit state, the resources occupied by the task are released in a timely manner, effectively avoiding waste of resources. The released resources can be used to execute other tasks, thereby improving the task execution efficiency of other tasks.

[0039] It should be noted that the application scenarios described in the embodiments of this application above are for more clearly explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those of ordinary skill in the art know that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are equally applicable to similar technical problems. The resource management methods, devices, equipment, and computer storage media provided by the embodiments of this application can be applied to various application scenarios that require resource management.

[0040] The following introduces the specific embodiments of the resource management method, device, equipment, and computer storage media provided by the embodiments of this application. First, the resource management method is introduced.

[0041] Figure 1 It is a schematic flowchart of a resource management method provided by an embodiment of this application.

[0042] As Figure 1 shown, the resource management method provided by the embodiments of this application includes steps S101 to S103.

[0043] S101: Determine the task communication port corresponding to the task publishing end, and determine the distributed task distribution server corresponding to the task communication port.

[0044] In step S101, the monitoring process applying the technical solution provided by the embodiments of this application can determine the task communication port corresponding to the task publishing end and the distributed task distribution server corresponding to the task communication port before the task publishing end publishes the target task and the task execution end executes the target task.

[0045] The task communication port can specifically be used to be responsible for the task communication between the task publishing end and the task execution end.

[0046] The distributed task distribution server can specifically be used to be responsible for task distribution and management of the target task published by the task publishing end. In the embodiments provided by this application, the specific type and structure corresponding to the distributed task distribution server are not strictly limited.

[0047] In one embodiment, the distributed task distribution server may be the Gearman server mentioned in the above example. In other embodiments, the distributed task distribution server may also be other server sides for processing parallel tasks, such as Celery, Apache Kafka, etc., which can be flexibly selected according to the actual application scenarios and requirements.

[0048] The task execution end can specifically be used to execute the target task published by the task publishing end through the task communication port of the distributed task distribution server, and interact with the task publishing end through the task communication port.

[0049] In one embodiment provided by the present application, the monitoring process can directly use the default task communication port that is default used for task communication as the task communication port corresponding to the task publishing end. The default distributed task distribution server corresponding to the default task communication port can be used as the distributed task distribution server corresponding to the task publishing end.

[0050] After determining the corresponding task communication port and the distributed task distribution server, the task publishing end can publish the target task through the task communication port connected to the distributed task distribution server, and the task execution end connected to the task communication port can execute the target task after receiving the task information corresponding to the target task.

[0051] In addition, in the present application, considering that at the current stage, when the client publishes tasks and communicates with the worker through the distributed task processing system (such as Gearman), most of them will not actively select and change the communication port, and directly operate through the default communication port.

[0052] When multiple clients publish tasks simultaneously and communicate with multiple workers simultaneously, only the unique default task communication port is responsible for the huge task communication process, which is very likely to cause port communication congestion, thereby affecting the communication efficiency of all parallel execution tasks.

[0053] Therefore, to solve the above technical problems, in another embodiment provided by the present application, when the monitoring process determines that the default task communication port is occupied, it can allocate an unoccupied task communication port for the task publishing end to be responsible for the task communication between the task publishing end and the task execution end. The specific processing process can refer to Figure 2 .

[0054] Figure 2 It is a schematic flow diagram of a method for determining the task communication port corresponding to the task publishing end provided by an embodiment of the present application.

[0055] As Figure 2As shown in the figure, the specific process for determining the task communication port corresponding to the task publishing end provided in the embodiments of the present application includes steps S201 to S202.

[0056] S201: When the task publishing end requests to allocate a task communication port, detect whether the default task communication port corresponding to the task publishing end is occupied.

[0057] In step S201, the monitoring process can determine whether the default task communication port corresponding to the task publishing end is already in an occupied state before the task publishing end publishes the target task or when the task publishing form requests to allocate a task communication port for task publishing.

[0058] S202: In the case where it is determined that the default task communication port is occupied, determine an unoccupied task communication port as the task communication port corresponding to the task publishing end.

[0059] In step S202, if it is determined that the default task communication port is in an occupied state, the monitoring process can determine the port identifier corresponding to the unoccupied task communication port from other preset task communication ports. Then, the monitoring process can use the unoccupied task communication port corresponding to the determined port identifier as the task communication port corresponding to the task publishing end.

[0060] The specific implementation method for determining the port identifier of the unoccupied task communication port is not strictly limited in the present application. Taking the distributed task distribution server as the Gearman server as an example, in an embodiment provided in the present application, when the Gearman server is used as the distributed task distribution server, the port identifier of the corresponding default task communication port is 4730. When it is determined that the default task communication port 4730 is occupied, the monitoring process can increment the port identifier numerically, so as to determine an unoccupied task communication port from other preset task communication ports except the default communication port.

[0061] Taking the default task communication port 4730 of the Gearman server as an example, the above-mentioned process of numerically incrementing the port identifier can be understood as determining whether the task communication port with the port identifier 4731 is occupied. If the task communication port 4731 is also in an occupied state, then determine whether the task communication port with the port identifier 4732 is occupied, and so on, until an unoccupied task communication port is found as the task communication port of the task publishing end. Among them, the preset number of other available task communication ports except the default task communication port is not strictly limited in the present application and can be flexibly set according to the actual application scenario and actual requirements.

[0062] Through the above process of using the determined unoccupied communication port as the task communication port corresponding to the task publisher, it can effectively avoid the problem of port congestion caused by multiple task publishers and task executors interacting task information through a single task communication port, thereby effectively improving the information interaction efficiency between the task publisher and the task executor and the execution efficiency of the target task. While improving the task execution rate, it also reduces the resource occupancy to a certain extent.

[0063] In addition to the above, considering that when the default task communication port is occupied, the corresponding default distributed task distribution server may also be processing the task corresponding to the default task communication port, that is, the default distributed task distribution server is also in an occupied state. At this time, if the unoccupied task communication port assigned to the task publisher is still connected to the default distributed task distribution server, it may still affect the execution efficiency of the target task and the task communication between the task publisher and the task executor.

[0064] Therefore, to solve the above problems, in another embodiment provided in this application, when the monitoring process determines that the default task communication port is occupied and determines that the unoccupied task communication port is the task communication port corresponding to the task publisher, for the unoccupied task communication port, a new distributed task distribution server can be newly started, and the newly started distributed task distribution server is used as the distributed task distribution server corresponding to the connection between the task publisher and the determined unoccupied task communication port.

[0065] It should be noted that the newly started distributed task distribution server can specifically be an actual distributed task distribution server device or a server with the function of a distributed task distribution server. The specific form is not strictly limited in this application and can be flexibly set according to the actual application scenario.

[0066] Through the above process of starting a new distributed task distribution server for the determined unoccupied task communication port, it can effectively improve the task communication efficiency between the subsequent task publisher and the task executor and the execution efficiency of the target task, and greatly avoid the communication congestion problem caused by multiple task publishers and multiple task executors using the same distributed task distribution server at the same time.

[0067] S102: Determine the task status corresponding to the target task by monitoring the communication situation between the task communication port and the task publisher.

[0068] In step S102, when the task executor executes the target task published by the task publisher, the monitoring process can determine the task status corresponding to the target task during the execution process by monitoring the communication situation between the task communication port and the task publisher.

[0069] Among them, the communication situation can be specifically determined based on the task communication information sent by the task publisher received by the task communication port, and the task communication information is communication information related to the target task. The specific content of the communication information may include, but is not limited to, the task content information corresponding to the target task (such as task type, task execution time, task execution content, etc.), the reply heartbeat information for the heartbeat signal sent by the task publisher to the task executor, the reply information for the task execution data sent by the task publisher to the task executor, and so on.

[0070] In an embodiment provided by the present application, the monitoring process can monitor the communication situation of the task communication port according to a preset monitoring period. Specifically, for each preset monitoring period, the monitoring process can monitor whether the task communication port receives the task communication information sent by the task publisher within this monitoring period.

[0071] In the case where it is determined that the task communication port does not receive the task communication information sent by the task publisher within this monitoring period, the monitoring process can determine that the task publisher has exited the execution process of the target task during this monitoring period and may have disconnected from the task communication port. At this time, the monitoring process can determine that the task status corresponding to the target task is the exit status.

[0072] The reason for only monitoring the task communication information sent by the task publisher is as follows: when the task executor disconnects from the task communication port when the task execution is completed or in other cases, the task publisher will also disconnect the communication connection with the task communication end by itself. Therefore, only by monitoring the task communication information sent by the task publisher can the task status be determined.

[0073] In addition, when a single task publisher conducts task communication with multiple task executors through the task communication port at the same time, it is impossible to directly clear the occupation of the task communication port and resources by the task publisher and all other task executors just because a certain task executor disconnects from the task communication port. Therefore, in the above embodiment, only the task communication information sent by the task publisher to the task communication port can be monitored to ensure that the determined task status of the target task is true and valid.

[0074] It should be noted that in the embodiments provided by the present application, the specific duration of the above-mentioned monitoring period is not strictly limited and can be flexibly set according to the actual application scenario and actual needs. It should be noted that the duration of the monitoring period should be a relatively short time compared to the heartbeat period corresponding to the heartbeat mechanism between the task publisher and the task executor. For example, if the heartbeat period corresponding to the heartbeat mechanism is 10 minutes, the preset monitoring period in the embodiments of the present application can be 2 minutes or other shorter time intervals.

[0075] Among them, the reason why the time interval of the monitoring period can be shorter than the heartbeat period is specifically as follows: If the heartbeat period between the task publishing end and the task executing end is set too short, it will cause the task executing end and the task publishing end to frequently send and reply heartbeat signals, resulting in unnecessary resource occupation and waste. The monitoring period only monitors the task communication port and does not generate or send any interaction information. Therefore, even if the monitoring period is set with a short time interval, it will not seriously affect the allocation and occupation of computing resources. At the same time, it can also timely monitor the sending situation of task communication information between the task publishing end and the task communication port to determine the task status of the target task.

[0076] Through the above processing process, the monitoring process can timely determine the task status of the target task according to the information sending situation of the task publishing end to the task communication port, so that the resource release for the target task can be timely carried out. Compared with only using the heartbeat mechanism with a long heartbeat period to determine the connection between the task publishing end and the task executing end, the above processing process is more timely, and can effectively avoid the occupation of resources by invalid target tasks through subsequent synchronized resource release. The released resources can be used by other tasks, and to a certain extent, it also improves the execution efficiency of other valid tasks.

[0077] S103: In the case of determining that the task status is the exit status, determine the task occupied resources corresponding to the target task and release the task occupied resources.

[0078] In step S103, the monitoring process can determine the task occupied resources corresponding to the target task and release the task occupied resources in the case of determining that the task status corresponding to the target task is the exit status through the above steps.

[0079] The task occupied resources corresponding to the target task can specifically be the device resources obtained by the task executing end from the task resource scheduling system when executing the target task. The task resource scheduling system can be, for example, a distributed resource manager (SunGrid Engine, SGE), LSF (Load Sharing Facility), etc. Taking the LSF system as an example, the task occupied resources corresponding to the target task can specifically be the processor resources (Core) and license resources (license) corresponding to the LSF system. The processor resources can specifically be used to represent the processor cores occupied by the target task during execution, and the license resources can specifically be used to represent the usage license for a specific software or service allocated to the target task.

[0080] In an embodiment provided by the present application, when the monitoring process determines that the task status corresponding to the target task is the exit status, it can be determined that the task publisher has exited the task execution process at this time, which means that even if the task executor sends a heartbeat signal to the task publisher through the task communication port in the next heartbeat cycle, no reply will be received. Among them, the task status of the target task being the exit status may specifically include, but is not limited to, that the target task has been executed, the execution of the target task has been aborted, and the target task has been actively cancelled by the task publisher.

[0081] Then, the monitoring process can clear the occupation of the task communication port by the task publisher and the task executor, and determine the task occupied resources corresponding to the target task according to the task identifier corresponding to the target task. Immediately afterwards, the monitoring process can generate and execute a task termination instruction for the target task to release the task occupied resources corresponding to the target task.

[0082] It should be noted that when the exit status of the target task is specifically that the target task has been executed, the task occupied resources corresponding to the target task may have been released when the target task was executed. At this time, the monitoring process can re-detect whether the task occupied resources corresponding to the target task that has been executed have been successfully released through the above processing process to ensure that resources are not occupied and wasted invalidly.

[0083] Through the above processing process of releasing the task occupied resources corresponding to the target task with the exit status, it is possible to timely determine and release the computing power resources that have already been invalid task occupations, greatly avoiding the invalid occupation and waste of resources during various task executions, enabling more resources to be freed up for executing other valid tasks that require resources, and thus improving the task execution efficiency to a certain extent.

[0084] It should be noted that the target task mentioned in the above steps can be recorded in a cluster task resource scheduling system with task resource allocation capabilities such as SGE (Sun Grid Engine), LSF (Load Sharing Facility), etc., and the task occupied resources corresponding to the target task can be the processor resources and license resources occupied by the target task in the cluster task resource scheduling system where it is located.

[0085] As mentioned above, for example, cluster task resource scheduling systems of the SGE and LSF types will implement task recording and submission through the method of job arrays, which may lead to the situation that not all submitted tasks can be allocated resources at the first moment due to insufficient resources, and continuous resource acquisition is required.

[0086] And suppose that when a target task is still in the resource grabbing state, the task publishing end corresponding to the target task has chosen to give up executing the target task. At this time, the cluster task resource scheduling system will not cancel the target task immediately, but will continue to grab resources, and after successfully grabbing resources, it will wait for a worker timeout (Worker Timeout) to cancel the resource occupation of the target task. It can be seen that in this case, not only may it cause a waste of nearly two worker timeout cycles (waiting time for resource grabbing and invalid resource occupation time after the task has been canceled), but it will also cause resources to be occupied by invalid tasks that no longer need to be executed during the worker timeout cycle. While invalid tasks are not executed, resources cannot be allocated to other tasks due to the existence of invalid tasks, resulting in a waste of computing resources and affecting the execution efficiency of other tasks.

[0087] Therefore, in order to solve the problems caused when cluster task resource scheduling systems such as SGE and LSF types submit tasks in a job array manner, in another embodiment provided in the present application, when the task occupied resources corresponding to the target task are the processor resources and license resources occupied by the target task in the cluster task resource scheduling system where the target task is located, the monitoring process can execute a task termination instruction for the target task through the cluster task resource scheduling system to release the processor resources and license resources occupied by the target task in the cluster task resource scheduling system.

[0088] The above-mentioned task termination instruction can specifically be similar to the bkill command in the LSF system, the qdel command in the SGE system, etc., and the specific purpose can be understood as killing or deleting the task. In addition, the specific type of the above-mentioned cluster task resource scheduling system is not strictly limited in the embodiment provided by the present application, and can be the SGE system and LSF system mentioned in the above-mentioned embodiment. In other embodiments, it can also be other cluster task resource scheduling systems that submit tasks through the job array mode, such as Slurm (Simple Linux Utility for Resource Management), PBS Pro (Portable Batch System Professional), etc., which can be flexibly applied according to actual application scenarios and requirements.

[0089] Through the above-mentioned process of releasing the task resources corresponding to the target task in the cluster task resource scheduling system, the resource waste problem caused by the job array task submission method mentioned above can be effectively solved, invalid tasks can be killed in time, and resources can be released in time or the continuous capture of resources can be canceled, effectively improving the execution efficiency of other effective tasks that require resource allocation.

[0090] In addition to the above, since the task communication port corresponding to the task publishing end in the above embodiments is not the default task communication port, that is, when the task communication port corresponding to the task publishing end is not the unoccupied task communication port re-determined in the previous steps, the corresponding distributed distribution server is not the default distributed distribution server corresponding to the default task communication port, but a newly started distributed distribution server.

[0091] In this case, when it is determined that the task status corresponding to the target task is the exit status, while clearing the occupation of the task communication port by the task publishing end and the task execution end, the work of the newly started distributed distribution server due to the unoccupied task communication port should also be stopped.

[0092] Therefore, in another embodiment provided by the present application, when the monitoring process determines that the task status corresponding to the target task is the exit status, it can determine whether the task communication port corresponding to the task publishing end is the default task communication port, so as to judge whether the distributed task distribution server corresponding to the task publishing end is the default distributed task distribution server corresponding to the default communication port.

[0093] When it is determined that the task communication port corresponding to the task publishing end is not the default task communication port, it means that the task communication port corresponding to the task publishing end is the unoccupied task communication port determined in the above embodiments, and the corresponding distributed task distribution server is also a newly started distributed task distribution server different from the default distributed task distribution server.

[0094] In this case, the monitoring process can stop the work of the distributed task distribution server corresponding to the task communication port while clearing the occupation of the task communication port by the task publishing end and the task execution end.

[0095] This part of the processing process can be simply understood as that when the corresponding task communication port of the newly started distributed task distribution server is not occupied, it also needs to stop being used, and wait until the next unoccupied task communication port other than the default task communication port is used for task communication, and then start and be responsible for the task communication between the task publishing end and the task execution end through the unoccupied task communication port.

[0096] Through the above processing of stopping the work of the newly started distributed task distribution server, it can effectively ensure that while canceling the occupation of the task communication port by the task publishing end and the task execution end, the resources occupied by the newly started distributed task distribution server are released in time, avoiding waste of resources and ineffective operation of the distributed task distribution server.

[0097] It should be further noted that while the above embodiments are used to release resources for the target task in the exit state, the monitoring process can also timely clean up the relevant task data generated during the execution of the target task, and count and sort out the execution error information generated during the execution of the target task.

[0098] In an embodiment provided by the present application, when the monitoring process determines that the task status corresponding to the target task is the exit state, it can perform data cleaning on the target task data corresponding to the target task, and at the same time, it can also count the execution error record information generated during the execution of the target task.

[0099] The target task data may specifically include, but is not limited to, temporary files, task execution logs, working directories, etc. generated during the execution of the target task. The execution error record information can be used to generate an error report for the target task and send it to relevant personnel for error statistics and recording.

[0100] Through the above processing process, while timely releasing the task occupied resources corresponding to the target task in the exit state, it can clean up the task data corresponding to the target task to timely release storage resources and cache space for other tasks to use. Moreover, it can also count the execution error record information corresponding to the target task, so that in the subsequent process, the execution process of the target task can be optimized and adjusted according to the execution error record information, avoiding repeated execution errors of the same type of target tasks, and thus improving the execution efficiency of the target task.

[0101] The following combines Figure 3 , and an implementation manner combining the above multiple embodiments is exemplarily described.

[0102] Figure 3 It is a schematic flowchart of a specific embodiment of a resource management method provided by an embodiment of the present application. As Figure 3 shown, a resource management method provided by an embodiment of the present application may include steps S301 to S305.

[0103] In this embodiment, the distributed task distribution server corresponding to the task publishing end may be the Gearman server mentioned in the above embodiment, and the cluster task resource scheduling system where the target task is located may be the LSF system mentioned in the above embodiment.

[0104] S301: When the task publishing end publishes a target task, determine the task communication port corresponding to the task publishing end. If the default task communication port of the Gearman server is occupied, determine an unoccupied task communication port as the task communication port corresponding to the task publishing end.

[0105] In step S301, when the execution entity (such as the monitoring process or server in the above steps) applying the technical solution provided in the embodiment of the present application publishes a target task through the task publishing end or submits batch tasks through the LSF system, it can determine whether the default task communication port of the Gearman server is occupied.

[0106] When it is determined that the default task communication port of the Gearman server is occupied, the port identifier of the default task communication port can be incremented numerically, and an unoccupied task communication port can be determined from other task communication ports except the default communication port as the task communication port corresponding to the task publishing end.

[0107] S302: If the task publishing end corresponds to an unoccupied task communication port among other preset task communication ports except the default communication port, start a new Gearman server to be responsible for the task communication between the task publishing end and the task execution end.

[0108] In step S302, the execution entity applying the technical solution provided in the embodiment of the present application can start a new Gearman server different from the default Gearman server corresponding to the default task communication port at the unoccupied task communication port determined for the task publishing end in step S301. Moreover, when the task occupied resources corresponding to the target task are released subsequently, the newly started Gearman server here will also be stopped.

[0109] S303: During the process of the task execution end executing the target task published by the task publishing end, monitor the task communication between the task publishing end and the task communication port.

[0110] In step S303, the execution entity applying the technical solution provided in the embodiment of the present application can monitor the task communication between the task publishing end and the task communication port by determining whether the task publishing end sends task communication information to the task communication port within a preset monitoring period during the execution of the target task. According to the monitoring result, determine the task status corresponding to the target task. Among them, the monitoring process can be started when it is determined that the task execution end starts to execute the target task and ends when the task occupied resources corresponding to the target task are released subsequently.

[0111] S304: When it is monitored that the task communication between the task publishing end and the task communication port disappears, determine that the task status of the target task is the exit status, release the occupation of the task communication port by the task publishing end and the task execution end, and execute a task termination instruction through the LSF system where the target task is located to kill the target task and release the task occupied resources.

[0112] In step S304, when the execution entity applying the technical solution provided in the embodiment of the present application determines that the task status of the target task is the exit status during the monitoring process of step S303, it can release the occupation of the task communication port allocated in step S301 by the task publishing end and the task execution end. If the task communication port allocated in step S301 is other than the default communication port, it can also stop the operation of the Gearman server corresponding to the task communication port to release resources in a timely manner.

[0113] Meanwhile, when it is determined that the task status of the target task is the exit status, a notification message indicating that the target task should be terminated can be sent to the LSF system, so that the LSF system can determine the target task according to the task identifier corresponding to the target task and execute the task termination instruction for the target task.

[0114] If the target task has been allocated resources and is in the process of task execution at this time, the target task can be killed in a timely manner through the task termination instruction (bkill command) to release the task resources. If, due to the submission method of the job array, the target task is still in the resource fetching state, the task termination instruction for the target task can be executed in a timely manner to kill the task in advance before the resource fetching is successful, cancel the resource fetching in a timely manner, and prevent the ineffective occupation and waste of resources.

[0115] S305: Stop the communication monitoring of the target task, delete the target task data corresponding to the target task, and count the execution error record information during the execution process of the target task.

[0116] In step S305, when the execution entity applying the technical solution provided in the embodiment of the present application determines that the task status of the target task is the exit status and releases the task occupation resources corresponding to the target task, it can stop the monitoring of the task communication between the task publishing end and the task execution end corresponding to the target task.

[0117] Meanwhile, according to the task privacy protection requirements of the task publishing end or the task execution end, the target task data corresponding to the target task can be deleted, and the execution error record information during the execution process of the target task can be counted for use in optimizing task execution and avoiding errors during the execution of subsequent target tasks of the same type.

[0118] The above is the specific implementation of the resource management method provided by the embodiments of this application. By monitoring the task communication between the task publisher and the task executor on the task communication port of the distributed task distribution server, the connection status between the task publisher and the task communication port and the task status of the target task can be accurately detected. When it is determined that the target task is in the exit state, the task-occupied resources corresponding to the target task are released in a timely manner, avoiding the ineffective occupation and waste of computing power resources. The released resources can be used to execute other tasks, thereby improving the execution efficiency of other tasks to a certain extent.

[0119] In addition, the embodiments of this application can also determine the task communication port assigned to the task publisher before the task publisher publishes a task. When the default task communication port is occupied, an unoccupied task communication port is determined to be responsible for the task communication between the task publisher and the task executor. This effectively solves the problem that multiple task publishers and multiple task executors simultaneously occupy a task communication port, effectively avoids the possible port congestion problem, and thus significantly improves the execution efficiency of the target task.

[0120] Based on the resource management method provided by the above embodiments, correspondingly, this application also provides a specific implementation of a resource management device. Please refer to the following embodiments.

[0121] Figure 4 The following is a schematic structural diagram of a resource management device provided by another embodiment of this application. The resource management device 400 specifically includes: a port determination unit 401, a status determination unit 402, and a resource management unit 403.

[0122] The port determination unit 401 is used to determine the task communication port corresponding to the task publisher and the distributed task distribution server corresponding to the task communication port. The task communication port is used for task communication between the task publisher and the task executor, and the task executor is used to execute the target task published by the task publisher through the distributed task distribution server.

[0123] The status determination unit 402 is used to determine the task status corresponding to the target task by monitoring the communication status between the task communication port and the task publisher.

[0124] The resource management unit 403 is used to determine the task-occupied resources corresponding to the target task and release the task-occupied resources when it is determined that the task status is the exit state.

[0125] The resource management device 400 provided by the embodiment of the present application can accurately detect the connection status of the task publisher and the task status of the target task by monitoring the task communication on the communication port. Thus, when it is determined that the target task is in an exit state, the task resources occupied by the target task can be released in a timely manner, avoiding the ineffective occupation and waste of resources. The released resources can be used to execute other tasks, and to a certain extent, the execution efficiency of other tasks is improved.

[0126] In some embodiments, the port determination unit 401 is specifically configured to:

[0127] When the task publisher requests to allocate a task communication port, detect whether the default task communication port corresponding to the task publisher is occupied;

[0128] In the case where it is determined that the default task communication port is occupied, determine an unoccupied task communication port as the task communication port corresponding to the task publisher.

[0129] In some embodiments, the port determination unit 401 is specifically configured to:

[0130] For the unoccupied task communication port, start a new distributed task distribution server as the distributed task distribution server corresponding to the unoccupied task communication port.

[0131] In some embodiments, the status determination unit 402 is specifically configured to:

[0132] For each preset monitoring period, monitor whether the task communication port receives task communication information sent by the task publisher within the monitoring period, where the task communication information is communication information related to the target task;

[0133] If not, determine that the task status corresponding to the target task is in an exit state.

[0134] In some embodiments, the resource management unit 403 is specifically configured to:

[0135] In the case where it is determined that the task status is in an exit state, determine that the task publisher exits the task execution process, and the exit state includes that the target task is executed completely, the target task execution is aborted, and the target task is cancelled by the task publisher;

[0136] Clear the occupation of the task communication port by the task publisher and the task executor, and determine the task occupied resources corresponding to the target task according to the task identifier corresponding to the target task;

[0137] Execute the task termination instruction for the target task to release the task occupied resources.

[0138] In some embodiments, the resource management unit 403 is specifically configured to:

[0139] Execute a task termination instruction for a target task and release the resources occupied by the task, including:

[0140] Execute a task termination instruction for a target task through a cluster task resource scheduling system, and release the processor resources and license resources occupied by the target task in the cluster task resource scheduling system.

[0141] In some embodiments, the resource management unit 403 is specifically configured to:

[0142] Determine whether the task communication port corresponding to the task publisher is the default communication port;

[0143] If not, while clearing the occupation of the task communication port by the task publisher and the task executor, stop the operation of the distributed task distribution server corresponding to the task communication port.

[0144] In some embodiments, the resource management unit 403 is further configured to:

[0145] In the case where the task status is determined to be the exit status, perform data cleaning on the target task data corresponding to the target task, and count the execution error record information corresponding to the target task during the task execution process.

[0146] The device in the above embodiments is used to implement the corresponding resource management method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0147] Figure 5 It is a schematic diagram of the hardware structure of a resource management device provided in another embodiment of the present application.

[0148] The electronic device 500 may include a processor 501 and a memory 502 storing computer program instructions.

[0149] Specifically, the above-mentioned processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0150] The memory 502 may include a mass storage for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 502 may include removable or non-removable (or fixed) media. Where appropriate, the memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, the memory 502 is a non-volatile solid-state memory.

[0151] In a particular embodiment, the memory 502 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0152] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present application.

[0153] The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement any one of the resource management methods in the above embodiments.

[0154] In one example, the electronic device may further include a communication interface 503 and a bus 504. Among them, for example, the processor 501, the memory 502, and the communication interface 503 are connected through the bus 504 to complete communication with each other.

[0155] The communication interface 503 is mainly used to implement communication between the modules, devices, units, and / or devices in the embodiments of the present application.

[0156] The bus 504 includes hardware, software, or both, and couples the components of the online data flow metering device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 504 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0157] The electronic device of the above embodiment is used to implement the corresponding resource management method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated herein.

[0158] In addition, in combination with the resource management method in the above embodiments, an embodiment of the present application can be implemented by providing a computer storage medium. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the resource management methods in the above embodiments is implemented.

[0159] In addition, in combination with the resource management method in the above embodiments, an embodiment of the present application can be implemented by providing a computer program product. When the instructions of the computer program product are executed by the processor of the electronic device, any one of the resource management methods in the above embodiments is implemented.

[0160] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present application (including the claims) is limited to these examples; within the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of brevity.

[0161] The functional blocks shown in the above-described structural block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present application are programs or code segments for performing the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via a data signal carried in a carrier wave. A "machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical discs, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0162] It should also be noted that in the exemplary embodiments mentioned in the present application, some methods or devices are described based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0163] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each block in the flowchart and / or block diagram, and the combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more blocks of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It should also be understood that each block in the block diagram and / or flowchart, and the combinations of blocks in the block diagram and / or flowchart, can also be implemented by dedicated hardware that performs the specified functions or actions, or by a combination of dedicated hardware and computer instructions.

[0164] As described above, this is only the specific implementation manner of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the devices, modules, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein. It should be understood that the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present application.

Claims

1. A resource management method, characterized in that: include: Determine a task communication port corresponding to a task issuing end, and determine a distributed task distribution server corresponding to the task communication port, wherein the task communication port is used for task communication between the task issuing end and a task execution end, and the task execution end is used to execute a target task issued by the task issuing end through the distributed task distribution server; Determine the task status corresponding to the target task by monitoring the communication status between the task communication port and the task issuing terminal; When it is determined that the task state is the exit state, the task occupied resources corresponding to the target task are determined, and the task occupied resources are released.

2. The method according to claim 1, characterized in that Determine the task communication port corresponding to the task publishing end, including: Detecting whether the default task communication port corresponding to the task issuing end is occupied; In the case where it is determined that the default task communication port is occupied, an unoccupied task communication port is determined as the task communication port corresponding to the task issuing end.

3. The method according to claim 2, characterized in that Determining the distributed task distribution server corresponding to the task communication port includes: For the unoccupied task communication port, a new distributed task distribution server is started as the distributed task distribution server corresponding to the unoccupied task communication port.

4. The method according to claim 1, characterized in that: Determining the task status corresponding to the target task by monitoring the communication status between the task communication port and the task issuing end includes: For each preset monitoring period, monitoring whether the task communication port receives task communication information sent by the task issuing end within the monitoring period, wherein the task communication information is communication information related to the target task; If not, it is determined that the task state corresponding to the target task is an exit state.

5. The method according to claim 1, characterized in that In the case where it is determined that the task state is the exit state, determining the task occupied resources corresponding to the target task, and releasing the task occupied resources, including: In the case where it is determined that the task status is an exit status, determining that the task issuing end exits the task execution process, wherein the exit status includes the target task execution completion, the target task execution suspension, and the target task being cancelled by the task issuing end; Clearing the occupation of the task communication port by the task issuing end and the task executing end, and determining the task occupation resources corresponding to the target task according to the task identifier corresponding to the target task; Execute a task termination instruction for the target task to release the resources occupied by the task.

6. The method according to claim 5, characterized in that The task occupied resources at least include processor resources and license resources occupied by the target task in the cluster task resource scheduling system where the target task is located; Executing a task termination instruction for the target task to release resources occupied by the task includes: The cluster task resource scheduling system executes a task termination instruction for the target task to release processor resources and license resources occupied by the target task in the cluster task resource scheduling system.

7. The method according to claim 5, characterized in that The method further comprises: Determine whether the task communication port corresponding to the task issuing end is a default communication port; If not, while clearing the occupation of the task communication port by the task issuing end and the task executing end, the distributed task distribution server corresponding to the task communication port is stopped.

8. The method according to claim 1, characterized in that The method further comprises: When it is determined that the task status is the exit status, data cleaning is performed on the target task data corresponding to the target task, and execution error record information corresponding to the target task during the task execution process is counted.

9. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the resource management method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the resource management method according to any one of claims 1 to 8 is implemented.

Images