Job resource management method and device, electronic equipment and system

Abstract

The invention discloses a job resource management method and device, electronic equipment and a system, relates to the technical field of computers, and can improve the accuracy and efficiency of matching job resources for job requests. The method comprises the steps that a client receives first information indicating that one or more historical jobs requested by the client are executed in a computing node through a first session; then, in response to the received first information, the client obtains connection information of a first session established between the client and the computing node, and obtains job information of a first job to be executed; wherein the connection information comprises one or a combination of the following items: an identifier of the client, an identifier of the computing node and a port adopted by the first session, and the job information comprises an identifier of the first job. Further, the client sends the connection information and the job information to the computing node through a second session, and receives second information which is sent by the computing node and indicates the computing node to allocate job resources for the first job through the second session.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic equipment and system for managing operation resources. Background Art

[0002] Computer Aided Engineering (CAE) and Electronic Design Automation (EDA) software commonly used in the manufacturing industry encapsulates the Message Passing Interface (MPI) and uses the Secure Shell Protocol (SSH) as the communication method between nodes when the MPI job is running. In this way, if SSH is not controlled, the system does not limit the resource usage of the job by default, and the job will have problems such as resource overuse, which will affect the resource usage of other jobs on the computing node.

[0003] When a client (user) initiates a new job request to a computing node, the computing node determines the resources that need to be allocated to the new job request by matching the new job request with the last connected job among the multiple jobs that the client already has. However, this method may result in job matching errors, causing the resources allocated to the new job request to be inconsistent with the expected resources, which in turn causes abnormal situations in the new job. Therefore, the current accuracy of matching job resources for job requests is low and the job efficiency is poor. Summary of the invention

[0004] The present application provides a method, device, electronic device and system for managing job resources, thereby improving the accuracy and efficiency of matching job resources to jobs.

[0005] In a first aspect, a method for managing job resources is provided. The method can be applied to a client, and a first session is established between the client and a computing node.

[0006] The client receives first information sent by the computing node through the first session, indicating that one or more historical jobs requested by the client have been executed in the computing node. Then, in response to the received first information, the client obtains connection information of the first session established between the client and the computing node, and obtains job information of the first job to be executed. The connection information includes one or more of the following items: the identifier of the client, the identifier of the computing node, and the port used by the first session, and the job information includes the identifier of the first job. Further, the client sends the connection information and job information to the computing node through the second session, and receives second information sent by the computing node through the second session, indicating the job resources allocated by the computing node to the first job based on the one or more historical jobs.

[0007] Since the client can receive the first information of the computing node through the first session, which indicates that one or more historical jobs requested by the client have been executed in the computing node, the connection information of the first session and the job information of the first job to be executed are obtained according to the first information, and the connection information and the job information are sent to the computing node through the second session, so that the computing node allocates job resources for the first job according to the one or more historical jobs. Finally, the client receives the second information of the computing node through the second session, and determines the job resources allocated by the computing node for the first job. Through the above method, the client can send the connection information of the first session and the job information of the first job to be executed to the computing node, so that the computing node matches the appropriate job resources for the first job. Therefore, the solution provided by the present application can improve the accuracy and efficiency of matching job resources for jobs.

[0008] In one possible design, a first session established between a client and a computing node is used to verify whether a historical job requested by the client exists on the computing node, and a second session established between the client and the computing node is: a connection established between the client and the computing node through a target command, and the target command is used to obtain connection information of the first session and job information of the first job to be executed.

[0009] The client can establish a second session with the computing node through a target command, and send the connection information of the first session and the job information of the first job to be executed to the computing node through the second session, so that the computing node can match appropriate job resources for the first job based on the connection information of the first session and the job information of the first job to be executed, thereby improving the accuracy of matching resources for the job.

[0010] In a possible design, the client includes a first job management module, and the computing node includes a second job management module, and the first job management module and the second job management module are used to complete information exchange between the client and the computing node. Therefore, the client sends connection information and job information to the computing node through the second session, which can be specifically: the client first sends the connection information and job information to the first job management module, and then the first job management module sends the connection information and job information to the second job management module through the second session, so as to send the connection information and job information to the computing node.

[0011] The client can first send the connection information and the job information to the local first job management module, and based on the communication mechanism between the job management modules, the first job management module sends the connection information and the job information to the second job management module of the computing node through the second session. Based on the communication mechanism between the job management modules, the local first job management module is used to send the connection information and the job information to the second job management module of the computing node, thereby improving the efficiency of managing job resources.

[0012] In another possible design, based on the second job management module included in the computing node, the client sends connection information and job information to the computing node through the second session. Specifically, the client first obtains the address information of the second job management module in the computing node through the target command, and then the client establishes a connection between the client and the second job management module through the target command according to the address information of the second job management module. Thus, based on the connection between the client and the second job management module, the client can directly send connection information and job information to the second job management module in the computing node through the second session.

[0013] The client can obtain the address information of the second job management module in the computing node through the target command, and then directly establish a connection between the client and the second job management module in the computing node through the target command, so that the connection information and the job information can be directly sent to the second job management module of the computing node through the second session. Instead of sending the connection information and the job information to the second job management module of the computing node through the first job management module of the client based on the communication mechanism between the job management modules, the information transmission efficiency is improved.

[0014] In another possible design, the client also needs to first send a connection request for the first job to the computing node to request the computing node to establish a first session; then, after the computing node receives the connection request for the first job, it returns a connection response to the client, so that when the client receives the connection response returned by the computing node, the first session is established with the computing node.

[0015] The client first needs to establish a first session with the computing node, and then execute subsequent steps based on the first session to achieve management of job resources.

[0016] In a second aspect, a method for managing job resources is provided. The method can be applied to a computing node, and a first session is established between the computing node and a client.

[0017] The computing node sends first information indicating that one or more historical jobs requested by the client have been executed in the computing node to the client through the first session. Then, the computing node receives the connection information of the first session and the job information of the first job to be executed sent by the client through the second session; wherein the connection information includes one or more of the following: the identifier of the client, the identifier of the computing node, and the port used by the first session, and the job information includes the identifier of the first job. Thus, the computing node sends second information indicating the job resources allocated by the computing node to the first job according to the one or more historical jobs to the client through the second session and executes the first job based on the job resources allocated to the first job.

[0018] Since the computing node can first send the first information to the client indicating that one or more historical jobs requested by the client have been executed in the computing node, the client can feedback the relevant information of the first job, and receive the connection information of the first session of the client and the job information of the first job to be executed through the second session. Furthermore, the computing node sends an indication to the client through the second session: the computing node sends the second information of the job resources allocated to the first job according to one or more historical jobs, and executes the first job based on the job resources allocated to the first job. Through the above method, the computing node can match the appropriate job resources for the first job based on the connection information of the first session sent by the client and the job information of the first job to be executed. Therefore, the solution provided in the present application can improve the accuracy and efficiency of matching job resources for jobs.

[0019] In one possible design, a first session established between a client and a computing node is used to verify whether a historical job requested by the client exists on the computing node, and a second session established between the client and the computing node is a connection established between the client and the computing node through a target command, and the target command is used to obtain connection information of the first session and job information of the first job to be executed.

[0020] The client can establish a second session with the computing node through a target command, and send the connection information of the first session and the job information of the first job to be executed to the computing node through the second session, so that the computing node can match appropriate job resources for the first job based on the connection information of the first session and the job information of the first job to be executed, thereby improving the accuracy of matching resources for the job.

[0021] In a possible design, the client includes a first job management module, and the computing node includes a second job management module, and the first job management module and the second job management module are used to complete information exchange between the client and the computing node. Therefore, the computing node receives the connection information of the first session and the job information of the first job to be executed sent by the client through the second session, which can be specifically: the second job management module included in the computing node receives the connection information and job information sent by the first job management module of the client through the second session.

[0022] The client can first send the connection information and the job information to the local first job management module, and based on the communication mechanism between the job management modules, the first job management module sends the connection information and the job information to the second job management module of the computing node through the second session. Based on the communication mechanism between the job management modules, the local first job management module is used to send the connection information and the job information to the second job management module of the computing node, thereby improving the efficiency of managing job resources.

[0023] In another possible design, based on a second job management module included in the computing node, the computing node receives connection information of the first session and job information of the first job to be executed sent by the client through a second session. Specifically, the second job management module included in the computing node first establishes a connection with the client, and then receives the connection information and job information sent by the client through the second session based on the second job management module.

[0024] The client can first establish a connection with the second job management module in the computing node, so that the connection information and the job information can be directly sent to the second job management module of the computing node through the second session. Instead of using the communication mechanism between the job management modules, the connection information and the job information are sent to the second job management module of the computing node through the first job management module of the client, thereby improving the information transmission efficiency.

[0025] In another possible design, the computing node also needs to receive a connection request for the first job sent by the client, and first determine whether there is a historical job requested by the client that has been executed based on the received connection request for the first job. Thus, when it is determined that there are one or more historical jobs requested by the client that have been executed, the computing node sends a connection response to the client and establishes a first session with the client.

[0026] The computing node first needs to establish a first session with the client, and then determine whether there is a historical job requested by the client that has been executed based on the first session, and send a connection response to the client to establish the first session with the client, so as to perform subsequent steps of matching job resources for the job.

[0027] In a third aspect, a device for managing job resources is provided. The device for managing job resources is applied to a client, a first session is established between the client and a computing node, and the device for managing job resources includes: a first communication module and a first processing module.

[0028] The first communication module is used for the client to receive first information of the computing node through the first session, and the first information indicates that one or more historical jobs requested by the client have been executed in the computing node.

[0029] The above-mentioned first processing module is used for the client to respond to the first information to obtain the connection information of the first session and the job information of the first job to be executed, wherein the connection information includes one or a combination of the following items: the client's identifier, the computing node's identifier and the port used by the first session, and the job information includes the identifier of the first job.

[0030] The first communication module is also used for the client to send connection information and operation information to the computing node through the second session.

[0031] The first communication module is also used for the client to receive second information of the computing node through a second session, where the second information indicates: the job resources allocated by the computing node to the first job according to one or more historical jobs.

[0032] In one possible design, the first session is used to verify whether the historical job requested by the client exists on the computing node, and the second session is the connection between the client and the computing node established through the target command, and the target command is used to obtain the connection information of the first session and the job information of the first job to be executed.

[0033] In another possible design, the client includes a first job management module, and the computing node includes a second job management module. The first job management module and the second job management module are used to complete information exchange between the client and the computing node; the first communication module is specifically used for the client to send connection information and job information to the first job management module. The first communication module is also specifically used for the first job management module to send connection information and job information to the second job management module through the second session.

[0034] In another possible design, the computing node includes a second job management module, and the first communication module is specifically used for the client to obtain the address information of the second job management module in the computing node through a target command. The first processing module is specifically used for the client to establish a connection between the client and the second job management module through a target command according to the address information of the second job management module; and the first communication module is specifically used for the client to send connection information and job information to the second job management module in the computing node through a second session.

[0035] In another possible design, the first communication module is also used by the client to send a connection request for the first job to the computing node, and the connection request for the first job is used to request to establish a first session; the first communication module is also used by the client to receive a connection response returned by the computing node and establish a first session with the computing node.

[0036] In a fourth aspect, a management device for job resources is provided. The management device for job resources is applied to a computing node. A first session is established between the computing node and a client. The management device for job resources includes: a second communication module and a second processing module.

[0037] The second communication module is used for the computing node to send first information to the client through the first session, and the first information indicates that one or more historical jobs requested by the client have been executed in the computing node.

[0038] The above-mentioned second communication module is also used for the computing node to receive the connection information of the first session of the client and the job information of the first job to be executed through the second session, the connection information includes one or a combination of the following items: the client's identifier, the computing node identifier and the port used by the first session, and the job information includes the identifier of the first job.

[0039] The second communication module is also used for the computing node to send second information to the client through the second session, where the second information indicates: the job resources allocated by the computing node to the first job according to one or more historical jobs.

[0040] The second processing module is used to calculate the node to execute the first job based on the job resources allocated to the first job.

[0041] In one possible design, the first session is used to verify whether the historical job requested by the client exists on the computing node, and the second session is the connection between the client and the computing node established through the target command, and the target command is used to obtain the connection information of the first session and the job information of the first job to be executed.

[0042] In one possible design, the client includes a first job management module, and the computing node includes a second job management module. The first job management module and the second job management module are used to complete information interaction between the client and the computing node; the above-mentioned second communication module is specifically used for the second job management module to receive connection information and job information sent by the first job management module through the second session.

[0043] In another possible design, the computing node includes a second job management module, and the second communication module is specifically used for the second job management module to establish a connection with the client; the second communication module is also specifically used for the second job management module to receive the client's connection information and job information through the second session.

[0044] In another possible design, the second communication module is further used for the computing node to receive a connection request for a first job sent by a client, and the connection request for the first job is used to request to establish a first session. The second processing module is further used for the computing node to determine whether there is a historical job requested by the client that has been executed based on the connection request for the first job. The second communication module is further used for the computing node to send a connection response to the client to establish a first session with the client when it is determined that there are one or more historical jobs requested by the client that have been executed.

[0045] In a fifth aspect, an electronic device is provided, which is a client or a computing node, the client comprising a memory and a processor, the memory being coupled to the processor; the memory being used to store computer program code, the computer program code comprising computer instructions; when the computer instructions are executed by the processor, the client executes a method as described in the first aspect and any possible design thereof, or as described in the second aspect and any possible design thereof.

[0046] In a sixth aspect, a management system for job resources is provided, which includes a client and a computing node, and is used to execute the method described in the first aspect and any possible design method thereof, or the method described in the second aspect and any possible design method thereof.

[0047] In the seventh aspect, a computer storage medium is provided, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect and any possible design thereof, or the method described in the second aspect and any possible design thereof.

[0048] In an eighth aspect, a computer program product is provided, which, when executed on a computer, enables the computer to execute the method described in the first aspect and any possible design thereof, or the method described in the second aspect and any possible design thereof.

[0049] It can be understood that the beneficial effects that can be achieved by the method described in the second aspect and any possible design thereof provided above, the job resource management device described in the third aspect and any possible design thereof, the job resource management device described in the fourth aspect and any possible design thereof, the electronic device described in the fifth aspect, the job resource management system described in the sixth aspect, the computer storage medium described in the seventh aspect, and the computer program product described in the eighth aspect can be referred to the beneficial effects in the first aspect and any possible design thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a process for a scheduler provided in this application to match job resources to a job;

[0051] Figure 2 A schematic diagram of the composition of a management system architecture for operation resources provided in this application;

[0052] Figure 3 A schematic diagram of the composition of another management system architecture of job resources provided in this application;

[0053] Figure 4 A schematic diagram of the composition of another management system architecture of job resources provided in this application;

[0054] Figure 5 A flowchart of a method for managing operation resources provided for this application;

[0055] Figure 6 A schematic diagram of the composition of another management system architecture of job resources provided in this application;

[0056] Figure 7 A schematic diagram of the composition of a management device for operation resources provided in this application;

[0057] Figure 8 A schematic diagram of another operating resource management device provided by the present application;

[0058] Fig. 9 A schematic diagram of the structural composition of an electronic device provided in this application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] In this application, the character " / " generally indicates that the objects before and after are in an "or" relationship. For example, A / B can be understood as A or B.

[0061] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0062] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include other steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0063] In addition, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present concepts in a specific way.

[0064] At present, when the mainstream scheduler in the industry matches job resources for jobs, it adds a customized pluggable authentication module (PAM) SSH daemon (SSHD) authentication component (for example, pam_custom.so module) to the computing node (for example, SSH Server node). When the computing node receives the SSH-based job request sent by the client, it uses the customized authentication component to communicate with the Agent service on the computing node to determine whether the client has a running job on the computing node and authenticate the client's connection request. When the client's connection request is successfully authenticated, the computing node writes the process identifier (Process ID, PID) corresponding to the job request to the control group (Control Group, Cgroup) corresponding to the job request to manage the sub-processes remotely started by the client, so that the resources available to the job and its sub-processes are limited by the resources applied for by the job request on the computing node, avoiding the situation of excessive resource usage.

[0065] It should be noted that Cgroup is a group of processes that perform resource restrictions, priority adjustment, and account management operations to control and manage the allocation of system resources. Agent service is a scheduler subservice deployed on the computing node, responsible for the management of jobs and resources on the computing node. A job is a collection of program instances that need to be executed to complete a specific computing business. When a user submits a job, he needs to declare the resources required to execute the job, the organization that provides the resources, and the queue to which it belongs. The actual execution of the job is completed through tasks.

[0066] For example, Figure 1 As shown in the figure, the process of matching job resources for jobs by the current mainstream scheduler is as follows:

[0067] S101. After receiving the SSH-based job request sent by the client, the PAM module on the computing node first verifies it through the default PAM SSHD modules, and then sends it to the customized PAM SSHD authentication component: pam_custom.so module (ie, authentication module).

[0068] S102. Connect to the Agent service in the computing node through the pam_custom.so module, and send the job request to the Agent service.

[0069] S103. After receiving the job request, the Agent service checks whether there is a running job corresponding to the client. When it is determined that there is a running job corresponding to the client, the PID corresponding to the job request is written into the control group corresponding to the job to limit the session resources of the job.

[0070] S104. After the Agent service processing is completed, a response message is returned to the pam_custom.so module. After receiving the response message, the pam_custom.so module returns the job request authentication result to the PAM module according to the response message. The PAM module configures the rules to decide whether to continue to execute other authentication modules and finally determine whether to allow the client's job request to log in.

[0071] However, in the above method, if Figure 2 As shown in the figure, when the computing node includes jobs A and B, since the current job request does not contain job information, the computing node usually matches the job request with the job resources of the latest running job. For example, after the Agent service receives the job request, it queries the existing running jobs A and B. The start time of job A is T2, and the start time of job B is T1. T2 is later than T1. Then the Agent service will match the job resources of job A, which has a later start time, to the current job request.

[0072] In this case, there is a certain probability that the job resources may be matched incorrectly, and the client process executing the job and the computing node process executing the job will not be in the same job. This will not meet the requirements of MPI multi-node job running (that is, the job session started by the application through SSH is always in the same job), resulting in resource usage not meeting expectations, which in turn leads to job resource overuse or job failure. As the cluster scales up and the job model becomes more complex, the probability of job resource mismatch increases.

[0073] like Figure 3 As shown in the figure, after receiving the job request, the Agent service finds that there are running jobs A and C. The start time of job A is T2, and the job resource amount is 512M; the start time of job C is T3, and the job resource amount is 128M. T3 is later than T2, so the Agent service will match the job resources of job C with a later start time to the current job request. However, the requested resources of job A are 512M. If the job resources of job C with a later start time are matched to the current job request, the job resources matched to the current job request are 128M. The allocated resource amount does not match the requested resource amount, which will cause resource matching failure and thus job failure.

[0074] To this end, an embodiment of the present application provides a method for managing job resources. The method can be applied to the process of matching job resources for jobs. In the present application, the client can send the connection information of the session and the job information of the first job to be executed to the computing node based on the first information of the historical job executed by the computing node, and the computing node queries the job resources matching the first job from the executed historical jobs according to the connection information of the session and the job information of the first job. Compared with the poor accuracy of the job resources caused by the computing node allocating job resources to the job to be executed according to the last executed job resources in the conventional technology, in the present application, since the client can communicate the connection information of the session with the computing node, in the matching process of the computing node executing the job resources, not only can the matching situation of the job resources of the historical job executed locally be referred to, but also the connection information of the session between the client and the computing node can be combined, so that the job resources allocated by the computing node for the first job are matched with the idle job resources in the computing node, thereby improving the accuracy and efficiency of the matching process of the job resources.

[0075] refer to Figure 4 The method for managing job resources provided in the embodiment of the present application can be applied to an implementation environment composed of a client and a computing node (i.e., a management system for job resources). Figure 4 As shown, the implementation environment may include a client 401 and a computing node 402 .

[0076] The client 401 includes: an SSH client module, a target command and an Agent service.

[0077] The SSH client module refers to a control module of the client 401 , and the SSH client module is used to establish a session with the computing node 402 to implement job management.

[0078] The target command refers to an SSH hook command extended in the client 401, which can be a Local command. The target command is used to obtain the client ID, the computing node ID, and the port used by the session, and to obtain the ID of the job to be executed. In some possible examples, the target command can also be called a custom data collection command.

[0079] The Agent service refers to a scheduler subservice deployed on a client or computing node, responsible for the management of jobs and resources on the client or computing node. The Agent service in the client 401 includes a first job management module, which is used to implement the communication function with the computing node.

[0080] The computing node 402 includes: a PAM module, an Agent service, and a Cgroup module.

[0081] PAM module refers to Pluggable Authentication Modules (PAM), which is an authentication mechanism that separates the services provided by the system from the authentication methods of the services by providing some dynamic link libraries and a unified API. This allows system administrators to flexibly configure different authentication methods for different services as needed without changing the service programs. It also facilitates adding new authentication methods to the system.

[0082] The PAM module includes an SSHD authentication component, which is a customized authentication component. The customized authentication component may be a pam_custom.so module, which is used to determine whether there is a historical job requested by a client that has been executed on the computing node 402 .

[0083] The Agent service in the computing node 402 includes a second job management module and an authentication module. The first job management module in the client and the second job management module in the computing node are used to complete information interaction between the client and the computing node.

[0084] The Cgroup module refers to the control group in the computing node, which is used to perform operations such as resource limitation, priority adjustment, and account management on a group of processes to control and manage the allocation of system resources.

[0085] An application is running on the client 401 and the computing node 402, and the application may include but is not limited to executing various jobs. Taking the application including multiple jobs to be executed (such as job A and job B) as an example, when the client 401 needs to execute a certain function, a job is established and the job resources are requested from the computing node 402, so that the client 401 and the computing node 402 interact with each other with relevant data information, and the required job resources are allocated to the job request of the client 401 on the computing node 402.

[0086] It should be noted that, combined with Figure 4 As shown, the communication mechanism between job management modules can be used. The client first sends the connection information and job information to the local first job management module, and then sends the connection information and job information to the second job management module of the computing node through the local first job management module of the client based on the second session. The job management module is a module in the Agent service. The communication mechanism between job management modules can be understood as follows: when running MPI jobs on multiple nodes, the nodes can communicate with each other through the Agent job management module.

[0087] Exemplarily, the client in the embodiments of the present application may be a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an augmented reality (AR) or a virtual reality (VR) device, and the embodiments of the present application do not impose any special restrictions on the specific form of the device.

[0088] The execution subject of the job resource management method provided in the present application may be the central processing unit (CPU) of the client, or a control module in the client for implementing the management of job resources, or an application system in the client for implementing the management of job resources.

[0089] The computing node in the embodiment of the present application is used to process jobs. The computing node can be a server, desktop, notebook, mobile phone, cloud device or virtual machine, etc. A way to access service logic can be provided for use by the client application program (system). The computing node can provide a simple and manageable access mechanism to system resources for the application. It also provides services such as the implementation of Hyper Text Transfer Protocol (HTTP) and database connection management.

[0090] The technical solution provided in the embodiment of the present application can be applied to the above implementation environment. The implementation environment described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. It is known to those skilled in the art that with the evolution of the implementation environment, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0091] The methods in the following embodiments can all be implemented in the client and computing node with the above implementation environment. In the following embodiments, the execution subject of the job resource management method provided by the present application is the client and computing node as an example to illustrate the method of the embodiment of the present application.

[0092] The present application embodiment provides a method for managing job resources, such as Figure 5 As shown, the method for managing the job resources may include S501-S506.

[0093] S501: The client first sends a connection request for a first job to a computing node, so as to request the computing node to establish a first session through the connection request for the first job.

[0094] Corresponding to the process of S501 , the computing node receives a connection request for the first job sent by the client.

[0095] In a possible implementation, after the computing node receives the connection request for the first job sent by the client, the computing node determines whether there is a historical job requested by the client that has been executed based on the connection request for the first job; if it is determined that there are one or more historical jobs requested by the client that have been executed, the computing node sends a connection response to the client. When the client receives the connection response returned by the computing node, the first session with the computing node is established.

[0096] It can be understood that the client first needs to send a connection request for the first job to the computing node, and only after completing the connection with the computing node can the first session be established. The connection request for the first job is also used to trigger the computing node to determine whether there is at least one historical job requested by the client locally, and the connection request for the first job may include information such as the identifier of the client, the identifier of the computing node, and the port used by the first session.

[0097] In one possible implementation, the computing node connects to the second job management module of the computing node through a target component (for example, a customized PAM SSHD authentication component) based on a connection request of the first job, and determines whether the second job management module has at least one historical job requested by the client; and when it is determined that the second job management module has at least one historical job requested by the client, a connection is established with the client to establish a first session.

[0098] It should be noted that determining whether the client has a historical job requested by the client on the computing node is for matching appropriate job resources for the first job based on the job resources of the historical jobs.

[0099] In some examples, when the client needs to execute the first job on the computing node, the client needs to send a connection request for the first job to the computing node, so that after receiving the connection request for the first job, the computing node connects to the local Agent service of the computing node through a customized PAM SSHD authentication component (which can be a pam_custom.so module), determines whether the client has a requested historical job on the computing node, and authenticates the connection request for the first job. After successful authentication, the connection information (the client identifier, the computing node identifier, and the port used by the first session, etc.) included in the connection request for the first job needs to be cached in the local Agent service of the computing node.

[0100] It should be noted that the establishment of a connection between the client and the computing node can be understood as the client connecting to the Agent service in the computing node through the customized PAMSSHD authentication component (ie, the pam_custom.so module).

[0101] The client first sends a connection request for the first job to the computing node, and then, when the client establishes a connection with the computing node, first determines whether there is at least one historical job requested by the client on the computing node. When it is determined that there is at least one historical job requested by the client on the computing node, the client first needs to establish a first session with the computing node, and then execute subsequent steps based on the first session to achieve management of job resources.

[0102] S502: The computing node sends first information to the client.

[0103] Corresponding to the process of S502, the client receives the first information of the computing node through the first session.

[0104] The first information indicates that one or more historical jobs requested by the client have been executed in the computing node, and a first session is established between the client and the computing node. The first session is used to verify whether the historical jobs requested by the client exist on the computing node.

[0105] In a possible implementation manner, after the client establishes a connection with the computing node and completes establishment of the first session, the computing node may send the first information to the client through the first session.

[0106] S503: The client obtains connection information of the first session and job information of the first job to be executed in response to the first information.

[0107] The connection information includes one or a combination of the following: an identifier of the client, an identifier of the computing node, and a port used by the first session, and the job information includes an identifier of the first job.

[0108] It should be noted that the client identifier is also called the client ID, which is a string that can uniquely identify the client and is often used in applications such as big data analysis and tracking. The main function of the client identifier is to allow websites or applications to identify the identity of visitors and provide them with different content and services. For example, the client identifier can be: Client_1 or a specific name.

[0109] The identifier of the computing node refers to an identification code used to indicate the identity information of the computing node, and may be a character string. For example, the identifier of the computing node may be: Node A or Node B, etc.

[0110] The port used by the first session refers to a network port in the client used to carry the first session, and may be specifically indicated by a port number of the network port, for example, the port number may be Port_1 or Port_2.

[0111] In the client, the identifier of the first job may be generated during the process of the client running the application. Alternatively, in some optional situations, the identifier of the first job may also be sent to the client by other devices. For example, the job identifier of the first job is: Job_1 or other types of identifiers, etc. This application does not limit the description method of the job identifier.

[0112] In one possible implementation, when a first session is established between a client and a computing node and it is determined that at least one historical job requested by the client exists on the computing node, the client can obtain connection information of the first session and job information of the first job to be executed through a target command.

[0113] In a possible implementation, the target command may be an extended SSH hook command, such as a localCommand command, which is a custom data collection command for collecting connection information of the first session and job information of the first job.

[0114] In some examples, after a client establishes a first session with a computing node and verifies that at least one historical job requested by the client exists on the computing node, it is necessary to first cache the connection information included in the connection request of the first job into the Agent service of the computing node, and then return a connection response message to the client.

[0115] Furthermore, by executing the target command through the client, the client identifier (user), the computing node identifier (host) and the port identifier (port) corresponding to the first job, and the identifier of the first job (jobId) can be collected through the target command, and the collected information can be passed to the local Agent service of the client.

[0116] S504: The client sends connection information and operation information to the computing node through the second session.

[0117] Corresponding to the process of S504, the computing node receives the connection information of the first session of the client and the job information of the first job to be executed through the second session.

[0118] The second session refers to a session in which the client sends connection information and job information to the computing node, which is a session between the client and the second job management module in the computing node; or, it is a session between the first job management module in the client and the second job management module in the computing node.

[0119] In a possible implementation, the second session is a connection between the client and the computing node established through a target command, and the target command is used to obtain connection information of the first session and job information of the first job to be executed.

[0120] Compared with the first session in the above embodiment, the similarities between the second session and the first session are: both are sessions between the client and the computing node for transmitting data information; but the difference between the second session and the first session is: the first session is a session between the client and the PAM module in the computing node, which is used to verify the job request; the second session is a session between the client (or the first job management module) and the second job management module in the computing node, which is used for the client to send connection information and job information to the computing node, so as to allocate job resources to the job based on the connection information and job information.

[0121] Exemplarily, the second session may refer to but is not limited to the following types of communication connections: utilizing the communication mechanism between job management modules, the Agent service on the client side sends information to the Agent service on the target node side, thereby realizing information transmission between the client and the computing node.

[0122] It can be understood that the computing node receives the connection information of the first session of the client and the job information of the first job to be executed through the second session, including: the second job management module first establishes a connection with the client, and then the second job management module receives the connection information and job information of the client through the second session.

[0123] Combination Figure 4The management system architecture of the job resources shown is an exemplary illustration of the above S504: the SSHclient module sends connection information and job information to the first job management module, so that the first job management module sends connection information and job information to the second job management module through the second session.

[0124] It can be understood that the computing node receives the connection information of the first session of the client and the job information of the first job to be executed through the second session, including: the second job management module receives the connection information and job information sent by the first job management module through the second session.

[0125] The client can first send the connection information and the job information to the local first job management module, and based on the communication mechanism between the job management modules, the first job management module sends the connection information and the job information to the second job management module of the computing node through the second session. Based on the communication mechanism between the job management modules, the local first job management module is used to send the connection information and the job information to the second job management module of the computing node, thereby improving the efficiency of managing the job resources. There is no need to first establish a connection between the client and the second job management module of the computing node, and then directly send the connection information and the job information to the second job management module of the computing node through the client, thereby reducing the amount of tasks to establish connections between modules.

[0126] In a possible implementation, the client sends the connection information and the job information to the computing node through the second session, including: the client obtains the address information of the second job management module in the computing node through a target command, and then establishes a connection between the client and the second job management module through a target command according to the address information of the second job management module. Finally, the connection information and the job information are sent to the second job management module in the computing node through the second session.

[0127] It should be noted that if Figure 6 As shown, in order to reduce the number of modules that are passed through when sending connection information and job information, the client can directly send the connection information and job information to the second job management module of the computing node without passing through the local first job management module when transmitting the connection information and job information. In this case, it is necessary to first establish a connection between the client and the second job management module of the computing node based on the address information of the second job management module of the computing node.

[0128] In this implementation method, it is necessary to first establish a connection between the client and the second job management module of the computing node based on the address information of the second job management module of the computing node, so that when the connection information and job information are sent from the client to the second job management module of the computing node, there is no need to pass through the local first job management module of the client.

[0129] The client can obtain the address information of the second job management module in the computing node through the target command, and then directly establish a connection between the client and the second job management module in the computing node through the target command, so that the connection information and the job information can be directly sent to the second job management module of the computing node through the second session. Instead of sending the connection information and the job information to the second job management module of the computing node through the first job management module of the client based on the communication mechanism between the job management modules, the information transmission efficiency is improved.

[0130] S505: The computing node may determine, based on the connection information and the job information, a first job resource matching the first job from at least one job resource corresponding to at least one historical job requested by the client.

[0131] In one possible implementation, the computing node can compare information such as the client identifier, the computing node identifier, and the port used by the first session included in the connection information with information corresponding to at least one historical job requested by the executed client, thereby determining a historical job related to the first job from the at least one historical job.

[0132] Furthermore, the job resources corresponding to the historical jobs related to the first job are determined as the job resources matching the first job.

[0133] In a possible implementation, the computing node may also compare the job information of the first job with the job information of at least one historical job to determine a historical job related to the first job.

[0134] It should be noted that, after determining the job resources matching the first job, the computing node may obtain the second information based on the resource information of the job resources matching the first job.

[0135] S506: The computing node sends second information to the client through the second session.

[0136] Corresponding to the process of S506 , the client receives the second information of the computing node through the second session.

[0137] The second information indicates: the job resources allocated by the computing node to the first job according to one or more historical jobs.

[0138] In a possible implementation, the computing node further executes the first job based on the job resources allocated to the first job.

[0139] In a possible implementation, the computing node determines a job resource matching the first job from at least one job resource based on the connection information and the job information, and executes the first job based on the determined job resource. The at least one job resource is a resource that executes at least one historical job, and one job resource executes one job.

[0140] It should be noted that the job resources may be central processing unit (CPU) resources, or may be graphics processing unit (GPU) resources, etc.

[0141] In one possible implementation, after the Agent service in the computing node receives the connection information and the job information through the second job management module, it needs to determine whether the information included in the connection information and the job information matches the information included in the connection request of the first job. If the information matches, the PID of the first job is written into the control group corresponding to the job to ensure that the job session started by the application through SSH always runs in the same job.

[0142] In some examples, the computing node determines a first job resource that matches the first job from multiple job resources based on the resource amount of the job resources for executing each historical job. For example, the resource amount of the job resources required by the first job is 512M, so it is necessary to determine a job resource with a resource amount of 512M from multiple job resources and allocate it to the first job.

[0143] The embodiment of the present application, through the extended SSH hook command (i.e., the local Command command, a custom data collection command), and by utilizing the mutual communication mechanism between the job management modules in the Agent service, while the client is connected to the computing node, additional information is supplemented and transmitted to help the Agent service select the correct job, so that the job session and the subprocess can correctly use the corresponding job resources, and solve the problem that the job resources cannot be accurately matched when there are multiple jobs of the same user in the computing node. It can ensure that the job session started by the application through SSH always runs in the same job, avoiding the situation where the process of the client executing the job and the process of the computing node executing the job are not in the same job, and failing to meet the requirement that the session of the MPI multi-node running job always runs in the same job, thereby avoiding the situation where the resource usage does not meet the expectation, resulting in the situation where the job resources are overused or the job fails.

[0144] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the client and the computing node. It is understandable that the client and the computing node include hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the steps of a method for managing job resources of each example described in the embodiment disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or electronic device software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0145] The embodiment of the present application can divide the client and the computing node into functional modules or functional units according to the above method example. For example, each functional module or functional unit can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules or functional units. Among them, the division of modules or units in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation.

[0146] Please refer to Figure 7 , which shows a schematic diagram of a management device for job resources provided in an embodiment of the present application, the management device for job resources is applied to a client, and a first session is established between the client and a computing node. Figure 7 As shown, the operation resource management device 700 may include: a first communication module 701 and a first processing module 702 .

[0147] The first communication module 701 is used for the operation resource management device 700 to execute: the client receives the first information of the computing node through the first session, and the first information indicates: the computing node has executed one or more historical jobs requested by the client. For example, the first communication module 701 is used to support the operation resource management device 700 to execute S501 in the above method embodiment, and / or other processes for the technology described herein.

[0148] The first processing module 702 is used for the operation resource management device 700 to execute: the client obtains the connection information of the first session and the operation information of the first job to be executed in response to the first information, the connection information includes one or a combination of the following items: the client's identifier, the computing node's identifier and the port used by the first session, and the operation information includes the identifier of the first job. For example, the first processing module 702 is used to support the operation resource management device 700 to execute S502 in the above method embodiment, and / or other processes for the technology described herein.

[0149] The first communication module 701 is also used for the operation resource management device 700 to execute: the client sends connection information and operation information to the computing node through the second session. For example, the first communication module 701 is used to support the operation resource management device 700 to execute S503 in the above method embodiment, and / or other processes for the technology described herein.

[0150] The first communication module 701 is also used for the operation resource management device 700 to execute: the client receives second information of the computing node through the second session, and the second information indicates: the operation resource allocated by the computing node to the first operation according to one or more historical operations. For example, the first communication module 701 is used to support the operation resource management device 700 to execute S504 in the above method embodiment, and / or other processes for the technology described herein.

[0151] Please refer to Figure 8 , which shows a schematic diagram of a management device for a job resource provided in an embodiment of the present application, the management device for the job resource is applied to a computing node, and a first session is established between the computing node and the client. Figure 8 As shown, the operation resource management device 800 may include: a second communication module 801 and a second processing module 802 .

[0152] The second communication module 801 is used by the job resource management device 800 to execute: the computing node sends first information to the client through the first session, and the first information indicates: one or more historical jobs requested by the client have been executed in the computing node.

[0153] The second communication module 801 is also used for the job resource management device 800 to execute: the computing node receives the connection information of the first session of the client and the job information of the first job to be executed through the second session, the connection information includes one or a combination of the following items: the client's identifier, the computing node identifier and the port used by the first session, and the job information includes the identifier of the first job.

[0154] The second communication module 801 is also used for the job resource management device 800 to execute: the computing node sends second information to the client through the second session, and the second information indicates: the job resources allocated by the computing node to the first job according to one or more historical jobs.

[0155] The second processing module 802 is used for the job resource management device 800 to execute: the computing node executes the first job based on the job resources allocated to the first job.

[0156] Some other embodiments of the present application provide an electronic device. The electronic device is a client or a computing node, and the electronic device may include: a memory and one or more processors. The memory and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device may execute the various functions or steps executed by the client in the above method embodiment.

[0157] The present application embodiment also provides an electronic device 900, such as Fig. 9 As shown, the electronic device 900 includes at least one processor 901 and at least one interface circuit 902. The processor 901 and the interface circuit 902 can be interconnected via lines. For example, the interface circuit 902 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 902 can be used to send signals to other devices (such as the processor 901). Exemplarily, the interface circuit 902 can read instructions stored in the memory and send the instructions to the processor 901. When the instructions are executed by the processor 901, the electronic device can execute the various steps in the above embodiments. Of course, the electronic device can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0158] An embodiment of the present application also provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.

[0159] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0160] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0165] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for managing operation resources, characterized in that: Applied to a client, a first session is established between the client and a computing node, the method comprising: The client receives first information of the computing node through the first session, wherein the first information indicates that one or more historical jobs requested by the client have been executed in the computing node; The client obtains, in response to the first information, connection information of the first session and job information of a first job to be executed, wherein the connection information includes one or a combination of the following: an identifier of the client, an identifier of the computing node, and a port used by the first session, and the job information includes an identifier of the first job; The client sends the connection information and the job information to the computing node through a second session; The client receives second information of the computing node through the second session, where the second information indicates: job resources allocated by the computing node to the first job according to the one or more historical jobs.

2. The method according to claim 1, characterized in that The first session is used to verify whether the historical job requested by the client exists on the computing node. The second session is a connection between the client and the computing node established through a target command. The target command is used to obtain connection information of the first session and job information of the first job to be executed.

3. The method according to claim 1 or 2, characterized in that: The client includes a first job management module, and the computing node includes a second job management module, wherein the first job management module and the second job management module are used to complete information interaction between the client and the computing node; The client sends the connection information and the job information to the computing node through the second session, including: The client sends the connection information and the job information to the first job management module; The first job management module sends the connection information and the job information to the second job management module through the second session.

4. The method according to claim 1 or 2, characterized in that: The computing node includes a second job management module, and the client sends the connection information and the job information to the computing node through a second session, including: The client obtains address information of a second job management module in the computing node through a target command; The client establishes a connection between the client and the second job management module through the target command according to the address information of the second job management module; The client sends the connection information and the job information to the second job management module in the computing node through the second session.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The client sends a connection request for the first job to the computing node, where the connection request for the first job is used to request to establish the first session; The client receives a connection response returned by the computing node, and establishes the first session with the computing node.

6. A method for managing operation resources, characterized in that: Applied to a computing node, a first session is established between the computing node and a client, the method comprising: The computing node sends first information to the client through the first session, wherein the first information indicates that one or more historical jobs requested by the client have been executed in the computing node; The computing node receives, through the second session, connection information of the first session of the client and job information of a first job to be executed, wherein the connection information includes one or a combination of the following: an identifier of the client, an identifier of the computing node, and a port used by the first session, and the job information includes an identifier of the first job; The computing node sends second information to the client through the second session, where the second information indicates: job resources allocated by the computing node to the first job according to the one or more historical jobs; The computing node executes the first job based on the job resources allocated to the first job.

7. The method according to claim 6, characterized in that The first session is used to verify whether the historical job requested by the client exists on the computing node. The second session is a connection between the client and the computing node established through a target command. The target command is used to obtain connection information of the first session and job information of the first job to be executed.

8. The method according to claim 6 or 7, characterized in that: The client includes a first job management module, and the computing node includes a second job management module, wherein the first job management module and the second job management module are used to complete information interaction between the client and the computing node; The computing node receives the connection information of the first session of the client and the job information of the first job to be executed through the second session, including: The second job management module receives the connection information and the job information sent by the first job management module through the second session.

9. The method according to claim 6 or 7, characterized in that: The computing node includes a second job management module, and the computing node receives connection information of the first session of the client and job information of the first job to be executed through a second session, including: The second job management module establishes a connection with the client; The second job management module receives the connection information and the job information of the client through the second session.

10. The method according to any one of claims 6 to 9, characterized in that: The method further comprises: The computing node receives a connection request for the first job sent by the client, where the connection request for the first job is used to request to establish the first session; The computing node determines, based on the connection request of the first job, whether there is a historical job of the client request that has been executed; When it is determined that there are one or more historical jobs requested by the client that have been executed, the computing node sends a connection response to the client to establish the first session with the client.

11. A management device for operation resources, characterized in that: The operation resource management device is applied to a client, a first session is established between the client and a computing node, and the operation resource management device includes: A first communication module, configured for the client to receive first information of the computing node through the first session, wherein the first information indicates that one or more historical jobs requested by the client have been executed in the computing node; A first processing module is configured to, in response to the first information, obtain, by the client, connection information of the first session and job information of a first job to be executed, wherein the connection information includes one or a combination of the following: an identifier of the client, an identifier of the computing node, and a port used by the first session, and the job information includes an identifier of the first job; The first communication module is further used for the client to send the connection information and the operation information to the computing node through the second session; The first communication module is further used for the client to receive second information about the computing node through the second session, where the second information indicates: job resources allocated by the computing node to the first job according to the one or more historical jobs.

12. A management device for operation resources, characterized in that: The operation resource management device is applied to a computing node, a first session is established between the computing node and a client, and the operation resource management device includes: A second communication module is used for the computing node to send first information to the client through the first session, wherein the first information indicates that one or more historical jobs requested by the client have been executed in the computing node; The second communication module is further configured to cause the computing node to receive, through the second session, connection information of the first session of the client and job information of a first job to be executed, wherein the connection information includes one or a combination of the following: an identifier of the client, an identifier of the computing node, and a port used by the first session, and the job information includes an identifier of the first job; The second communication module is further used for the computing node to send second information to the client through the second session, where the second information indicates: the job resources allocated by the computing node to the first job according to the one or more historical jobs; The second processing module is used for the computing node to execute the first job based on the job resources allocated to the first job.

13. An electronic device, characterized in that: The electronic device is a client or a computing node, and includes a memory and a processor, wherein the memory is coupled to the processor; the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-5 or any one of claims 6-10.

14. A management system for operation resources, characterized in that: The management system of the job resources includes a client and a computing node, and the management system of the job resources is used to execute the method as claimed in any one of claims 1 to 5 or any one of claims 6 to 10.

Images