Method, device and storage medium for allocating processing tasks in a distributed system

Abstract

The embodiment of the present application provides a method, device and storage medium for allocating processing tasks in a distributed system, wherein the method includes: receiving a login request initiated by a target user; responding to the login request, allocating target processing resources on a target distributed node to the target user based on resource usage information of the target distributed node and historical usage information of the target user, wherein the resource usage information is used to indicate the usage of processing resources in the target distributed node, and the historical usage information is used to indicate the usage of processing resources by the target user at a historical time; sending a login response to the target user; receiving the target processing task issued by the target user through the target processing resource, and allocating the target processing task to multiple distributed nodes. Through the present application, the problem of low efficiency in the allocation of processing tasks in a distributed system is solved, thereby achieving the effect of improving the efficiency of the allocation of processing tasks in a distributed system.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more specifically, to a method, device, and storage medium for allocating processing tasks in a distributed system. Background Art

[0002] A distributed system is a computing cluster composed of multiple servers. Users log in to the target distributed node and publish computing tasks to the distributed nodes in the distributed system. Currently, the distributed system allows multiple users to use it simultaneously. However, when the processing resources in the distributed node used for login are limited, multiple users publishing processing tasks on the same distributed node will jointly occupy the resources of the distributed node, resulting in mutual interference of processing resources. When the application run by a user occupies a large amount of processing resources, the performance of the distributed node is reduced and other users cannot access or use it normally, resulting in reduced processing efficiency of the distributed system.

[0003] In view of the problems in related technologies such as low efficiency in allocating processing tasks in distributed systems, no effective solutions have been proposed. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, and storage medium for allocating processing tasks in a distributed system, so as to at least solve the problem of low efficiency in allocating processing tasks in a distributed system in the related art.

[0005] According to one embodiment of the present application, a method for allocating processing tasks in a distributed system is provided. The distributed system includes multiple distributed nodes, each of which is configured to execute received processing tasks. The method is applied to a target distributed node configured with a login function among the multiple distributed nodes, the login function being configured to receive and allocate processing tasks published by a user on the distributed system to the multiple distributed nodes. The method includes:

[0006] receiving a login request initiated by a target user, wherein the login request is used to request to log in to the distributed system;

[0007] In response to the login request, allocating target processing resources on the target distributed node to the target user based on resource usage information of the target distributed node and historical usage information of the target user, wherein the resource usage information is used to indicate usage of the processing resources in the target distributed node, and the historical usage information is used to indicate usage of the processing resources by the target user in historical time;

[0008] Sending a login response to the target user, wherein the login response is used to indicate that the target user is allowed to log in to the distributed system and to publish a processing task on the target processing resource;

[0009] The target processing task issued by the target user is received through the target processing resource, and the target processing task is distributed to the multiple distributed nodes.

[0010] In an exemplary embodiment, allocating target processing resources on the target distributed node to the target user based on the resource usage information of the target distributed node and the historical usage information of the target user includes: detecting the resource usage parameters of the target distributed node and the historical usage parameters of the target user, wherein the resource usage information includes the resource usage parameters, the historical usage information includes the historical usage parameters, the resource usage parameters are used to indicate the amount of resources allowed to be allocated on the target distributed node, and the historical usage parameters are used to indicate the amount of usage of the allocated processing resources by the target user at the historical time; filtering the target processing resources from the processing resources indicated by the resource usage parameters based on the parameter relationship between the resource usage parameters and the historical usage parameters; and allocating the target processing resources to the target user.

[0011] In an exemplary embodiment, the target processing resource is screened from the processing resources indicated by the resource usage parameter based on the parameter relationship between the resource usage parameter and the historical usage parameter, including: when the parameter relationship is used to indicate that the resource usage parameter is greater than or equal to the historical usage parameter, determining the processing resource with the usage amount indicated by the historical usage parameter in the processing resources indicated by the resource usage parameter as the target processing resource; when the parameter relationship is used to indicate that the resource usage parameter is less than the historical usage parameter, determining the processing resource with a preset usage amount in the processing resources indicated by the resource usage parameter as the target processing resource.

[0012] In an exemplary embodiment, allocating the target processing resources to the target user includes: creating a target login image, wherein the target login image includes the environmental resources allowed for the target user to log in to the distributed system; calling the target login image to encapsulate the target processing resources to obtain a target login container, wherein the target login container is configured with a container port of the target login container, and the container port is used to log the target user in to the target login container.

[0013] In an exemplary embodiment, the creation of the target login image includes: creating an initial login image, wherein the initial login image includes the environmental resources allowed for the target user to log in to the distributed system; adding a cluster service, an authentication service, and a synchronization service to the initial login image to obtain the target login image, wherein the cluster service is used to add a container built by the target login image to the distributed system, the authentication service is used to indicate that the container built by the target login image belongs to the distributed system, and the synchronization service is used to add the login function to the container built by the target login image; storing the target login image in a container image repository, wherein the container image repository is used to store container images of all containers allowed to be deployed in the distributed system;

[0014] The calling of the target login image to encapsulate the target processing resources to obtain the target login container includes: calling the image run command to start the target login image in the container image repository, and configuring the number of processor cores and memory space capacity of the target login image through a resource configuration command to obtain the target login container, wherein the target processing resources include the number of processor cores and the memory space capacity, and the resource configuration command includes: a first configuration command and a second configuration command, the first configuration command is docker run-d--cpus=a, docker run-d--cpus represents the operation of configuring the number of processor cores in the target login image, and a is the value of the configured number of processor cores, and the second configuration command is docker run-dm b, docker run-dm represents the operation of configuring the memory space capacity of the target login image, and b is the value of the configured memory space capacity.

[0015] In an exemplary embodiment, after configuring the number of processor cores and memory space capacity of the target login image through a resource configuration command to obtain the target login container, the method further includes: configuring the container connection attributes and container storage attributes of the target login image through a third configuration command, wherein the container connection attributes are used to indicate the calling path of the target login container, and the container storage attributes are used to indicate the storage location of data in the target login container.

[0016] In an exemplary embodiment, receiving the target processing task published by the target user through the target processing resource includes: receiving a connection request sent by the target user to the target processing resource, wherein the connection request is used to request to establish a communication connection with the target processing resource; establishing a target connection between the target user and the target processing resource in response to the connection request; and receiving the target processing task published by the target user to the target processing resource through the target connection.

[0017] According to another embodiment of the present application, a device for allocating processing tasks in a distributed system is provided. The distributed system includes multiple distributed nodes, each of which is configured to execute received processing tasks. The device is applied to a target distributed node configured with a login function among the multiple distributed nodes, the login function being configured to receive and allocate processing tasks published by a user on the distributed system to the multiple distributed nodes. The device includes:

[0018] A receiving module, configured to receive a login request initiated by a target user, wherein the login request is used to request to log into the distributed system;

[0019] an allocation module, configured to respond to the login request and allocate target processing resources on the target distributed node to the target user based on resource usage information of the target distributed node and historical usage information of the target user, wherein the resource usage information is used to indicate usage of the processing resources in the target distributed node, and the historical usage information is used to indicate usage of the processing resources by the target user in historical time;

[0020] a sending module, configured to send a login response to the target user, wherein the login response is used to indicate that the target user is allowed to log in to the distributed system and to allow the target user to publish a processing task on the target processing resource;

[0021] The processing module is configured to receive the target processing tasks issued by the target user through the target processing resources, and distribute the target processing tasks to the multiple distributed nodes.

[0022] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0023] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0024] According to another embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps of any of the above method embodiments when executed by a processor.

[0025] Through this application, a login request initiated by a target user is received, wherein the login request is used to request to log in to a distributed system; in response to the login request, a target processing resource on a target distributed node is allocated to the target user based on the resource usage information of the target distributed node and the historical usage information of the target user, wherein the resource usage information is used to indicate the usage of the processing resources in the target distributed node, and the historical usage information is used to indicate the usage of the processing resources by the target user in historical time; a login response is sent to the target user, wherein the login response is used to indicate that the target user is allowed to log in to the distributed system and that the target user is allowed to publish processing tasks on the target processing resources; the target processing tasks published by the target user are received through the target processing resources, and the target processing tasks are allocated to multiple distributed nodes. Since the target processing resources that are allowed to be used by the target user on the target distributed node are allocated to each target user who requests to log in to the distributed system, it is avoided that multiple users log in to the distributed system and that the processing resources are preempted in the target distributed node. Therefore, the problem of low efficiency in the allocation of processing tasks in the distributed system can be solved, thereby achieving the effect of improving the efficiency of the allocation of processing tasks in the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a hardware structure frame of a server device for a method of allocating processing tasks in a distributed system according to an embodiment of the present application. Figure 2

[0027] Figure 2 The process of the method for allocating processing tasks in a distributed system according to an embodiment of the present application is as follows Figure 2

[0028] Figure 3 This is a schematic diagram of the structure of a target processing resource of a target user according to an embodiment of the present application. Figure 2

[0029] Figure 4 This is a schematic diagram of a configuration structure of a target distributed node according to an embodiment of the present application. Figure 2

[0030] Figure 5 This is a schematic diagram of the structure of a target login container according to an embodiment of the present application. Figure 2

[0031] Figure 6 This is a schematic diagram of a distributed node in a distributed system according to an embodiment of the present application. Figure 2

[0032] Figure 7 This is a structural framework of a distribution device for processing tasks in a distributed system according to an embodiment of the present application. Figure 2

[0033] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure diagram of a server device for a method of allocating processing tasks in a distributed system according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 for communication functions and an input and output device 108. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0037] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for allocating processing tasks in the distributed system in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by a communication provider of the server device. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] In this embodiment, a method for allocating processing tasks in a distributed system is provided. The distributed system includes multiple distributed nodes, each of which is configured to execute received processing tasks. The method is applied to a target distributed node configured with a login function among the multiple distributed nodes. The login function is configured to receive and allocate to the multiple distributed nodes processing tasks published by a user on the distributed system. Figure 2 is a flowchart of a method for allocating processing tasks in a distributed system according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0040] Step S202: receiving a login request initiated by a target user, wherein the login request is used to request to log into the distributed system;

[0041] Step S204: In response to the login request, allocating target processing resources on the target distributed node to the target user based on the resource usage information of the target distributed node and the historical usage information of the target user, wherein the resource usage information is used to indicate the usage of the processing resources in the target distributed node, and the historical usage information is used to indicate the usage of the processing resources by the target user in historical time;

[0042] Step S206: Sending a login response to the target user, wherein the login response is used to indicate that the target user is allowed to log in to the distributed system and to publish a processing task on the target processing resource;

[0043] Step S208: receiving the target processing task issued by the target user through the target processing resource, and distributing the target processing task to the multiple distributed nodes.

[0044] Through the above steps, a login request initiated by a target user is received, wherein the login request is used to request to log in to a distributed system; in response to the login request, a target processing resource on a target distributed node is allocated to the target user according to the resource usage information of the target distributed node and the historical usage information of the target user, wherein the resource usage information is used to indicate the usage of the processing resources in the target distributed node, and the historical usage information is used to indicate the usage of the processing resources by the target user in historical time; a login response is sent to the target user, wherein the login response is used to indicate that the target user is allowed to log in to the distributed system and that the target user is allowed to publish processing tasks on the target processing resources; the target processing tasks published by the target user are received through the target processing resources, and the target processing tasks are allocated to multiple distributed nodes. Since the target processing resources that are allowed to be used by the target user on the target distributed node are allocated to each target user who requests to log in to the distributed system, it is avoided that multiple users log in to the distributed system and that the processing resources are preempted in the target distributed node. Therefore, the problem of low efficiency in the allocation of processing tasks in the distributed system can be solved, thereby achieving the effect of improving the efficiency of the allocation of processing tasks in the distributed system.

[0045] Optionally, in this embodiment, the distributed system includes multiple distributed nodes. A user connects to a target distributed node using the target distributed node's login function and issues processing tasks to the distributed system through the target distributed node. The target distributed node may, but is not limited to, dispatch the user-issued processing tasks to other distributed nodes for executing the processing tasks and return the distributed node's processing results to the user. The distributed system may, but is not limited to, be implemented in the form of a cluster, i.e., a distributed system composed of a group of closely cooperating computer nodes, wherein the computer nodes are the distributed nodes in the distributed system.

[0046] Optionally, in this embodiment, the above-mentioned processing tasks may include, but are not limited to: mathematical model calculation, algorithm calculation, processing and analysis of large-scale data sets, image recognition, video encoding and decoding, image rendering, and any other tasks that require high-performance computing capabilities to accelerate processing.

[0047] Optionally, in this embodiment, the user may interact with the distributed system through, but not limited to, a command line interface, a graphical user interface, or an API (Application Programming Interface), and the distributed system schedules the execution of tasks based on the availability of processing resources and the priority of processing tasks.

[0048] Optionally, in this embodiment, a login function may be configured for one or more distributed nodes in the distributed system, but is not limited to the configuration, and the distributed node with the login function may be determined as a distributed node. Alternatively, a server with the login function may be added to the distributed system as a distributed node in the distributed system.

[0049] In the technical solution provided in the above step S202, the user who initiated the login request through a command line interface, a graphical user interface, an API, etc. can be determined as the target user, but is not limited to it.

[0050] In the technical solution provided in the above step S204, the distributed nodes in the distributed system use the processing resources they include to perform processing tasks. Taking the server as a distributed node in the distributed system as an example, the processing resources on the distributed node may include but are not limited to: CPU (Central Processing Unit), GPU (Graphics Processing Unit), RAM (Random Access Memory) and other hardware and software resources.

[0051] Optionally, in this embodiment, the target distributed node allows multiple users to log in to the target distributed node at the same time and use the processing resources on the target distributed node to publish processing tasks. Each user who has logged in to the target distributed node has processing resources allocated to him or her that he or she is allowed to use. Each user can, but is not limited to, perform multiple login and logout operations in the target distributed node. However, in order to avoid errors caused by repeated logins of the same user, only users who are not logged in are allowed to request to log in to the target distributed node.

[0052] Optionally, in this embodiment, it is possible but not limited to detecting the amount of used processing resources in the target distributed node as the resource usage information of the target distributed node, or detecting the amount of unused processing resources in the target distributed node as the resource usage information of the target distributed node, or detecting the amount of unused and allocated processing resources in the target distributed node as the resource usage information of the target distributed node, etc.

[0053] Optionally, in this embodiment, it is possible but not limited to obtaining the amount of processing resources used by the target user in historical time as the historical usage information of the target user, or obtaining the minimum amount of processing resources used by the target user in historical time as the historical usage information of the target user, etc.

[0054] Optionally, in this embodiment, allocating target processing resources on the target distributed node to the target user based on the resource usage information of the target distributed node and the historical usage information of the target user may include, but is not limited to: determining the resource allocation amount of other users who have logged in to the target distributed node based on the resource usage information of the target distributed node, determining the minimum resource amount of processing resources used by the target user in historical time based on the historical usage information of the target user, and when the resource allocation amount of other users is greater than or equal to the minimum resource amount, determining the processing resources indicated by the minimum resource amount as the target processing resources; or, when the resource allocation amount of other users is less than the minimum resource amount, determining the processing resources indicated by the resource allocation amount as the target processing resources.

[0055] Alternatively, unused processing resources in the target distributed node may be determined based on the resource usage information of the target distributed node, and then the target processing resources may be extracted from the unused processing resources in the target distributed node based on the target user's historical usage information. Alternatively, the target processing resources may be allocated to the target user in other ways based on the resource usage information of the target distributed node and the target user's historical usage information.

[0056] It should be noted that when the target user is logging into the target distributed node for the first time, or when the target user's historical usage information is not obtained, the process of allocating target processing resources on the target distributed node to the target user is executed based on the resource usage information of the target distributed node.

[0057] In an exemplary embodiment, the target processing resources on the target distributed node can be allocated to the target user based on the resource usage information of the target distributed node and the historical usage information of the target user in the following manner, but not limited to: detecting the resource usage parameters of the target distributed node and the historical usage parameters of the target user, wherein the resource usage information includes the resource usage parameters, the historical usage information includes the historical usage parameters, the resource usage parameters are used to indicate the amount of resources allowed to be allocated on the target distributed node, and the historical usage parameters are used to indicate the usage of the allocated processing resources by the target user at the historical time; filtering the target processing resources from the processing resources indicated by the resource usage parameters according to the parameter relationship between the resource usage parameters and the historical usage parameters; and allocating the target processing resources to the target user.

[0058] Optionally, in this embodiment, the target user's usage of the allocated processing resources in historical time may include, but is not limited to: the target user's average usage of the allocated processing resources in historical time, the target user's highest usage of the allocated processing resources in historical time, the target user's lowest usage of the allocated processing resources in historical time, etc.

[0059] In an exemplary embodiment, the target processing resource can be screened from the processing resources indicated by the resource usage parameter based on the parameter relationship between the resource usage parameter and the historical usage parameter in the following manner, but is not limited to: when the parameter relationship is used to indicate that the resource usage parameter is greater than or equal to the historical usage parameter, the processing resource with the usage amount indicated by the historical usage parameter in the processing resources indicated by the resource usage parameter is determined as the target processing resource; when the parameter relationship is used to indicate that the resource usage parameter is less than the historical usage parameter, the processing resource with a preset usage amount in the processing resources indicated by the resource usage parameter is determined as the target processing resource.

[0060] Optionally, in this embodiment, when the amount of resources allowed to be allocated on the target distributed node is greater than or equal to the usage of the allocated processing resources by the target user in historical time, the processing resource with the lowest usage indicated by the historical usage parameters among the processing resources indicated by the resource usage parameters can be determined as the target processing resource, but is not limited to; or, the processing resource with the highest usage indicated by the historical usage parameters among the processing resources indicated by the resource usage parameters can be determined as the target processing resource; or, the processing resource with the average usage indicated by the historical usage parameters among the processing resources indicated by the resource usage parameters can be determined as the target processing resource, etc.

[0061] Optionally, in this embodiment, when the amount of resources allowed to be allocated on the target distributed node is less than the target user's usage of the allocated processing resources in historical time, the processing resources with a preset usage amount among the processing resources indicated by the resource usage parameters are determined as the target processing resources.

[0062] Optionally, in this embodiment, the preset usage can be but is not limited to a pre-set fixed value, or it can be determined based on the amount of processing resources allocated by the target distributed node to other logged-in users, for example: the average amount of processing resources owned by the logged-in users in the target distributed node is used as the preset usage; or, the highest amount of processing resources owned by the logged-in users in the target distributed node is used as the preset usage; or, the lowest amount of processing resources owned by the logged-in users in the target distributed node is used as the preset usage, etc.

[0063] In an exemplary embodiment, the target processing resources can be allocated to the target user in the following manner, but is not limited to: creating a target login image, wherein the target login image includes the environmental resources allowed for the target user to log in to the distributed system; calling the target login image to encapsulate the target processing resources to obtain a target login container, wherein the target login container is configured with a container port of the target login container, and the container port is used to log the target user in to the target login container.

[0064] Optionally, in this embodiment, the target login image includes environmental resources allowed for the target user to log in to the distributed system, which may include but are not limited to: packaged software, libraries, configuration files, and dependencies, etc.

[0065] Optionally, in this embodiment, the target distributed node may be, but is not limited to, deployed with a platform that has the functions of building a container, encapsulating the target processing resources into a target login container, and configuring the target login container's container port in the target login container. Alternatively, the target distributed node may be, but is not limited to, deployed with the functions of building a container, encapsulating the target processing resources into a target login container, and configuring the target login container's container port in the target login container.

[0066] Optionally, in this embodiment, the target user can use the target login container to remotely log into the virtual environment, i.e., the target processing resources allocated to the target user on the target distributed node. The target login container isolates the target processing resources allocated to the target user from the processing resources of other users. In other words, the target user is only allowed to publish processing tasks to the distributed nodes in the distributed system through the target processing resources encapsulated in the target login container. In other words, the method for allocating processing tasks in a distributed system proposed in this application implements the allocation and restriction of processing resources for users on target distributed nodes configured with login functions, the isolation of the user's login environment on the target distributed node, and the customized configuration of the login container based on the actual application scenario of the method.

[0067] In an exemplary embodiment, the target login image can be created in the following manner, but is not limited to: creating an initial login image, wherein the initial login image includes the environmental resources allowed for the target user to log in to the distributed system; adding a cluster service, an authentication service, and a synchronization service to the initial login image to obtain the target login image, wherein the cluster service is used to add a container built by the target login image to the distributed system, the authentication service is used to indicate that the container built by the target login image belongs to the distributed system, and the synchronization service is used to add the login function to the container built by the target login image; storing the target login image in a container image repository, In which, the container image repository is used to store container images of containers that are fully allowed to be deployed in the distributed system; the target login image can be called in, but is not limited to, the following manner to encapsulate the target processing resources to obtain the target login container: calling the image run command to start the target login image in the container image repository, and configuring the number of processor cores and memory space capacity of the target login image through the resource configuration command to obtain the target login container, wherein the target processing resources include the number of processor cores and the memory space capacity, and the resource configuration command includes: a first configuration command and a second configuration command, the first configuration command is dockerrun-d--cpus=a, docker run-d--cpus represents the operation of configuring the number of processor cores in the target login image, and a is the value of the configured number of processor cores, the second configuration command is docker run-dm b, dockerrun-dm represents the operation of configuring the memory space capacity of the target login image, and b is the value of the configured memory space capacity.

[0068] Optionally, in this embodiment, in order to add the target login image constructed by the target login image to the distributed system, it is possible but not limited to deploying cluster services, authentication services and synchronization services in the initial login image when creating the target login image used by the target login image. Taking the distributed system scheduling processing tasks published by users on the distributed system through slurm (a cluster job scheduling software) as an example, the slurmd service is installed and configured in the initial login image as a cluster service, and the slurmd service is used to subsequently add the login container to the cluster in the form of a slurm computing node; the munge service is installed and configured in the initial login image as an authentication service, and the munge service is used to realize authentication between the login container and other nodes in the cluster; the ldap client service is installed and configured in the initial login image as a synchronization service, and the ldap client service is used to synchronize cluster users to the login container.

[0069] Optionally, in this embodiment, the container image repository is used to store and manage container images of containers that are allowed to be deployed in a distributed system. The container image repository can be, but is not limited to, a local image repository of the target distributed node, or a container image repository stored in the cloud.

[0070] In an exemplary embodiment, after the target login container is obtained by configuring the number of processor cores and memory space capacity of the target login image through a resource configuration command, the target login image can be configured in the following manner, but is not limited to: configuring the container connection attributes and container storage attributes of the target login image through a third configuration command, wherein the container connection attributes are used to indicate the calling path of the target login container, and the container storage attributes are used to indicate the storage location of data in the target login container.

[0071] Optionally, in this embodiment, taking the target distributed node as a server host as an example, configuring the container connection properties of the target login image may include, but is not limited to, specifying a mapping of a host port to a container port through the parameter -p[host port]:[container port] in the third configuration command. The host port may be, but is not limited to, a predetermined SSH (Secure Shell) port number or an SSH port number assigned to the server host. It should be noted that the container port is typically the default SSH port number 22.

[0072] Optionally, in this embodiment, taking the target distributed node as a server host as an example, configuring the container storage properties of the target login image may include, but is not limited to: specifying the mapping of the host data path to the data path within the container through the parameter -v[host data path]:[data path within the container] in the third configuration command, setting both the host data path and the data path within the container to the same path to indicate the storage location of the data in the container, and the above-mentioned same path may be the user home directory path pre-assigned to the server host.

[0073] In the technical solution provided in the above step S206, it is possible but not limited to sending a login response to the target user and establishing a connection between the target user and the target login container.

[0074] Optionally, in this embodiment, the distributed system is a slurm cluster, the slurm cluster schedules the processing tasks published by the user on the distributed system through slurm, and the target distributed node is the login node in the slurm cluster as an example, Figure 3 is a schematic diagram of a target processing resource structure of a target user according to an embodiment of the present application, such as Figure 3As shown, the user resource setting module in the slurm cluster can, but is not limited to, detect the login resources to be configured for the target user (i.e., target processing resources), including the number of CPU cores and memory capacity, when the login node receives a login request initiated by the target user, and the connection configuration for allowing the target user to use the login resources (i.e., configuring a mapping relationship for connecting the target login container with the target user), such as: SSH port number, user home directory, login node name, and login container internal IP (Internet Protocol) address.

[0075] Optionally, in this embodiment, the distributed system is a slurm cluster, and the slurm cluster schedules the processing tasks published by the user on the distributed system through slurm. Figure 4 is a schematic diagram of a configuration structure of a target distributed node according to an embodiment of the present application, such as Figure 4 As shown, first, in order to further simplify the process of slurm computing nodes obtaining configuration files, reduce implementation difficulty, and enable rapid deployment as slurm management nodes in a slurm cluster (i.e., distributed nodes configured with management functions in a distributed system), for example, the value of the parameter SlurmctldParameters in the slurm configuration file slurm.Conf is set to enable_configless. The cluster configuration module in the slurm cluster can, but is not limited to, configure a slurm computing node as a slurm login node (i.e., the target distributed node) in the following ways:

[0076] Create a login container partition in the slurm cluster to add the login container as a computing node to the partition. For example, use the scontrol command to create a partition and specify the partition name. Set the login container partition to hidden mode in the slurm cluster. After setting it to hidden mode, ordinary users cannot view the partition information. For example, use the sControl command to set the login container partition parameter Hidden to yes. Set the login container partition permissions in the distributed system so that ordinary users cannot submit jobs (i.e., process tasks). With this setting, ordinary user jobs will not be assigned to the partition. For example, use the scontrol command to set the login container partition parameter RootOnly to yes.

[0077] By setting the login container partition to a hidden state, it can only be viewed by administrators and not by ordinary users, thus ensuring the privacy of information. Through slurm's configuration-free mode, the deployment of slurm clusters and the addition of nodes are simplified, avoiding the tedious operation of synchronizing computing node configuration files caused by changes to slurm configuration files. The partition creation method through the scontrol command makes it more efficient to change partition information and avoids the tedious operation of frequent changes to configuration files.

[0078] Optionally, in this embodiment, the distributed system is a slurm cluster, the slurm cluster schedules the processing tasks published by the user on the distributed system through slurm, and the target login container is configured in the target distributed node through docker as an example, Figure 5 This is a schematic diagram of the structure of a target login container according to an embodiment of the present application. Figure 5 As shown, the container deployment module can, but is not limited to, configure a login container for a user on a slurm login node in a slurm cluster in the following ways:

[0079] First, install Docker on the target distributed node and configure the Docker local image repository (i.e., container image repository) on the target distributed node.

[0080] Create a mirror file (i.e., target login mirror) on the target distributed node. The requirements for mirror file creation include: deploying slurmd in the mirror, configuring the slurmd service startup option "--conf-server slurm management node IP address: slumctld service port number", and configuring the slurmd service to start automatically at boot; deploying the Idap client in the mirror, configuring the Idap server address to be the unified Idap server address of the cluster, and configuring the Idap client service to start automatically; deploying munge in the mirror, configuring it to have the same key file as other physical nodes in the cluster, and configuring the munge service to start automatically.

[0081] Push the prepared image file to the local Docker image warehouse;

[0082] Use the image file as follows Figure 3 The target processing resource shown starts the user login container (ie, the target login container):

[0083] Use the command docker run (image run command) to start the Docker container (i.e., the target login container), and configure the resources for the number of CPU cores (i.e., the number of processor cores), memory size (i.e., memory space capacity), SSH login port, user home directory, and login node name using the following commands:

[0084] The number of CPU cores used by the docker container is specified by the parameter "--cpu". The specific command is: dockerrun -d --cpus=a, which is set to the value a of the number of processor cores in the target processing resource selected from the processing resources indicated by the resource usage parameter based on the parameter relationship between the resource usage parameter and the historical usage parameter.

[0085] The parameter "-m" specifies the memory size allowed for the docker container. The specific command is: dockerrun -dm b, which is set to the value b of the memory space capacity in the target processing resource selected from the processing resources indicated by the resource usage parameter based on the parameter relationship between the resource usage parameter and the historical usage parameter.

[0086] To connect the Docker container to the server host (i.e., the target user), you can further complete the following configurations, but are not limited to:

[0087] Use the parameter "-p [host port]: [container port]" to specify the mapping of the host port to the container port, and set the host port to Figure 3 The SSH port number shown,

[0088] Use the parameter "-v [host data path]: [container data path]" to specify the mapping from "host data path" to "container data path". Set both "host data path" and "container data path" to the following: Figure 3 The "user home directory" path shown;

[0089] Use the parameter "--hostname" to specify the host name used by the docker container and set it as follows Figure 3 The "login node name" shown in the figure uses the parameter "--ip" to specify the IP address used by the docker container and set it as follows Figure 3 As shown in the "Login container internal IP address", specify the docker container name through the parameter "--name", and set it as follows Figure 3 The "User Name" shown.

[0090] In the user login container, use the "slurmd -C" command to obtain the container configuration information and add it to the cluster management node cluster configuration slurm.conf. In the cluster, add the user login container to the cluster as a computing node. Use the scontrol update command to specify the nodes parameter, which points to the login container hostname. Add the corresponding container as follows: Figure 4 In the "Login Container Partition" shown.

[0091] In a distributed system, distributed nodes with management functions use the sinfo command to view user login container status and CPU and memory resource information.

[0092] Users log in to the node external IP and Figure 3 Log in to the container using the SSH port shown in the figure and submit the processing task in the container.

[0093] In the method for allocating processing tasks in a distributed system proposed in the present application, using a container as the login environment for users in the distributed system can better control user access rights. When the login container exists, the user can access the cluster normally, otherwise the user cannot access it; and the isolation of the user login environment is achieved through the container, and the CPU and memory resources used by the user login environment are restricted through the container, which is more flexible and the login environment is independent. Multiple image files can be customized for different applications for users to choose from, to achieve customization of the login environment for application scenarios, and increase the flexibility of cluster use; by presetting the user login SSH port, the user login port is fixed, avoiding usage problems caused by dynamic changes in the port.

[0094] In the technical solution provided in the above step S208, it is possible but not limited to storing the processing tasks sent by the user through the memory space in the target processing resource, and then pre-processing the processing tasks through the processor core in the target processing resource and distributing them to the corresponding distributed nodes.

[0095] In an exemplary embodiment, the target processing task published by the target user can be received through the target processing resource in the following manner, but is not limited to: receiving a connection request sent by the target user to the target processing resource, wherein the connection request is used to request to establish a communication connection with the target processing resource; establishing a target connection between the target user and the target processing resource in response to the connection request; and receiving the target processing task published by the target user to the target processing resource through the target connection.

[0096] In one exemplary embodiment, an example of a distributed system is provided. Figure 6 is a schematic diagram of a distributed node in a distributed system according to an embodiment of the present application, such as Figure 6 As shown, the distributed nodes in the distributed system include: a target distributed node configured with a login function, a control distributed node configured with a control function, and a computing distributed node for executing processing tasks. The target distributed node can be, but is not limited to, a computing node with a login function added. The specific description is as follows:

[0097] The target user joins the distributed system through the target login container. The target login container serves as the operating system for the user to log in. The target processing resources allocated to it include the number of CPU cores and memory size. In this solution, the processing resources allocated to the target login container are limited in amount. In addition to the target processing resources allocated to the target login container, in order to allow the target user to use the target login container, it is also necessary to configure the SSH login port, user home directory, target distributed node name and login container internal IP address for the target user and target login container. Among them, the SSH login port is used to set the internal SSH port of the user login container to be mapped to the port of the target distributed node, allowing the user to log in to the container directly through the SSH protocol through the external IP of the target distributed node; the user home directory is used to map the user's home directory on the login node to the user login container, keeping the user's home directory path inside the container and the login node physical host consistent; the login node name is used to specify the host name for the container logged in by the user; the login container internal IP address is used to specify a fixed IP address for each user's login container.

[0098] Perform relevant configurations on the distributed nodes and target distributed nodes in the distributed system, including the following: enable the configuration-free mode for the management distributed nodes to further simplify the process of obtaining configuration files for the computing distributed nodes, reduce the implementation difficulty, and achieve rapid deployment; create a separate login container partition in the distributed system, specifically for adding the login container as a computing node to the partition (i.e., the target distributed node); set the login container partition in the distributed system to hidden mode. After setting it to hidden mode, ordinary users cannot view the partition information; set the login container partition in the distributed system to prevent ordinary users from submitting jobs. Through this setting, ordinary user jobs will not be assigned to the partition.

[0099] Users log in to the distributed system through a login container. Therefore, the Docker container engine must be deployed on the target distributed node of the distributed system to create and start the login container. A local Docker image repository must also be deployed to store and manage the different container image files created.

[0100] When creating the image file used by the login container, you can configure the following in the image, but are not limited to configuring it, to add the login container to the cluster as a computing distributed node: install and configure the slurmd service, which is mainly used to add the login container to the distributed system as a computing node; install and configure the munge service, which is mainly used to implement user authentication between the login container node and other nodes in the cluster; install and configure the ldap client service, which is mainly used to synchronize cluster users to the login container node;

[0101] To start a login container, you need to select the corresponding image file to create a login container based on the login information and resource restrictions pre-configured for the user, including CPU, memory, SSH login port, user home directory, and login host name. The configuration information required for starting a login container includes the following:

[0102] Based on the preset CPU resources, the number of CPU cores of the user logging into the container is limited; based on the preset memory resources, the memory size of the user logging into the container is limited; based on the preset SSH port number information, the SSH port inside the container is mapped to the preset SSH port number; in a high-performance cluster, users can use the SSH protocol for access, and the login container is a virtual environment for users to log in to the cluster, so the SSH port also needs to be opened to the outside world. This application proposes a method of mapping the default SSH port of the container to the login server to achieve SSH access to the port corresponding to the login server to access the environment when the user logs in.

[0103] Based on the preset user home directory information, map the user home directory from the target distributed node to the same directory in the container; based on the preset login node name information, specify the host name in the container.

[0104] The steps of adding a user login container to a distributed system as a computing distributed node include: obtaining the correspondence between the IP address and host name of the user login container; adding all user login containers to the distributed system as computing distributed nodes; adding containers created in the distributed system to the created login container partition; and a cluster management user using the sinfo command to view the status of each user login container in the login partition and resource configuration information.

[0105] Containers are a lightweight virtualization technology at the operating system level that runs on a host machine in a special form. The same host machine can run multiple containers, and the CPU, memory and other resources used by the containers can be restricted or allocated. The operating resources and operating system environments of different containers are isolated from each other. Starting a container requires an image file, which contains the basic environment required by the container and the custom installed software operating environment. Compared with the slurm architecture in related technologies, this application chooses to deploy docker containers on the slurm login node and use the container as the operating system carrier for user login. At the same time, the container exists in the form of a computing node in the slurm cluster. As a slurm computing node, the container must include the services required by the slurm computing node, including slurmd, munge, and ldap client. In high-performance clusters, users usually access nodes through the SSH protocol. The login container serves as the virtual environment for users to log in to the cluster, so it also needs to open the SSH port to the outside world. This is achieved in this application by mapping the default SSH port of the container to the login server. When the user logs in, he can access the environment by SSH accessing the corresponding port of the login server.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the method described in each embodiment of the present application.

[0107] In this embodiment, a device for allocating processing tasks in a distributed system is also provided. The distributed system includes multiple distributed nodes, and the distributed nodes are used to execute received processing tasks. The device is applied to a target distributed node configured with a login function among the multiple distributed nodes. The login function is used to receive and allocate processing tasks posted by users on the distributed system to the multiple distributed nodes. The device is used to implement the above-mentioned embodiments and preferred implementation methods, and those that have been explained will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceived.

[0108] Figure 7is a structural block diagram of a distribution device for processing tasks in a distributed system according to an embodiment of the present application, such as Figure 7 As shown, the device includes:

[0109] A receiving module 72 is configured to receive a login request initiated by a target user, wherein the login request is used to request to log into the distributed system;

[0110] an allocation module 74, configured to respond to the login request and allocate target processing resources on the target distributed node to the target user based on resource usage information of the target distributed node and historical usage information of the target user, wherein the resource usage information indicates usage of the processing resources in the target distributed node, and the historical usage information indicates usage of the processing resources by the target user over a historical period of time;

[0111] a sending module 76 configured to send a login response to the target user, wherein the login response is used to indicate that the target user is allowed to log in to the distributed system and to allow the target user to publish a processing task on the target processing resource;

[0112] The processing module 78 is configured to receive the target processing task issued by the target user through the target processing resource, and distribute the target processing task to the multiple distributed nodes.

[0113] The device receives a login request initiated by a target user, wherein the login request is used to request to log into a distributed system; responds to the login request, allocates target processing resources on a target distributed node to the target user based on resource usage information of the target distributed node and historical usage information of the target user, wherein the resource usage information is used to indicate the usage of processing resources in the target distributed node, and the historical usage information is used to indicate the usage of processing resources by the target user in historical time; sends a login response to the target user, wherein the login response is used to indicate that the target user is allowed to log into the distributed system and that the target user is allowed to publish processing tasks on the target processing resources; receives the target processing tasks published by the target user through the target processing resources, and distributes the target processing tasks to multiple distributed nodes. Since each target user requesting to log into the distributed system is allocated a target processing resource that is allowed to be used by the target user on the target distributed node, it avoids the situation where multiple users log into the distributed system and the occurrence of processing resource preemption in the target distributed node, thereby solving the problem of low efficiency in the allocation of processing tasks in the distributed system and achieving the effect of improving the efficiency of the allocation of processing tasks in the distributed system.

[0114] In an exemplary embodiment, the allocation module includes:

[0115] a detection unit, configured to detect a resource usage parameter of the target distributed node and a historical usage parameter of the target user, wherein the resource usage information includes the resource usage parameter, the historical usage information includes the historical usage parameter, the resource usage parameter is used to indicate an amount of resources allowed to be allocated on the target distributed node, and the historical usage parameter is used to indicate an amount of usage of the allocated processing resources by the target user during the historical time;

[0116] a screening unit, configured to screen the target processing resource from the processing resources indicated by the resource usage parameter according to a parameter relationship between the resource usage parameter and the historical usage parameter;

[0117] An allocating unit is configured to allocate the target processing resource to the target user.

[0118] In an exemplary embodiment, the screening unit is further used to: when the parameter relationship is used to indicate that the resource usage parameter is greater than or equal to the historical usage parameter, determine the processing resource with the usage amount indicated by the historical usage parameter in the processing resources indicated by the resource usage parameter as the target processing resource; when the parameter relationship is used to indicate that the resource usage parameter is less than the historical usage parameter, determine the processing resource with a preset usage amount in the processing resources indicated by the resource usage parameter as the target processing resource.

[0119] In an exemplary embodiment, the screening unit is further used to: create a target login image, wherein the target login image includes the environmental resources allowed for the target user to log in to the distributed system; call the target login image to encapsulate the target processing resources to obtain a target login container, wherein the target login container is configured with a container port of the target login container, and the container port is used to log the target user in to the target login container.

[0120] In an exemplary embodiment, the screening unit is also used to: create an initial login image, wherein the initial login image includes the environmental resources allowed for the target user to log in to the distributed system; add a cluster service, an authentication service and a synchronization service to the initial login image to obtain the target login image, wherein the cluster service is used to add the container built by the target login image to the distributed system, the authentication service is used to indicate that the container built by the target login image belongs to the distributed system, and the synchronization service is used to add the login function to the container built by the target login image; store the target login image in a container image repository, wherein the container image repository is used to store container images of all containers allowed to be deployed in the distributed system.

[0121] In an exemplary embodiment, the screening unit is also used to: call the image run command to start the target login image in the container image repository, and configure the number of processor cores and memory space capacity of the target login image through the resource configuration command to obtain the target login container, wherein the target processing resources include the number of processor cores and the memory space capacity, and the resource configuration command includes: a first configuration command and a second configuration command, the first configuration command is docker run-d--cpus=a, docker run-d--cpus represents the operation of configuring the number of processor cores in the target login image, a is the value of the configured number of processor cores, the second configuration command is docker run-dm b, docker run-dm represents the operation of configuring the memory space capacity of the target login image, and b is the value of the configured memory space capacity.

[0122] In an exemplary embodiment, after configuring the number of processor cores and memory space capacity of the target login image through a resource configuration command to obtain the target login container, the apparatus further includes:

[0123] A configuration module is used to configure the container connection attributes and container storage attributes of the target login image through a third configuration command, wherein the container connection attributes are used to indicate the calling path of the target login container, and the container storage attributes are used to indicate the storage location of data in the target login container.

[0124] In an exemplary embodiment, the receiving module includes:

[0125] a first receiving unit, configured to receive a connection request sent by the target user to the target processing resource, wherein the connection request is used to request to establish a communication connection with the target processing resource;

[0126] an establishing unit, configured to establish a target connection between the target user and the target processing resource in response to the connection request;

[0127] The second receiving unit is configured to receive the target processing task published by the target user to the target processing resource through the target connection.

[0128] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0129] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above method embodiments when run.

[0130] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0131] An embodiment of the present application further provides an electronic device, Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present application, such as Figure 8 As shown, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0132] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0133] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0134] An embodiment of the present application further provides another computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above method embodiments are implemented.

[0135] An embodiment of the present application further provides a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above method embodiments.

[0136] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0137] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0138] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for allocating processing tasks in a distributed system, characterized in that: A distributed system includes multiple distributed nodes, each of which is used to execute received processing tasks. The method is applied to a target distributed node configured with a login function among the multiple distributed nodes, the login function being used to receive and assign processing tasks published by a user on the distributed system to the multiple distributed nodes. The method includes: receiving a login request initiated by a target user, wherein the login request is used to request to log in to the distributed system; responding to the login request, allocating target processing resources on the target distributed node to the target user based on resource usage information of the target distributed node and historical usage information of the target user, wherein the resource usage information is used to indicate usage of processing resources in the target distributed node, and the historical usage information is used to indicate usage of processing resources by the target user in historical time; Sending a login response to the target user, wherein the login response is used to indicate that the target user is allowed to log in to the distributed system and to publish a processing task on the target processing resource; The target processing task issued by the target user is received through the target processing resource, and the target processing task is distributed to the multiple distributed nodes.

2. The method according to claim 1, characterized in that The allocating target processing resources on the target distributed node to the target user according to the resource usage information of the target distributed node and the historical usage information of the target user includes: detecting resource usage parameters of the target distributed node and historical usage parameters of the target user, wherein the resource usage information includes the resource usage parameters, the historical usage information includes historical usage parameters, the resource usage parameters are used to indicate an amount of resources allowed to be allocated on the target distributed node, and the historical usage parameters are used to indicate an amount of usage of the allocated processing resources by the target user during the historical time; selecting the target processing resource from the processing resources indicated by the resource usage parameter according to a parameter relationship between the resource usage parameter and the historical usage parameter; Allocate the target processing resource to the target user.

3. The method according to claim 2, characterized in that The selecting the target processing resource from the processing resources indicated by the resource usage parameter according to the parameter relationship between the resource usage parameter and the historical usage parameter includes: In a case where the parameter relationship indicates that the resource usage parameter is greater than or equal to the historical usage parameter, determining, among the processing resources indicated by the resource usage parameter, a processing resource having a usage amount indicated by the historical usage parameter as the target processing resource; In a case where the parameter relationship indicates that the resource usage parameter is less than the historical usage parameter, a processing resource with a preset usage amount among the processing resources indicated by the resource usage parameter is determined as the target processing resource.

4. The method according to claim 3, characterized in that The allocating the target processing resource to the target user includes: Creating a target login image, wherein the target login image includes environment resources that the target user is allowed to use when logging into the distributed system; The target login image is called to encapsulate the target processing resource to obtain a target login container, wherein a container port of the target login container is configured in the target login container, and the container port is used to log the target user into the target login container.

5. The method according to claim 4, characterized in that The step of creating a target login image includes: Creating an initial login image, wherein the initial login image includes environment resources allowed for the target user to log in to the distributed system; Adding a cluster service, an authentication service, and a synchronization service to the initial login image to obtain the target login image, wherein the cluster service is used to add a container built by the target login image to the distributed system, the authentication service is used to indicate that the container built by the target login image belongs to the distributed system, and the synchronization service is used to add the login function to the container built by the target login image; Storing the target login image in a container image repository, wherein the container image repository is used to store container images of all containers that are allowed to be deployed in the distributed system; The calling of the target login image to encapsulate the target processing resource to obtain a target login container includes: Call the image run command to start the target login image in the container image repository, and configure the number of processor cores and memory space capacity of the target login image through the resource configuration command to obtain the target login container, wherein the target processing resources include the number of processor cores and the memory space capacity, and the resource configuration command includes: a first configuration command and a second configuration command, the first configuration command is docker run-d--cpus=a, docker run-d--cpus represents the operation of configuring the number of processor cores in the target login image, and a is the value of the configured number of processor cores, the second configuration command is docker run-dm b, docker run-dm represents the operation of configuring the memory space capacity of the target login image, and b is the value of the configured memory space capacity.

6. The method according to claim 5, characterized in that After configuring the number of processor cores and memory space capacity of the target login image through the resource configuration command to obtain the target login container, the method further includes: The container connection attributes and container storage attributes of the target login image are configured through a third configuration command, wherein the container connection attributes are used to indicate the calling path of the target login container, and the container storage attributes are used to indicate the storage location of data in the target login container.

7. The method according to claim 1, characterized in that The receiving, through the target processing resource, the target processing task issued by the target user, includes: receiving a connection request sent by the target user to the target processing resource, wherein the connection request is used to request to establish a communication connection with the target processing resource; In response to the connection request, establishing a target connection between the target user and the target processing resource; The target processing task published by the target user to the target processing resource through the target connection is received.

8. A device for allocating processing tasks in a distributed system, characterized in that: A distributed system includes a plurality of distributed nodes, each of which is configured to execute received processing tasks. The apparatus is applied to a target distributed node configured with a login function among the plurality of distributed nodes, the login function being configured to receive and distribute processing tasks published by a user on the distributed system to the plurality of distributed nodes. The apparatus includes: A receiving module, configured to receive a login request initiated by a target user, wherein the login request is used to request to log into the distributed system; an allocation module, configured to respond to the login request and allocate target processing resources on the target distributed node to the target user based on resource usage information of the target distributed node and historical usage information of the target user, wherein the resource usage information is used to indicate usage of the processing resources in the target distributed node, and the historical usage information is used to indicate usage of the processing resources by the target user in historical time; a sending module, configured to send a login response to the target user, wherein the login response is used to indicate that the target user is allowed to log in to the distributed system and to allow the target user to publish a processing task on the target processing resource; The processing module is configured to receive the target processing tasks issued by the target user through the target processing resources, and distribute the target processing tasks to the multiple distributed nodes.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images