A task processing method, device and storage medium

Abstract

The present application discloses a task processing method, device and storage medium, the method comprising: determining the target resource file required by the target task container, the target task container being scheduled to run on at least one physical node; in response to the target resource file belonging to the hardware sensitive type, generating the first exclusive file identifier for each physical node for the target resource file respectively, to serve as the first file identifier of the target resource file respectively, the hardware sensitive type characterizing that the target resource files required by the physical nodes of different hardware conditions are different; finding the target resource files that match each first file identifier from the resource file pool and downloading them, wherein the target resource files are used when the target task container is run on the physical node. The present application can identify that the target resource file required by the target task container is a resource file of the hardware sensitive type, download the matching target resource file, and improve the adaptability of the resource file and the success rate of the target task container operation.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a task processing method, device and storage medium. Background Art

[0002] In the current cloud computing and big data processing fields, hybrid clusters have become an important means to improve resource utilization efficiency and flexibility. There are many types of GPUs on the market, each with unique hardware architecture and driver requirements. In order to accommodate these differences, hybrid clusters usually use virtualization technology or containerization technology to implement resource scheduling and management. In this case, users often do not know the specific execution environment of the task, and cannot prepare or specify the correct resource files in advance, resulting in the failure of some tasks with hardware-sensitive resource files to run in hybrid clusters. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a task processing method, device and storage medium, which can automatically download matching resource files and improve the success rate of task operation.

[0004] To solve the above technical problems, a technical solution adopted in the present application is: providing a task processing method, the method comprising: determining a target resource file required by a target task container, wherein the target task container is scheduled to run on at least one physical node; in response to the target resource file belonging to a hardware-sensitive type, generating a first exclusive file identifier for each physical node for the target resource file, respectively, to serve as the first file identifier of the target resource file, wherein the hardware-sensitive type represents that the target resource files required by physical nodes with different hardware conditions are different; finding out the target resource files that match each first file identifier from a resource file pool; and downloading the target resource files to the storage space of the corresponding physical node, respectively, wherein the target resource files are used when the physical node runs the target task container.

[0005] Among them, a first exclusive file identifier for each physical node is generated for the target resource file, including: obtaining at least one hardware information of each physical node; for each physical node, obtaining a hardware identifier corresponding to each hardware information of the physical node, and a universal file identifier of the target resource file; combining the hardware identifier corresponding to each hardware information and the universal file identifier to obtain a first exclusive file identifier for the physical node.

[0006] Among them, the hardware information of the physical node includes at least one of the central processing unit architecture and the graphics processing unit type; and / or, the hardware identifier and the general file identifier corresponding to each hardware information are combined to obtain the first exclusive file identifier for the physical node, including: obtaining a preset identifier representing the hardware sensitive type; combining the preset identifier, the hardware identifier corresponding to each hardware information and the general file identifier to obtain the first exclusive file identifier for the physical node.

[0007] Among them, determining the target resource file required by the target task container includes: obtaining resource file information of the target task container submitted by the user, the resource file information including the second file identifier of each target resource file required by the target task container, the second file identifier in the resource file information including a universal file identifier and / or a second exclusive file identifier, each universal file identifier represents a target resource file used by physical nodes of all hardware conditions, each second exclusive file identifier represents a target resource file used by a physical node of a hardware condition, and the second exclusive file identifier includes a preset identifier representing a hardware sensitive type and a universal file identifier representing the target resource file; extracting the universal file identifier from each second exclusive file identifier, wherein the extracted universal file identifier is used to generate a first exclusive file identifier for the physical node; determining the target resource file required by the target task container based on each universal file identifier, and determining the target resource file corresponding to the universal file identifier extracted from the second exclusive file identifier as a target resource file belonging to the hardware sensitive type.

[0008] Among them, extracting the universal file identifier from each second exclusive file identifier includes: filtering out the second file identifier containing a preset identifier from the resource file information to find out the second exclusive file identifier; removing the preset identifier and the hardware identifier related to the corresponding physical node in the second exclusive file identifier to obtain the universal file identifier.

[0009] Among them, before searching for target resource files that match each first file identifier from the resource file pool, it also includes: in response to the target resource file not belonging to the hardware-sensitive type, obtaining a universal file identifier of the target resource file as the first file identifier of the target resource file, the universal file identifier corresponding to the physical node of all hardware conditions; and / or, searching for target resource files that match each first file identifier from the resource file pool, including: obtaining a third file identifier of each resource file from the resource file pool; searching for a third file identifier that matches the first file identifier from the third file identifier of each resource file; and using the resource file corresponding to the found third file identifier as the target resource file.

[0010] Among them, the first file identifier includes the universal file identifier of the target resource file; after downloading the target resource files to the storage space of the corresponding physical nodes respectively, the method also includes: generating a soft link named with the universal file identifier of the target resource file for the physical node, wherein the physical node can access the target resource file through the soft connection.

[0011] The method further includes: obtaining the running hardware information specified by the user for the target task container; scheduling the target task container to at least one physical node based on the running hardware information, wherein the hardware information of at least one physical node matches the running hardware information.

[0012] Among them, obtaining the running hardware information specified by the user for the target task container includes: receiving the target environment parameters set by the user, and the values ​​of the target environment parameters are used to characterize the running hardware information; based on the running hardware information, scheduling the target task container to at least one physical node, including: rendering the target environment parameters in the form of annotations to the configuration file corresponding to the target task container; based on the annotations in the configuration file corresponding to the target task container, scheduling the target task container to at least one physical node.

[0013] Among them, based on the annotations in the configuration file corresponding to the target task container, the target task container is scheduled to at least one physical node, including: identifying the annotations in the configuration file corresponding to the target task container; converting the identified annotations into physical node labels, and scheduling the target task container to the physical node matching the physical node label; and / or, the step of receiving the target environment parameters set by the user, and the step of rendering the target environment parameters in the form of annotations to the configuration file corresponding to the target task container are performed by the resource management platform, and the resource management platform is also used to send the configuration file to the container management platform; based on the annotations in the configuration file corresponding to the target task container, the step of scheduling the target task container to at least one physical node is performed by the container management platform.

[0014] To solve the above technical problems, another technical solution adopted in the present application is to provide a device, which includes a processor and a memory connected to the processor, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement any of the task processing methods mentioned above.

[0015] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium, the storage medium has program instructions, and the program instructions are executed to implement any of the above-mentioned task processing methods.

[0016] The beneficial effects of the present application are: different from the prior art, the present application can identify that the target resource file required by the target task container is a hardware-sensitive type of resource file, generate a first exclusive file identifier for physical nodes with different hardware conditions, and search and download matching target resource files in the resource file pool according to the first exclusive file identifier, thereby improving the adaptability of the resource file and the success rate of the target task container operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of an embodiment of a task processing method of the present application.

[0018] Figure 2 This is a flowchart of confirming a target resource file according to an embodiment of a task processing method of the present application.

[0019] Figure 3 This is a flowchart of generating a first exclusive file identifier according to an embodiment of a task processing method of the present application.

[0020] Figure 4 This is a target task container scheduling flow chart of an embodiment of a task processing method of the present application.

[0021] Figure 5 It is a schematic diagram of the framework of an embodiment of a task processing device of the present application.

[0022] Figure 6 It is a schematic diagram of the framework of an embodiment of a device of the present application.

[0023] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0025] It should be understood that the reference to "embodiment" in this application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0026] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0027] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of a task processing method of the present application.

[0028] The method comprises the following steps:

[0029] Step S11: determining a target resource file required by a target task container, wherein the target task container is scheduled to run on at least one physical node;

[0030] The target task container can be an independent portable container unit in the target task, and the target task can be, but is not limited to, a big data processing task, a machine learning task, a deep learning task, etc. Containerizing tasks can schedule tasks efficiently and flexibly. The physical node can be provided with hardware for operation, including but not limited to a central processing unit (CPU), a graphics processing unit (GPU), etc. Different physical nodes can be configured with different central processing units and different graphics processing units. It is understandable that the architectures of different central processing units may be different, and the architectures of image processing of different models may be different. When the container runs on a physical node, it may use common resource files, and it may also use resource files closely related to the hardware architecture of the physical node, such as data, library files, configuration files, etc. of some specific architectures. Therefore, in order to ensure that the target task container runs correctly and efficiently on the physical node, it is necessary to provide the correct target resource files for its operation.

[0031] In some embodiments, see Figure 2 , Figure 2 This is a flowchart of confirming a target resource file in an embodiment of a task processing method of the present application, confirming the target resource file required by the target task container includes the following subdivision steps:

[0032] Step S211: Acquire resource file information of the target task container submitted by the user, where the resource file information includes the second file identifier of each target resource file required by the target task container.

[0033] The resource file information of the target task container submitted by the user includes at least all resource files of all hardware architectures that may be used when the target task container needs to run. For example, the target task container needs to use resource file A to run. Resource file A has file A' that matches the first hardware architecture and file A'' that matches the second hardware architecture. Therefore, the user can declare that at least files A' and A'' are needed when submitting the resource file information of the target task container. In some application scenarios, the second file identifier in the resource file information includes a universal file identifier. Each universal file identifier represents a target resource file that can be used by physical nodes of all hardware conditions. These resource files can be used directly by different types of physical architectures without modification or recompilation. Their format or encoding does not depend on a specific hardware architecture, so they can be used across platforms. In the following text, such files are classified as non-hardware sensitive types. For example, resource files that are not hardware sensitive types may be some standard configuration files, library files, etc. It can be understood that the universal file identifier is used to refer to the file type and purpose of the resource file. Specifically, the universal file identifier may be a common name in the industry corresponding to the file. For example, the universal file identifier of the compressed package file used in the test process may be text.zip.

[0034] In some application scenarios, the second file identifier in the resource file information includes a second exclusive file identifier, each second exclusive file identifier includes a target resource file used by a physical node of a hardware situation, and the second exclusive file identifier includes a preset identifier representing a hardware sensitive type and a general file identifier representing a target resource file. These resource files need to be specifically adapted and prepared according to the hardware architecture of the running task container. In the following text, such files are classified into hardware sensitive types. For example, resource files belonging to hardware sensitive types can be drivers or library files of some specific hardware architectures. The preset identifier of the hardware sensitive type is used to indicate that the resource file belongs to a hardware sensitive type resource file. The preset identifier can be a combination of several characters. The preset identifier can be used as a prefix of the resource file name or set at a certain predetermined position of the resource file name, so that the preset identifier can be located when the resource file is subsequently determined to be a sensitive type. For example, the preset identifier is the prefix "ARCH-" of the resource file name, a resource file is a hardware sensitive type resource file, and the general file identifier of the resource file is "cluster.zip", so the second exclusive file identifier of the resource file can be a file name including the "ARCH-" prefix and "cluster.zip".

[0035] In some application scenarios, the second exclusive file identifier also includes a hardware identifier related to the corresponding physical node. The hardware identifier related to the physical node can be a character used to refer to the corresponding hardware type or hardware architecture type, such as a central processing unit architecture, a graphics processor model, etc. For example, a resource file is a hardware-sensitive resource file. The central processing unit architecture to which the resource file belongs is "amd64", and the graphics processor model is "T4". The universal file identifier of the resource file is "cluster.zip", so the second exclusive file identifier of the resource file can be "ARCH-amd64-T4-cluster.zip". It can be understood that when the target task container needs to use resource file A, resource file A includes file A' that matches the first hardware architecture and file A'' that matches the second hardware architecture. The preset identifiers representing the hardware-sensitive type in the second exclusive file identifiers of file A' and file A'' are different, while the universal file identifiers representing the target resource file are the same.

[0036] Step S212: extract the universal file identifier from each second exclusive file identifier, wherein the extracted universal file identifier is used to generate the first exclusive file identifier for the physical node. Determine the target resource file required by the target task container based on each universal file identifier, and extract the target resource file corresponding to the universal file identifier from the second exclusive file identifier as a target resource file of the hardware sensitive type.

[0037] A resource file required for the target task container to run, the resource file corresponds to different target resource files according to the different hardware architectures it matches, and the universal file identifiers of the different target resource files corresponding to the resource file are the same, so the universal file identifier can be extracted and combined with the physical node where the current target task container runs to form a target resource file that matches the physical node. In some embodiments, the universal file identifier is extracted from each second exclusive file identifier, including: filtering out the second file identifier containing a preset identifier from the resource file information to find out the second exclusive file identifier. The resource file corresponding to the second exclusive file identifier is determined as a target resource file of a hardware-sensitive type, and the preset identifier in the second exclusive file identifier and the hardware identifier related to the corresponding physical node are removed to obtain a universal file identifier. In some specific embodiments, the resource file information includes the file name of the resource file, and the preset identifier is a preset field in the resource file name as a specific field. The second file identifier containing the preset identifier can be filtered out from the resource file information by traversing the specific fields of each file name of the resource file information, filtering out the file name of the resource file whose specific field matches the preset identifier, deleting the specific field corresponding to the preset identifier in the file name and the field corresponding to the hardware identifier related to the corresponding physical node, that is, deleting the fields other than the field corresponding to the universal file identifier in the file name to obtain the universal file identifier, which is used to generate the first exclusive file identifier of the target resource file required to match the physical node corresponding to the target task container. For example, the second file identifier in the resource file information of the target task container submitted by the user includes: "ARCH-amd64-T4.cluster.zip", "ARCH-aarch64-Atlas.cluster.zip", "test.zip". The prefix "ARCH-" of the second file identifier is preset, and "amd64-T4" and "aarch64-Atlas" are both hardware identifiers. By matching the prefix of the second file identifier in each resource file information, the resource file whose prefix of the second file identifier contains "ARCH-" is determined as a hardware-sensitive resource file, for example, "ARCH-amd64-T4.cluster.zip" and "ARCH-aarch64-Atlas.cluster.zip". After deleting the preset identifier and hardware identifier in these two second file identifiers, the universal file identifier "cluster.zip" is obtained. The universal file identifier "cluster.zip" is used to generate the first exclusive file identifier of the target resource file required to match the physical node corresponding to the target task container.

[0038] Step S12: In response to the target resource file being of a hardware-sensitive type, a first exclusive file identifier for each physical node is generated for the target resource file to serve as the first file identifier of the target resource file, wherein the hardware-sensitive type indicates that the target resource files required for physical nodes with different hardware conditions are different.

[0039] The target resource file required by the target task container is a resource file that needs to be specifically adapted and prepared according to the hardware architecture of the target task container, and the target resource file belongs to the hardware sensitive type. It is understandable that there are differences in the processor instruction set and other aspects of different hardware architectures, and the binary of the target resource files matching different hardware architectures may be different. When running the target task container, it is necessary to generate the first exclusive file identifier of the target resource file as the first file identifier based on the physical node where the target task container is currently running, so as to match the correct target resource file for the target task container to use.

[0040] In some embodiments, see Figure 3 , Figure 3 This is a flowchart of generating a first exclusive file identifier according to an embodiment of a task processing method of the present application, and generates a first exclusive file identifier for each physical node for a target resource file, including the following subdivision steps:

[0041] Step S321: Obtain at least one piece of hardware information of each physical node.

[0042] The physical node may be provided with hardware capable of running the task container, and the hardware may include at least one of a central processing unit and a graphics processing unit, and the hardware information of the physical node includes at least one of a central processing unit architecture and a graphics processing unit type. The hardware in some physical nodes includes a central processing unit, and the hardware information of the physical node includes a central processing unit architecture, for example, an x86 architecture or an ARM architecture. The central processing unit in some physical nodes is also equipped with a graphics processing unit, and the hardware information of the physical node includes a central processing unit architecture and a graphics processing unit type. For example, a machine with an X86 architecture is equipped with NVIDIA's T4 graphics processor, and the hardware information of the physical node includes information on the X86 architecture and model information on the T4 graphics processor.

[0043] Step S322: For each physical node, obtain the hardware identifiers corresponding to the hardware information of the physical node and the general file identifier of the target resource file.

[0044] The hardware identifier is used to characterize the hardware information of the physical node. The hardware identifier can be a character segment that characterizes the hardware information. The hardware information can be at least one of a CPU architecture or a GPU type. For example, a physical node includes a CPU architecture of amd64, and the hardware identifier of the physical node can be the character segment "amd64". A physical node includes a CPU architecture of aarch64 and a GPU architecture of Atlas, and the hardware identifier of the physical node can be the character segment "aarch64-Atlas". The target resource file that matches the hardware node can be screened out by combining the hardware information of the physical node with the universal file identifier of the target resource file of the physical node.

[0045] Step S323: combining the hardware identifier corresponding to each piece of hardware information and the universal file identifier to obtain a first exclusive file identifier for the physical node.

[0046] According to the hardware identifier of the hardware information of the physical node where the target task container is currently running, combined with the universal file identifier of the target resource file, a first exclusive file identifier for the resource file of the hardware-sensitive type is generated to match the correct target resource file for the target task container to use. In some application scenarios, the hardware identifier corresponding to each piece of hardware information and the universal file identifier are combined to obtain the first exclusive file identifier for the physical node, including: obtaining a preset identifier that represents the hardware-sensitive type. The preset identifier, the hardware identifier corresponding to each piece of hardware information, and the universal file identifier are combined to obtain the first exclusive file identifier for the physical node. For example, the preset identifier is the prefix "ARCH-", the hardware identifier of the physical node is "aarch64-Atlas", and the universal file identifier is "cluster.zip". The first exclusive file identifier of the physical node can be "ARCH-aarch64-Atlas.cluster.zip".

[0047] Step S13: searching the resource file pool for target resource files that match the first file identifiers.

[0048] The resource file pool is used to store resource files. The resource files in the resource file pool can be shared with each physical node. Each physical node can download resource files from the resource file pool for task operation. After the user uploads a resource file to the resource file pool for the first time, the resource file can be directly matched and downloaded from the resource file pool when it is needed later, without the need to upload it again.

[0049] In some embodiments, the resource file information of the target task container submitted by the user may also include the storage path of each target resource file. Each target resource file may be stored in a resource file pool or on the physical machine where the user is located. If the target resource file submitted by the user is not stored in the resource file pool, the target resource file needs to be downloaded and stored in the resource file pool so that it can be downloaded and used when the subsequent target task container or other task containers that need the file are running. In some specific embodiments, the resource file pool may be a Hadoop Distributed File System (HDFS), and the resource file pool is connected to each physical machine in communication to achieve resource file sharing.

[0050] In some embodiments, searching for target resource files that match each first file identifier from the resource file pool includes: obtaining a third file identifier of each resource file from the resource file pool. Searching for a third file identifier that matches the first file identifier from the third file identifier of each resource file. The resource file corresponding to the searched third file identifier is used as the target resource file. In some application scenarios, the third file identifier for a resource file that belongs to a hardware-sensitive type includes a preset identifier that characterizes the hardware-sensitive type, a hardware identifier corresponding to the resource file, and a general identifier of the resource file. In some application scenarios, the third file identifier for a resource file that does not belong to a hardware-sensitive type may be a general identifier of the resource file. Before executing step S13, a first file identifier for matching the target resource file may be set for the file of the type. Specifically, in response to the target resource file not belonging to the hardware-sensitive type, a general file identifier of the target resource file is obtained as the first file identifier of the target resource file, so as to match the corresponding target resource file in the file resource pool. The combination of each identifier in the third file identifier may be unified with the combination of each identifier in the first file identifier, so as to match the corresponding target resource file in the resource file pool using the first file identifier.

[0051] In some embodiments, before executing step S13, the following steps are also included:

[0052] Step S14: Download the target resource files to the storage space of the corresponding physical nodes respectively, wherein the target resource files are used when the physical nodes run the target task containers.

[0053] Automatically download the matching target resource file according to the hardware information of the physical node to ensure that the target task container can run stably.

[0054] In some embodiments, the first file identifier includes a universal file identifier of the target resource file. After the target resource file is downloaded to the storage space of the corresponding physical node respectively, the method further includes: generating a soft link named with the universal file identifier of the target resource file for the physical node, wherein the physical node can access the target resource file through the soft link. It can be understood that when the application on the physical node needs to access the target resource file, it only needs to use the universal file identifier to obtain the resource file matching the physical node through the soft link. For example, the third file identifier of the architecture-sensitive type target resource file matched in the file pool is "ARCH-aarch64-Atlas.cluster.zip". After the target resource file is downloaded to the storage space of the physical node, the universal file identifier "cluster.zip" can be used to set a soft link. When the application on the physical node needs to use the file, "cluster.zip" can be used directly. There is no need to additionally judge the hardware architecture of the current physical node in the application, and then decide which target resource file to use, which simplifies the development content of the application and reduces the development cost.

[0055] In some application scenarios, there are user tasks that adapt to multiple different hardware architectures, and multiple task containers of the task may be scheduled to run on hardware nodes of different hardware architectures. Steps S11 to S14 can be performed through a resource management platform, wherein the resource management platform can include a resource management and task scheduling framework YARN (Yet Another Resource Negotiator), and YARN runs on multiple physical nodes in the cluster. When a user submits a task, the resource files required for the task and the storage paths corresponding to these resource files can be specified by configuring the parameter yarn.ship-archives, and all hardware types of the resource files need to be included when specifying. These resource files need to follow the naming convention: ARCH-${Platform}-${GpuType}.${ResourceName}, wherein "ARCH-" is a preset identifier, "Platform" is a CPU architecture, "GpuType" is a GPU model, and "ResourceName" is a general file identifier. For example, the user submits the configuration parameters yarn.ship-archives: ARCH-amd64-T4.cluster.zip, ARCH-aarch64-Atlas.cluster.zip, test.zip.

[0056] During the process of starting and running the target task container, in response to the target task container requiring a hardware-sensitive resource file, YARN will search for hardware-sensitive resource files whose file names begin with "ARCH-" in the resource files specified by the user, and delete the preset identifier "ARCH-", and the hardware identifier: "Platform" field and "GpuType" field in the file names of these resource files. Obtain a general file identifier, such as "cluster.zip". YARN automatically obtains the CPU architecture and GPU model of the physical node where the current target container is running. Reassemble the target resource file name currently actually used by the target task container according to the above naming convention. Then, according to the target resource file name, traverse the files in the resource file pool pointed to by the file path of the resource file specified by the user to find out whether there is a resource file with the same name as the resource file. If it exists, download the resource file to the storage space of the physical node where the current target task container is running. If it does not exist, determine whether the current target task container is an AM container (Application Master container). If it is an AM container, no error is reported. If it is not an AM container, an error is reported and exited. AM containers do not apply for GPU resources when running, so it is normal that there is no corresponding resource file. During the process of starting and running the target task container, in response to the resource file required by the target task container being a hardware-sensitive resource file, YARN can directly traverse the files in the resource pool pointed to by the file path of each resource file specified by the user based on the name of the resource file submitted by the user, and search for whether a resource file with the same name as the resource file exists. If so, the resource file is downloaded to the local computer. If not, an error is reported and the system exits.

[0057] In some application scenarios, there are user tasks that are hardware-sensitive. For example, a task is only compatible with one or some physical nodes. If the task is directly run in a cluster environment with mixed deployment of multiple physical nodes, the task may fail to run due to random scheduling strategies that schedule the task to an unsupported physical node. For the above scenarios, this application can also schedule the task container to run on a physical node that matches the specified running hardware information based on the running hardware information specified by the user-specified target task container. For details, please refer to Figure 4 , Figure 4 This is a target task container scheduling flow chart of an embodiment of a task processing method of the present application, which specifically includes the following steps:

[0058] Step S41: Obtain the operating hardware information specified by the user for the target task container.

[0059] Specifically, the operating hardware includes but is not limited to a central processing unit (CPU) and a graphics processing unit (GPU), and the operating hardware information may include a graphics processing unit type and a central processing unit architecture.

[0060] In some embodiments, obtaining the operating hardware information specified by the user for the target task container includes: receiving the target environment parameters set by the user, and the value of the target environment parameters is used to characterize the operating hardware information. Specifically, the operating hardware information can be set in the environment parameters in the form of a key-value pair, wherein the value of each target environment parameter can refer to one type of operating hardware information, or can refer to multiple types of operating hardware information. For example, the value of the target environment parameter can be: T4, altlas, or Mix, wherein Mix can refer to both T4 and altlas.

[0061] Step S42: Based on the running hardware information, the target task container is scheduled to at least one physical node, wherein the hardware information of the at least one physical node matches the running hardware information.

[0062] Specifically, the target task container can be scheduled to a physical node with a CPU architecture specified by the user. The target task container can also be scheduled to a physical node with a CPU architecture equipped with a graphics processor type specified by the user, so as to achieve flexible scheduling of the target task container.

[0063] In some embodiments, based on the running hardware information, the target task container is scheduled to at least one physical node, including: rendering the target environment parameters in the form of annotations to the configuration file corresponding to the target task container. Based on the annotations in the configuration file corresponding to the target task container, the target task container is scheduled to at least one physical node. Specifically, the configuration file is used to define the specifications and configuration of a minimum deployable unit, and a minimum deployable unit may include one or more containers. In other words, a configuration file can correspond to a task container, and a configuration file can also correspond to multiple containers, which is not limited in this application. By parsing the configuration file corresponding to the target task container, searching for the annotation corresponding to the target task container in the configuration file, obtaining the running hardware information in the annotation, matching the physical node that adapts to the running hardware information according to the running hardware information, and scheduling the target task container to the node.

[0064] In some application scenarios, based on the annotations in the configuration file corresponding to the target task container, the target task container is scheduled to at least one physical node, including: identifying the annotations in the configuration file corresponding to the target task container. The identified annotations are converted into physical node labels, and the target task container is scheduled to the physical node that matches the physical node label. In some embodiments, the identified annotations can be converted into node affinity labels, all physical nodes are traversed, the physical nodes that match the node affinity labels are screened out, and the target task container is scheduled to the physical node. In a specific embodiment, the node affinity label can be a key-value pair, and the physical node has a corresponding key-value pair. When the key in the key-value pair of the node affinity label is the same as the key of the corresponding key-value pair of the physical node, and the value in the key-value pair of the node affinity label is also the same as the value of the corresponding key-value pair of the physical node, it is determined that the physical node matches the node affinity label.

[0065] In some application scenarios, the steps of receiving the target environment parameters set by the user and rendering the target environment parameters in the form of annotations to the configuration file corresponding to the target task container are performed by the resource management platform, and the resource management platform is also used to send the configuration file to the container management platform. Based on the annotations in the configuration file corresponding to the target task container, the step of scheduling the target task container to at least one physical node is performed by the container management platform.

[0066] In some specific embodiments, the resource management platform may include a resource management and task scheduling framework YARN (YetAnother Resource Negotiator), which runs on multiple physical nodes in the cluster. The container management platform may include the open source container orchestration platform Kubernetes. Specifically, when a user submits a task, it is necessary to clearly specify the operating hardware information required for the target task container, including the CPU architecture (such as x86 or ARM) and / or GPU model, etc. The user uses the environment variable parameter ENV provided by the YARN framework to pass the required GPU model and other hardware information to YARN as an environment variable (such as GPUMODEL) when submitting a task. After receiving this information, YARN will parse the values ​​of these environment variables and render them in the form of annotations to the Pod's configuration file pod.yaml. Pod.yaml is a configuration file in Kubernetes that is used to define the specifications and configuration of Pod. Pod is the smallest deployable computing unit in Kubernetes and consists of one or more containers. For example, if the hardware information specified by the user is a GPU model, the GPU model is added to the definition of the Pod as an annotation. After the Kubernetes cluster recognizes the annotations in the Pod configuration file, it converts these annotations into node affinity labels (Node Affinity) or node selectors (Node Selector). Based on these affinity labels or node selectors, the Kubernetes scheduler schedules the Pod to the physical node that meets its hardware requirements. Specifically, it traverses each physical node, selects the physical node that matches the node affinity label or node selector, and schedules the target task container to the physical node. After the Pod is scheduled to the target physical node, Kubernetes will be responsible for mounting the specified GPU device to the Pod to ensure that the target task container can use the correct GPU resources.

[0067] In the above scheme, the present application identifies that the target resource file required by the target task container is a hardware-sensitive type of resource file, generates a first exclusive file identifier for physical nodes with different hardware conditions, and searches and downloads the matching target resource file in the resource file pool according to the first exclusive file identifier, thereby improving the adaptability of the resource file and the success rate of the target task container operation.

[0068] See also Figure 5 , Figure 5It is a framework diagram of an embodiment of a task processing device of the present application. The task processing device 50 includes: a determination module 51, a matching module 52 and a download module 53. Among them, the determination module 51 is used to determine the target resource file required by the target task container, wherein the target task container is scheduled to run on at least one physical node. In response to the target resource file belonging to the hardware-sensitive type, the matching module 52 generates a first exclusive file identifier for each physical node for the target resource file, respectively, as the first file identifier of the target resource file, wherein the hardware-sensitive type represents that the target resource files required by the physical nodes with different hardware conditions are different; and finds the target resource files that match each first file identifier from the resource file pool. The download module 53 downloads the target resource files to the storage space of the corresponding physical node respectively, wherein the target resource files are used when the physical node runs the target task container.

[0069] In some embodiments, the matching module 52 generates a first exclusive file identifier for each physical node for the target resource file, including: obtaining at least one hardware information of each physical node; for each physical node, obtaining a hardware identifier corresponding to each hardware information of the physical node, and a universal file identifier of the target resource file; combining the hardware identifier corresponding to each hardware information and the universal file identifier to obtain a first exclusive file identifier for the physical node.

[0070] In some application scenarios, the hardware information of the physical node includes at least one of a central processing unit architecture and a graphics processing unit type. In some application scenarios, the matching module 52 combines the hardware identifier corresponding to each piece of hardware information with the general file identifier to obtain a first exclusive file identifier for the physical node, including: obtaining a preset identifier representing a hardware sensitive type; combining the preset identifier, the hardware identifier corresponding to each piece of hardware information, and the general file identifier to obtain a first exclusive file identifier for the physical node.

[0071] In some embodiments, the determination module 51 determines the target resource file required by the target task container, including: obtaining the resource file information of the target task container submitted by the user, the resource file information including the second file identifier of each target resource file required by the target task container, the second file identifier in the resource file information including a universal file identifier and / or a second exclusive file identifier, each universal file identifier represents a target resource file used by physical nodes of all hardware conditions, each second exclusive file identifier represents a target resource file used by a physical node of a hardware condition, and the second exclusive file identifier includes a preset identifier representing a hardware sensitive type and a universal file identifier representing the target resource file. The matching module 52 is also used to extract the universal file identifier from each second exclusive file identifier, wherein the extracted universal file identifier is used to generate a first exclusive file identifier for the physical node. Specifically, the determination module 51 is also used to determine the target resource file required by the target task container based on each universal file identifier, and determine the target resource file corresponding to the universal file identifier extracted from the second exclusive file identifier as a target resource file belonging to the hardware sensitive type.

[0072] In some embodiments, the matching module 52 is also used to extract a universal file identifier from each second exclusive file identifier, including: filtering out the second file identifier containing a preset identifier from the resource file information to find out the second exclusive file identifier; removing the preset identifier and the hardware identifier related to the corresponding physical node in the second exclusive file identifier to obtain a universal file identifier.

[0073] In some embodiments, the matching module 52 is also used to obtain a universal file identifier of the target resource file as the first file identifier of the target resource file in response to the target resource file not being a hardware-sensitive type before searching for target resource files that match each first file identifier from the resource file pool, and the universal file identifier corresponds to a physical node of all hardware conditions; and / or, searching for target resource files that match each first file identifier from the resource file pool, including: obtaining a third file identifier of each resource file from the resource file pool; searching for a third file identifier that matches the first file identifier from the third file identifier of each resource file; and using the resource file corresponding to the found third file identifier as the target resource file.

[0074] In some embodiments, the matching module 52 searches for target resource files that match each first file identifier from the resource file pool, including: obtaining the third file identifier of each resource file from the resource file pool; searching for the third file identifier that matches the first file identifier from the third file identifier of each resource file; and using the resource file corresponding to the found third file identifier as the target resource file.

[0075] In some embodiments, the first file identifier includes a universal file identifier of the target resource file; the download module 53 is also used to download the target resource files to the storage space of the corresponding physical nodes respectively, and the method also includes: generating a soft link named with the universal file identifier of the target resource file for the physical node, wherein the physical node can access the target resource file through the soft connection.

[0076] In some other embodiments, the determination module 51 is further used to obtain the operating hardware information specified by the user for the target task container; based on the operating hardware information, the target task container is scheduled to at least one physical node, wherein the hardware information of the at least one physical node matches the operating hardware information. The operating hardware information includes the type of graphics processor.

[0077] In some embodiments, the determination module 51 obtains the operating hardware information specified by the user for the target task container, including: receiving the target environment parameters set by the user, the value of the target environment parameters is used to characterize the operating hardware information. In some specific embodiments, the determination module 51 schedules the target task container to at least one physical node based on the operating hardware information, including: rendering the target environment parameters in the form of annotations to the configuration file corresponding to the target task container; scheduling the target task container to at least one physical node based on the annotations in the configuration file corresponding to the target task container.

[0078] In some embodiments, the determination module 51 schedules the target task container to at least one physical node based on the annotations in the configuration file corresponding to the target task container, including: identifying the annotations in the configuration file corresponding to the target task container; converting the identified annotations into physical node labels, and scheduling the target task container to the physical node that matches the physical node label. In some specific embodiments, the steps of receiving the target environment parameters set by the user and rendering the target environment parameters in the configuration file corresponding to the target task container in the form of annotations are performed by the resource management platform, and the resource management platform is also used to send the configuration file to the container management platform; based on the annotations in the configuration file corresponding to the target task container, the step of scheduling the target task container to at least one physical node is performed by the container management platform.

[0079] In the above scheme, the present application identifies that the target resource file required by the target task container is a hardware-sensitive type of resource file, generates a first exclusive file identifier for physical nodes with different hardware conditions, and searches and downloads the matching target resource file in the resource file pool according to the first exclusive file identifier, thereby improving the adaptability of the resource file and the success rate of the target task container operation.

[0080] See also Figure 6 , Figure 6Schematic diagram of the framework of the device of the present application. The device 60 includes a processor 61 and a memory 62 connected to the processor 61, the memory 62 stores program instructions, and the processor 61 can be any device with processing capabilities, such as a computer, a tablet computer, a mobile phone, etc.

[0081] The memory 62 is used to store the program data executed by the processor 61 and the data of the processor 61 during the processing process. For example, the amount of data, metadata, timing data, etc. are stored. Among them, the memory 62 includes a non-volatile storage part for storing the above program data. The processor 61 controls the operation of the processing device, and the processor 61 can also be called a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 61 can be implemented by multiple integrated circuit chips. The processor 61 calls the program data stored in the memory 62, and the processor 61 is used to execute program instructions to implement the data processing method of any of the above embodiments.

[0082] See also Figure 7 , Figure 7 Schematic diagram of a computer-readable storage medium of the present application. In this embodiment, the storage medium 70 stores program data 71 executable by a processor, and the program data 71 can be executed to implement the data processing method of any of the above embodiments.

[0083] The storage medium 70 may specifically be a medium that can store the program data 71, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the program data 71. The server may send the stored program data 71 to other devices for execution, or may execute the stored program data 71 by itself.

[0084] In some embodiments, the storage medium 70 may also be Figure 6 The memory 62 is shown.

[0085] In the above scheme, the present application identifies that the target resource file required by the target task container is a hardware-sensitive type of resource file, generates a first exclusive file identifier for physical nodes with different hardware conditions, and searches and downloads the matching target resource file in the resource file pool according to the first exclusive file identifier, thereby improving the adaptability of the resource file and the success rate of the target task container operation.

[0086] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A task processing method, characterized in that: The method comprises: The resource management platform determines a target resource file required by a target task container, wherein the target task container is scheduled to run on at least one physical node; In response to the target resource file belonging to the hardware sensitive type, at least one piece of hardware information of each of the physical nodes is obtained, and for each of the physical nodes, a hardware identifier corresponding to each piece of hardware information of the physical node and a universal file identifier of the target resource file are obtained; the hardware identifier corresponding to each piece of hardware information and the universal file identifier are combined to obtain a first exclusive file identifier for the physical node, which is used as a first file identifier of the target resource file, respectively, wherein the hardware sensitive type indicates that the target resource files required for the physical nodes with different hardware conditions are different; Find the target resource files that match the first file identifiers from the resource file pool; The target resource files are downloaded to the storage space of the corresponding physical nodes respectively, wherein the target resource files are used when the physical nodes run the target task containers to ensure that the target task containers run correctly on the corresponding physical nodes.

2. The method according to claim 1, characterized in that The hardware information of the physical node includes at least one of a central processing unit architecture and a graphics processing unit type; And / or, combining the hardware identifier corresponding to each piece of hardware information with the universal file identifier to obtain a first exclusive file identifier for the physical node includes: Obtaining a preset identifier representing the hardware sensitive type; The preset identifier, the hardware identifier corresponding to each piece of hardware information, and the universal file identifier are combined to obtain a first exclusive file identifier for the physical node.

3. The method according to claim 1, characterized in that: The target resource file required for determining the target task container includes: Obtain resource file information of the target task container submitted by the user, the resource file information including a second file identifier of each target resource file required by the target task container, the second file identifier in the resource file information including a universal file identifier and / or a second exclusive file identifier, each universal file identifier representing a target resource file that can be used by physical nodes of all hardware conditions, each second exclusive file identifier representing a target resource file used by a physical node of a certain hardware condition, and the second exclusive file identifier including a preset identifier representing the hardware sensitive type and a universal file identifier representing the target resource file; Extracting the universal file identifier from each of the second exclusive file identifiers, wherein the extracted universal file identifier is used to generate a first exclusive file identifier for the physical node; The target resource file required by the target task container is determined based on each of the universal file identifiers respectively, and the target resource file corresponding to the universal file identifier extracted from the second exclusive file identifier is determined as a target resource file belonging to the hardware sensitive type.

4. The method according to claim 3, characterized in that The step of extracting the universal file identifier from each of the second specific file identifiers includes: Filtering the second file identifier including the preset identifier from the resource file information to find out the second exclusive file identifier; The preset identifier in the second exclusive file identifier and the hardware identifier related to the corresponding physical node are removed to obtain the universal file identifier.

5. The method according to claim 1, characterized in that Before searching the resource file pool for target resource files that match the first file identifiers, the method further includes: In response to the target resource file not being of a hardware-sensitive type, obtaining a universal file identifier of the target resource file as a first file identifier of the target resource file, wherein the universal file identifier corresponds to a physical node of all hardware conditions; And / or, searching for target resource files matching each of the first file identifiers from a resource file pool includes: Obtaining a third file identifier of each resource file from the resource file pool; Finding a third file identifier that matches the first file identifier from the third file identifiers of the resource files; The resource file corresponding to the found third file identifier is used as the target resource file.

6. The method according to claim 1, characterized in that The first file identifier includes a universal file identifier of the target resource file; After downloading the target resource files to the storage spaces of the corresponding physical nodes respectively, the method further includes: A soft link named with the universal file identifier of the target resource file is generated for the physical node, wherein the physical node can access the target resource file through the soft link.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining the operating hardware information specified by the user for the target task container; Based on the running hardware information, the target task container is scheduled to the at least one physical node, wherein the hardware information of the at least one physical node matches the running hardware information.

8. The method according to claim 7, characterized in that The obtaining the operating hardware information specified by the user for the target task container includes: Receiving a target environment parameter set by a user, wherein a value of the target environment parameter is used to represent the operating hardware information; The step of scheduling the target task container to the at least one physical node based on the running hardware information includes: Rendering the target environment parameters in the form of annotations to the configuration file corresponding to the target task container; Based on the annotation in the configuration file corresponding to the target task container, the target task container is scheduled to the at least one physical node.

9. The method according to claim 8, characterized in that The step of scheduling the target task container to the at least one physical node based on the annotation in the configuration file corresponding to the target task container includes: Identify annotations in a configuration file corresponding to the target task container; Convert the identified annotation into a physical node label, and schedule the target task container to the physical node matching the physical node label; And / or, the step of receiving the target environment parameters set by the user and the step of rendering the target environment parameters in the form of annotations to the configuration file corresponding to the target task container are executed by a resource management platform, and the resource management platform is also used to send the configuration file to the container management platform; the step of scheduling the target task container to the at least one physical node based on the annotations in the configuration file corresponding to the target task container is executed by the container management platform.

10. A device, characterized in that: The device includes a processor and a memory connected to the processor, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the task processing method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The storage medium has program instructions, and the program instructions are executed to implement the task processing method according to any one of claims 1 to 9.

Images