Operator task issuing method and related equipment
By introducing a pre-defined queue into the PyTorch framework, the hierarchical distribution of operator tasks is achieved, which solves the problem of execution devices waiting due to the slow distribution speed of the management device, and improves the efficiency of operator task distribution and computing power utilization.
Patent Information
- Application Number
- CN202411103438.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2026-02-24
AI Technical Summary
In the PyTorch framework, the slow speed at which the management device issues operator tasks causes the execution device to wait, resulting in low computing power utilization.
By introducing a preset queue between the management device and the execution device, the hierarchical distribution of operator tasks is realized. The first device and the second device respectively process the features of the frame stack object and add them to the queue. The second device obtains and processes the tensor features from the queue and then distributes the operator tasks.
This improves the efficiency of operator task distribution, avoids waiting time for the execution device, and enhances the computing power utilization of the execution device.
Smart Images
Figure CN121560464A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method and related equipment for issuing operator tasks. Background Technology
[0002] PyTorch is an open-source machine learning framework that allows for operations and data processing using Python code, with tensors as its core data structure. In PyTorch's native workflow, due to dependencies between operators, the management unit needs to issue operator tasks serially. However, the issuance of operators by the management unit and the execution of operator computations by the execution unit are asynchronous. Therefore, ideally, the speed at which the management unit issues operators needs to be faster than the speed at which the execution unit executes them to ensure the execution pipeline runs continuously.
[0003] However, in practical problems, the process of the management device issuing operators is long and slow, while the execution device executes operators quickly. This can lead to situations where the execution device waits for the management device to issue operator tasks, which causes the execution device pipeline to be unable to operate, resulting in low computing power utilization of the execution device. Summary of the Invention
[0004] This application provides a method and related equipment for issuing operator tasks, which can improve the efficiency of issuing operator tasks and further improve the computing power utilization of the execution device.
[0005] Firstly, a method for issuing operator tasks is provided. This method is applied to the PyTorch framework, which includes a first device, a second device, and a third device. The method includes: the first device obtaining a first feature of a tensor corresponding to a first frame stack object; the first frame stack object corresponding to a first operator task; the first device adding the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue; the second device, if there are frame stack objects to be processed in the preset queue, obtaining a second frame stack object and the first feature of the tensor corresponding to the second frame stack object from the preset queue, and obtaining a second feature of the tensor corresponding to the second frame stack object; the second frame stack object corresponding to a second operator task; the second device obtaining the tensor corresponding to the second frame stack object based on the first and second features of the tensor corresponding to the second frame stack object, and issuing the second operator task to the third device.
[0006] As shown above, the first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue. Without waiting for the execution result, the first device can continue to add the second frame stack object and the first feature of the tensor corresponding to the second frame stack object to the preset queue. When the preset queue contains frame stack objects and the first feature of their corresponding tensors, the second device retrieves pairs of frame stack objects and their corresponding tensors from the preset queue and continues to execute the distribution process of the operator tasks corresponding to the frame stack objects. By hierarchically dividing the operator task distribution process through the first and second devices, asynchronous splitting of the operator task distribution process is achieved, improving the efficiency of operator distribution and further enhancing the computing power utilization of the execution device.
[0007] In one possible implementation, the first feature includes the shape of the tensor; the second feature includes the data type of the tensor and the data memory address.
[0008] As can be seen from the above, when the first feature includes the shape of the tensor and the second feature includes the data type and data memory address of the tensor, the first device obtains the first feature of the tensor corresponding to the frame stack object, and the second device obtains the second feature of the tensor corresponding to the frame stack object. This makes the speed of the first-level pipeline executed by the first device faster and the speed of the second-level pipeline executed by the second device slower, ensuring that the second-level pipeline executed by the second device can run continuously, thereby improving the efficiency of operator distribution.
[0009] In one possible implementation, if the first frame stack object and the second frame stack object are different frame stack objects, before the second device retrieves the first feature of the tensor corresponding to the second frame stack object from the preset queue, the method further includes: the first device retrieving the first feature of the tensor corresponding to the second frame stack object; the second frame stack object corresponds to the second operator task; and the first device adding the first feature of the tensor corresponding to the second frame stack object and the second frame stack object to the preset queue.
[0010] As can be seen from the above, when the first frame stack object and the second frame stack object are different frame stack objects, after the first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to the preset queue, the first device adds the second frame stack object and the first feature of the tensor corresponding to the second frame stack object to the preset queue. This ensures that the execution process of the first device adding the frame stack object and the first feature of the tensor corresponding to the frame stack object to the preset queue is not affected by the execution process or result of the second device.
[0011] In one possible implementation, after the first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue, the method further includes: the first device obtaining the first feature of the tensor corresponding to the third frame stack object; the third frame stack object corresponds to a third operator task; the third operator task is triggered after the first operator task; and the first device adding the third frame stack object and the first feature of the tensor corresponding to the third frame stack object to the preset queue.
[0012] As shown above, after the first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to the preset queue, the first device continues to execute the first stage of the pipeline, adding the third frame stack object and the first feature of the tensor corresponding to the third frame stack object to the preset queue. This achieves asynchronous execution between the first and second devices, improving the efficiency of operator task distribution.
[0013] In one possible implementation, the first device parses the first frame stack object to obtain first information; the first device determines the first feature of the tensor corresponding to the first frame stack object according to a preset mapping relationship; wherein, the preset mapping relationship includes the mapping relationship between the first information in the frame stack object and the first feature of the tensor corresponding to the frame stack object.
[0014] As can be seen from the above, the method by which the first device determines the first feature of the tensor corresponding to the first frame stack object based on the first information obtained by parsing the first frame stack object and the preset mapping relationship can improve the efficiency of the first device in obtaining the first feature of the tensor corresponding to the frame stack object, thereby improving the efficiency of operator task issuance.
[0015] In one possible implementation, the second device includes a native interpreter: the native interpreter parses a first feature for retrieving a second frame stack object from a preset queue and a second feature for the tensor corresponding to the second frame stack object; the native interpreter is also used to obtain the tensor corresponding to the second frame stack object based on the first and second features of the tensor corresponding to the second frame stack object.
[0016] As can be seen from the above, by using the native interpreter in the second device to obtain the first feature of the second frame stack object and the tensor corresponding to the second frame stack object from the preset queue, and to obtain the second feature of the tensor corresponding to the second frame stack object, and to determine the tensor of the second frame stack object, the efficiency of the second device in executing the second-level pipeline can be improved.
[0017] In one possible implementation, the second device further includes a dispatcher: the dispatcher is used to obtain a dispatch identifier based on the second frame stack object and the tensor corresponding to the second frame stack object; determine the execution code of the second operator task corresponding to the second frame stack object based on the dispatch identifier; and send the execution code of the second operator task corresponding to the second frame stack object to the third device.
[0018] In one possible implementation, the third device includes a graphics processing unit (GPU) and a corresponding code converter, or an embedded neural network processing unit (NPU) and a corresponding code converter; wherein the code converter for the GPU is used to convert the executable code into code adapted to the GPU; and the code converter for the NPU is used to convert the executable code into code adapted to the NPU.
[0019] As can be seen from the above, the third device can be any execution device chip. The above method of issuing operator tasks does not require customization or modification of the third device, which improves the flexibility of issuing operator tasks and thus improves the efficiency of issuing operator tasks.
[0020] Secondly, a first apparatus is provided. Embodiments of this application can divide the first apparatus into functional modules according to the method provided in the first aspect. For example, each function can be divided into its own functional modules, or two or more functions can be inherited into a single processing module. For instance, embodiments of this application can divide the first apparatus into an acquisition module and a processing module according to their functions. Descriptions of the possible technical solutions and beneficial effects of the various functional modules described above can be found in the technical solutions provided in the first aspect or its corresponding possible implementations, and will not be repeated here.
[0021] Thirdly, embodiments of this application provide a computing device cluster, which includes at least one computing device. Each computing device includes a processor and a memory for storing processor-executable instructions. The processor is configured to execute instructions, causing the computing device to perform the above-described method for issuing operator tasks.
[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the method for issuing operator tasks as described above.
[0023] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computing node reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing node to perform the operator task dispatching method provided in various optional implementations of the above aspects.
[0024] For a detailed description of the second to fifth aspects and their various implementations in the embodiments of this application, please refer to the detailed description in the first aspect and its various implementations; and for a detailed description of the beneficial effects of the second to fifth aspects and their various implementations, please refer to the beneficial effect analysis in the various implementations of the first aspect, which will not be repeated here.
[0025] These or other aspects of the embodiments of this application will become more apparent in the following description. Attached Figure Description
[0026] Figure 1 A schematic diagram of the execution process of an operator task is shown;
[0027] Figure 2 A schematic diagram of the execution process of an operator task is shown;
[0028] Figure 3 A schematic diagram of the hardware structure of a computing device 100 provided in an embodiment of this application is shown;
[0029] Figure 4 A schematic diagram of an operator task distribution framework 200 provided in an embodiment of this application is shown;
[0030] Figure 5 A schematic diagram of an operator task delivery timing provided in an embodiment of this application is shown;
[0031] Figure 6 A schematic diagram of the method flow for issuing operator tasks provided in an embodiment of this application is shown;
[0032] Figure 7 A schematic diagram of a PyTorch framework software architecture provided in an embodiment of this application is shown;
[0033] Figure 8 A schematic diagram of a first-level pipeline execution method provided in an embodiment of this application is shown;
[0034] Figure 9 A schematic diagram of a two-stage pipeline execution method provided in an embodiment of this application is shown;
[0035] Figure 10 A schematic diagram of a first device structure provided in an embodiment of this application is shown. Detailed Implementation
[0036] The terms “first,” “second,” “third,” and “fourth,” etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to define a specific order.
[0037] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0038] First, the application scenarios of the embodiments of this application will be introduced by way of example.
[0039] PyTorch is a widely used open-source machine learning library, applied extensively in many fields such as image recognition, natural language processing, and speech recognition. It is one of the important tools for deep learning research and development. PyTorch typically provides pre-written functions, classes, and tools, offering a range of common learning algorithms and techniques such as classification, regression, clustering, and dimensionality reduction.
[0040] PyTorch employs a dynamic graph mechanism, where dependencies exist between operator tasks, necessitating a serial delivery of these tasks. This delivery refers to the management unit (such as the central processing unit, CPU) passing the user-invoked operator task to the execution unit for computation. However, the delivery of operator tasks by the management unit and the execution unit's execution of those tasks are asynchronous. Figure 1 A schematic diagram of the execution process of an operator task is shown. For example... Figure 1 As shown, taking the operator dependencies as operator 1, operator 2, operator 3, operator 4, and operator 5 as an example, the management device issues operators in the following order according to these dependencies: operator 1, operator 2, operator 3, operator 4, and operator 5. Taking an execution device consisting of four graphics processing units (GPUs) as an example, after the management device issues operator 1, it does not need to wait for the execution result of operator 1 and can immediately continue issuing operator 2. Similarly, the management device does not need to wait for the execution result of operator 2; after operator 2 is issued, it immediately continues issuing operator 3. While the management device has completed issuing operators 1, 2, 3, 4, and 5 sequentially and is issuing operator 6, the execution device can obtain the operators issued by the management device and execute them serially. Specifically, GPU 01 executes operator 1; GPU 02 executes operator 2; GPU 03 executes operator 3; and GPU 04 executes operator 4. If GPU 01 completes the execution of operator 1 at this time, GPU 01 can obtain operator 5 and execute it.
[0041] However, if the management device issues operators slowly while the execution device executes operators quickly, the execution device may have to wait for operators to be issued, resulting in a waste of the execution device's computing power and thus low computing power utilization.
[0042] Figure 2 A schematic diagram of the execution process of an operator task is shown, such as... Figure 2 As shown, taking the dependency relationship between operators as operator 1, operator 2, operator 3, operator 4, and operator 5 as an example, the management device issues operators in the following order according to the dependency relationship: operator 1, operator 2, operator 3, operator 4, and operator 5. Taking an execution device with four GPUs as an example, the execution device can obtain the operators issued by the management device and execute them sequentially. While the management device has issued operators 1, 2, and 3, and is issuing operator 4, the execution device can obtain the issued operator and execute it sequentially. Specifically, GPU 01 executes operator 1; GPU 02 executes operator 2. After GPU 03 in the execution device obtains operator 3 issued by the management device and executes operator 3, the management device has not yet completed issuing operators 4 and 5. At this time, there are no successfully issued operators, and GPU 04 of the execution device is in an idle state. GPU 04 of the execution device needs to wait for the management device to complete the issuance of operator 4 or operator 5 before it can obtain and continue to execute operators. This results in a waste of the execution device's computing power resources, leading to low computing power utilization of the execution device.
[0043] In view of this, embodiments of this application provide a method for issuing operator tasks. This method is applied to the PyTorch framework, which includes a first device, a second device, and a third device. The method includes: the first device acquiring a first feature of a tensor corresponding to a first frame stack object; the first device adding the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue. The first frame stack object corresponds to a first operator task. If there are unprocessed frame stack objects in the preset queue, the second device acquires a second frame stack object and the first feature of the tensor corresponding to the second frame stack object from the preset queue, and acquires a second feature of the tensor corresponding to the second frame stack object. The second frame stack object corresponds to a second operator task. The second device obtains the tensor corresponding to the second frame stack object based on the first and second features of the tensor corresponding to the second frame stack object, and issues the second operator task to the third device.
[0044] In the above embodiments, the first device obtains the first feature of the tensor corresponding to the first frame stack object and adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue. Simultaneously, the second device obtains pairs of frame stack objects and the first features of the tensors corresponding to the frame stack objects from the preset queue, obtains the second feature of the tensors corresponding to the frame stack objects, and distributes the operator task to the third device. By simultaneously distributing the frame stack objects through the first and second devices, hierarchical distribution of operator tasks is achieved, saving time and improving efficiency. Because the efficiency of operator distribution is improved and the distribution time is shortened, the situation where the speed of operator distribution is slower than the speed of operator execution is avoided, thus preventing the execution device from waiting for operator distribution and further improving the computing power utilization of the execution device.
[0045] Secondly, the system architecture of the embodiments of this application will be described by way of example.
[0046] The operator task distribution method provided in this application can be applied to computing devices. Figure 3 A schematic diagram of the hardware structure of a computing device 100 provided in an embodiment of this application is shown. For example... Figure 3 As shown, computing device 100 may include CPU 110, GPU 120, memory 130, and bus 140. The CPU 110, GPU 120, and memory 130 are typically interconnected via bus 140, or interconnected in other ways. Computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of CPUs 110, GPUs 120, and memory 130 in computing device 100.
[0047] The CPU 110 is the core component of the computing device 100, responsible for executing computer program instructions, performing logical operations, and controlling the operation flow of the computing device. The CPU 110 has a first device and a second device running on it. The first device is used to obtain the first feature of the tensor corresponding to the first frame stack object. The first frame stack object corresponds to a first operator task. The first device is also used to add the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue. The second device is used to, if there are frame stack objects to be processed in the preset queue, obtain a second frame stack object and the first feature of the tensor corresponding to the second frame stack object from the preset queue, and obtain the second feature of the tensor corresponding to the second frame stack object; the second frame stack object corresponds to a second operator task. The second device is also used to obtain the tensor corresponding to the second frame stack object based on the first and second features of the tensor corresponding to the second frame stack object, and to send the second operator task to a third device.
[0048] GPU 120 is a processor in computing device 100 that performs image and graphics-related computations. GPU 120 is used to execute operator tasks; that is, the third device includes GPU 120.
[0049] Optionally, the computing device 100 also includes an embedded neural network processing unit (NPU). An NPU is a processor specifically designed to accelerate artificial intelligence tasks. The NPU can be used to perform operator tasks; that is, the third device includes an NPU.
[0050] The memory 130 may include volatile memory, such as random access memory (RAM). The memory 130 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0051] The memory 130 stores executable program code, which the CPU 110 executes to implement the functions of the first device and the second device, thereby realizing the method of issuing operator tasks. In other words, the memory 130 stores instructions for executing the method of issuing operator tasks.
[0052] Bus 140 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus 140 is represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 140 may include a path for transmitting information between various components of the computing device 100 (e.g., CPU 110, GPU 120, and memory 130).
[0053] Figure 4 This diagram illustrates an operator task distribution framework 200 provided in an embodiment of this application. Figure 4 As shown, the operator task distribution framework 200 includes a first device 410, a second device 420, a third device 430, and a preset queue.
[0054] The first device can run on CPU 110. The first device 410 is used to obtain the first feature of the tensor corresponding to the first frame stack object; and add the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue. The first frame stack object corresponds to the first operator task. Specifically, the first device can be a logic interpreter defined using a custom interface in the PyTorch framework, and is different from the native interpreter in the PyTorch framework. The preset queue can be in memory.
[0055] The second device can run on CPU 110. The second device 420 is used to, when there are frame stack objects to be processed in a preset queue, retrieve a second frame stack object from the preset queue, and retrieve the first feature of the tensor corresponding to the second frame stack object, and the second feature of the tensor corresponding to the second frame stack object; it is also used to, based on the first and second features of the tensor corresponding to the second frame stack object, obtain the tensor corresponding to the second frame stack object, and distribute the second operator task to the third device. The second frame stack object corresponds to the second operator task.
[0056] Specifically, the second device 420 includes a native interpreter and a dispatcher. The native interpreter retrieves a second frame stack object and a first feature of the tensor corresponding to the second frame stack object from a preset queue, and retrieves a second feature of the tensor corresponding to the second frame stack object. The native interpreter also obtains the tensor corresponding to the second frame stack object based on the first and second features. The dispatcher obtains a dispatch identifier based on the second frame stack object and the tensor corresponding to the second frame stack object; determines the execution code of the second operator task corresponding to the second frame stack object based on the dispatch identifier; and sends the execution code of the second operator task corresponding to the second frame stack object to the third device.
[0057] The third device 430 is used to execute the operator task corresponding to the frame stack object. The method for issuing operator tasks provided in this embodiment is not limited to the chip type of the third device and is applicable to various third devices. Specifically, the third device includes a graphics processing unit (GPU) and a corresponding code converter, or an embedded neural network processor (NPU) and a corresponding code converter. The GPU-corresponding code converter is used to convert the executable code into code adapted to the GPU; the NPU-corresponding code converter is used to convert the executable code into code adapted to the NPU.
[0058] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0059] For ease of understanding, the method for distributing operator tasks provided in this application is described below with reference to the accompanying drawings. This method for distributing operator tasks is applicable to... Figure 3 The computing device shown.
[0060] Figure 5 This diagram illustrates a timing sequence for the distribution of an operator task according to an embodiment of this application. Figure 5 As shown, taking the management device as CPU 110 and the operator distribution sequence as operator 1, operator 2, and operator 3 as an example, the operator distribution timing is described. When CPU 110 distributes operators 1, 2, and 3 serially, after CPU 110 distributes operator 1-01, it continues to distribute operator 1-02. Only after operators 1-01 and 1-02 have been distributed will CPU 110 begin distributing operators 2-01 and 2-02. Similarly, only after CPU 110 has distributed operators 2-01 and 2-02 will it begin distributing operators 3-01 and 3-02. Therefore, the time for CPU 110 to serially distribute operators 1, 2, and 3 is... Figure 5 As shown in T1. The time taken to send operator 1 includes the sum of the time taken to send operator 1-01 and the time taken to send operator 1-02; the time taken to send operator 2 includes the sum of the time taken to send operator 2-01 and the time taken to send operator 2-02; and the time taken to send operator 3 includes the sum of the time taken to send operator 3-01 and the time taken to send operator 3-02.
[0061] In this embodiment, the CPU 110 operates a first device and a second device, capable of hierarchically processing the operator distribution process. That is, after the first device completes the distribution processing of operator 1-01, the second device distributes operator 1-02. At this time, the first device does not need to obtain the distribution result of operator 1-02 from the second device; the first device can continue to distribute operator 2-01. After the first device completes processing operator 2-01, the second device distributes operator 2-02. At this time, the first device can distribute operator 3-01. After the first device completes processing operator 3-01, the second device distributes operator 3-02. Therefore, the time for the CPU 110 to distribute operators 1, 2, and 3 hierarchically is... Figure 5 As shown in T2. From Figure 5It can be seen that the time T2 for using the hierarchical distribution of operators is reduced by T0 compared to the time T0 for using the serial distribution of operators. Therefore, the operator task distribution method provided in this application embodiment can save the time for distributing operator tasks and improve the efficiency of operator task distribution. At the same time, since the amount of distribution tasks processed by the first device and the second device is relatively balanced, that is, the duration of the distribution tasks processed by the first device is relatively close to the duration of the distribution tasks processed by the second device, the hierarchical effect is more obvious, which can further save the time for distributing operator tasks and optimize the method of distributing operator tasks.
[0062] Figure 6 A schematic diagram illustrating the method flow for issuing operator tasks according to an embodiment of this application is shown. Figure 6 As shown, the method for issuing this operator task may include the following steps:
[0063] S101, the computing device obtains the first feature of the tensor corresponding to the first frame stack object through the first device.
[0064] The first frame stack object corresponds to the first operator task. The frame stack object is created by the management device (such as the CPU) based on the user script. It is a data structure used to manage the execution state of the code in the user script and to implement the operator tasks indicated by the code in the user script. The user script is written in Python and is used to initiate operator tasks. The first frame stack object of the computing device is the frame stack object currently being processed by the management device (such as the CPU). The first frame stack object is a frame stack object created by the management device (such as the CPU) based on the first user script. A tensor is a basic form of data identification, including features such as dimension, data type, shape, and data memory address. The first feature includes the shape of the tensor. The tensor corresponding to the first frame stack object is obtained by the management device (such as the CPU) after parsing and calculating the first frame stack object. The first feature of a tensor includes its shape.
[0065] The operator task distribution method provided in this application is applied to the PyTorch framework. The PyTorch framework includes a first device, a second device, and a third device.
[0066] S102, the computing device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue through the first device.
[0067] In other words, after the first device obtains the first feature of the tensor corresponding to the first frame stack object, it adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue as a task to be processed, waiting for the second device to continue execution.
[0068] In one possible implementation, the computing device obtains a first feature of the tensor corresponding to the second frame stack object through a first device; the second frame stack object corresponds to a second operator task; the first device adds the second frame stack object and the first feature of the tensor corresponding to the second frame stack object to a preset queue.
[0069] In other words, if the first frame stack object and the second frame stack object are different, the second frame stack object corresponds to the second operator task; the second operator task is triggered after the first operator task. After the first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to the preset queue, the first device does not need to wait for the result of the second device executing the first frame stack object and the first feature of the tensor corresponding to the first frame stack object. The first device obtains the first feature of the tensor corresponding to the second frame stack object according to the operator task triggering order; and adds the first feature of the second frame stack object and the first feature of the tensor corresponding to the second frame stack object to the preset queue.
[0070] Optionally, after the first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue, the method further includes: the computing device obtaining the first feature of the tensor corresponding to the third frame stack object through the first device; and the first device adding the third frame stack object and the first feature of the tensor corresponding to the third frame stack object to the preset queue.
[0071] The third frame stack object corresponds to the third operator task; the third operator task is triggered after the first operator task.
[0072] For example, after adding the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue, the first device parses the third frame stack object to obtain the first information according to the execution order of the operator tasks. The first device then obtains the first feature of the tensor corresponding to the third frame stack object according to a preset mapping relationship. Finally, the first device adds the third frame stack object and the first feature of the tensor corresponding to the third frame stack object to the preset queue.
[0073] S103, if there is a frame stack object to be processed in the preset queue through the second device, the computing device obtains the second frame stack object and the first feature of the tensor corresponding to the second frame stack object from the preset queue, and obtains the second feature of the tensor corresponding to the second frame stack object.
[0074] The second frame stack object corresponds to the second operator task; the first feature of the tensor corresponding to the second frame stack object and the second frame stack object is added to the preset queue by the first device.
[0075] In this embodiment, the first device and the second device can operate simultaneously. The first device adds each frame stack object and the first feature of the tensor corresponding to each frame stack object to a preset queue according to the order in which the operator tasks are triggered. The second device retrieves each frame stack object to be processed from the preset queue and processes it to obtain the second feature of the tensor corresponding to the second frame stack object.
[0076] S104, the computing device obtains the tensor corresponding to the second frame stack object through the second device based on the first and second features of the tensor corresponding to the second frame stack object.
[0077] The second feature includes the tensor's data type and data memory address.
[0078] The following example describes how the management device (such as the CPU) determines the first and second characteristics of the tensor corresponding to the first frame stack object.
[0079] In one possible implementation, where the first device is a logic interpreter defined through a custom interface in the PyTorch framework, and the second device is a native interpreter in the PyTorch framework, the first device parses the first frame stack object to obtain first information. The first device determines the first feature of the tensor corresponding to the first frame stack object based on a preset mapping relationship. The preset mapping relationship includes the mapping relationship between the first information in the frame stack object and the first feature of the tensor corresponding to the frame stack object. The native interpreter parses the first frame stack object and the first feature of the tensor corresponding to the first frame stack object from a preset queue, and obtains the second feature of the tensor corresponding to the first frame stack object; the native interpreter is also used to obtain the tensor corresponding to the first frame stack object based on the first and second features. The tensor corresponding to the first frame stack object includes both the first and second features.
[0080] for example, Figure 7 This diagram illustrates a PyTorch framework software architecture provided in an embodiment of this application. Figure 7 As shown, the PyTorch framework software architecture includes a logical interpreter 510 and a native interpreter 520. The logical interpreter 510 is a custom interpreter defined through a custom interface within the PyTorch framework, unlike the native interpreter 520 provided by the PyTorch framework. The native interpreter 520 is provided by the PyTorch framework and typically includes the CPython interpreter.
[0081] The first device uses a logic interpreter 510 to parse the code in the first frame stack object line by line to obtain the first information. Then, through a preset mapping relationship, it determines and outputs the first feature of the tensor corresponding to the first frame stack object; that is, it adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue. The logic interpreter 510 includes a preset mapping relationship. This preset mapping relationship includes the mapping relationship between the first information in the frame stack object and the first feature of the tensor corresponding to the frame stack object.
[0082] The second device utilizes a native interpreter 520 to retrieve pairs of first-frame stack objects and the first features of the tensors corresponding to the first-frame stack objects from a preset queue, and then interprets and executes the code in the first-frame stack objects line by line. Taking the CPython interpreter as an example, the CPython interpreter is a Python interpreter written in C. The CPython interpreter is typically used to interpret and execute Python programs line by line, obtaining the first and second features of the tensors corresponding to the frame stack objects, and translating Python high-level language instructions into machine instructions executable by the computing device.
[0083] In this embodiment, the second device of the computing device uses a CPython interpreter to parse and execute the code in the first frame stack object line by line, obtaining the second feature of the tensor corresponding to the first frame stack object. The CPython interpreter updates the memory address corresponding to the first feature of the tensor corresponding to the second feature of the tensor corresponding to the first frame stack object, as output by the first device, to the memory address corresponding to the second feature of the tensor corresponding to the first frame stack object, thereby determining the tensor corresponding to the first frame stack object. At this time, the tensor corresponding to the first frame stack object includes shape, data type, and data memory address.
[0084] Optionally, in the first device, if the logic interpreter 510 obtains the first feature of the tensor corresponding to the first frame stack object, and the execution logic is the same as that of the CPython interpreter in obtaining the first feature of the tensor corresponding to the first frame stack object, the first device parses the first frame stack object through the logic interpreter 510. After calculating and obtaining the first feature of the tensor corresponding to the first frame stack object, it does not continue execution and directly outputs the first feature of the tensor corresponding to the first frame stack object. The second device continues to interpret and execute the code in the first frame stack object line by line through the native interpreter to obtain the second feature of the tensor corresponding to the first frame stack object. Based on the first feature of the tensor corresponding to the first frame stack object output by the first device and the second feature of the tensor corresponding to the first frame stack object obtained by the second device, the tensor corresponding to the first frame stack object is finally obtained.
[0085] S105, the computing device sends the second operator task to the third device through the second device.
[0086] In this embodiment, the third device is used to execute the operator tasks corresponding to each frame stack object. Furthermore, the method for issuing operator tasks provided in this embodiment is not limited to the chip type of the third device and is applicable to various third devices.
[0087] In one possible implementation, the third device includes a graphics processing unit (GPU) and a corresponding code converter, or an embedded neural network processor (NPU) and a corresponding code converter.
[0088] The code converter for GPU is used to convert the executable code into code adapted to the GPU; the code converter for NPU is used to convert the executable code into code adapted to the NPU.
[0089] In one possible implementation, the second device further includes a dispatcher, which is used to obtain a dispatch identifier based on the tensor corresponding to the second frame stack object; determine the execution code of the second operator task corresponding to the second frame stack object based on the dispatch identifier; and send the execution code of the second operator task corresponding to the second frame stack object to the third device.
[0090] For example, the dispatcher in the second device obtains a dispatch identifier based on the tensor corresponding to the second frame stack object. Based on the dispatch identifier and the dispatch table of the second frame stack object, the dispatcher determines the specific function to be executed for the second operator task corresponding to the second frame stack object, thereby determining the execution code for the second operator task corresponding to the second frame stack object, and then sends the execution code to the third device. The dispatch table includes the dispatch identifier and a pointer to the corresponding function.
[0091] The following description will focus on the differences between the original PyTorch process for issuing operator tasks and the embodiments of this application, to illustrate the method for issuing the above-mentioned operator tasks.
[0092] In the original PyTorch workflow, firstly, a frame stack object is created for the user's script using the CPython interpreter, and the CPython interpreter parses and processes the frame stack object to obtain the tensor corresponding to the frame stack object. Secondly, the dispatch process begins. Based on the tensor corresponding to the frame stack object, the dispatcher distributes the operator task corresponding to the frame stack object to the third device.
[0093] In this embodiment, by splitting the original PyTorch process and adding a second-level pipeline, parallel processing of operator task distribution can be achieved, improving the efficiency of operator task distribution. The split first-level pipeline is executed by the first device. Figure 8 A schematic diagram of a first-level pipeline execution method provided in an embodiment of this application is shown. Figure 8As shown, the first device acquires a frame stack object (S201), wherein the frame stack object can be created by a management device (such as a CPU) for a user script, and the user script is used to indicate the operator task to be executed. The first device parses the frame stack object to obtain the first feature of the tensor corresponding to the frame stack object (S202). Specifically, the first device acquires a first frame stack object created by a management device (such as a CPU) for a first user script, and the first frame stack object corresponds to a first operator task. If the first device includes a logic interpreter defined by a custom interface in the PyTorch framework, and is different from the native interpreter in the PyTorch framework, the logic interpreter parses the first frame stack object to obtain first information, and determines the first feature of the tensor corresponding to the first frame stack object according to the first information and a preset mapping relationship. The preset mapping relationship includes the mapping relationship between the first information in the frame stack object and the first feature of the tensor corresponding to the frame stack object. The first device adds the frame stack object and the first feature of the tensor corresponding to the frame stack object to a preset queue (S203). Specifically, the first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue. After adding the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue, the first device obtains a second frame stack object created by a management device (such as a CPU) for a second user script. The second frame stack object corresponds to a second operator task, which is triggered after the first operator task. Similarly, the first device obtains the first feature of the tensor corresponding to the second frame stack object and adds the second frame stack object and the first feature of the tensor corresponding to the second frame stack object to the preset queue.
[0094] The split-level secondary flow is executed by the second device. Figure 9 A schematic diagram of a two-level pipeline execution method provided in an embodiment of this application is shown. Figure 9As shown, when there are frame stack objects to be processed in the preset queue, the second device obtains a pair of frame stack objects and the first feature of the tensor corresponding to the frame stack objects from the preset queue (S301). For example, the second device obtains a pair of first frame stack objects and the first feature of the tensor corresponding to the first frame stack objects from the preset queue. The second device obtains the second feature of the tensor corresponding to the frame stack objects (S302). Specifically, the second device includes a native interpreter. The native interpreter can parse the first frame stack object and obtain the second feature of the tensor corresponding to the first frame stack object. The second device obtains the tensor corresponding to the frame stack object based on the first and second features of the tensor corresponding to the frame stack object (S303). For example, the second device updates the memory address corresponding to the first feature of the tensor corresponding to the first frame stack object output by the first device to the memory address corresponding to the second feature of the tensor corresponding to the first frame stack object based on the first and second features of the tensor corresponding to the first frame stack object, thereby determining the tensor corresponding to the first frame stack object. The first feature includes the shape of the tensor; the second feature includes the data type and data memory address of the tensor. The tensor corresponding to the first frame stack object includes shape, data type, and data memory address. The second device sends the operator task corresponding to the frame stack object to the third device (S304). Specifically, the second device also includes a dispatcher. The dispatcher is used to determine the specific function corresponding to the operator task to be executed based on the feature information of the tensor. Taking the first frame stack object as an example, the dispatcher obtains the dispatch identifier based on the tensor corresponding to the first frame stack object. Based on the dispatch identifier and the dispatch table, the dispatcher determines the specific function of the first operator task corresponding to the first frame stack object to be executed, thereby determining the execution code of the first operator task corresponding to the first frame stack object, and sends the execution code to the third device. The dispatch table includes the dispatch identifier and the pointer to the corresponding function. After the second device sends the first frame stack object to the third device, the second device obtains the first feature of the second frame stack object and the tensor corresponding to the second frame stack object from the preset queue, and performs the sending and dispatching processing. The second frame stack object corresponds to the second operator task, and the second operator task is triggered after the first operator task. The third device includes a graphics processing unit (GPU) and a code converter corresponding to the GPU, or an embedded neural network processor (NPU) and a code converter corresponding to the NPU. Taking the third device as a GPU and the corresponding code converter as an example, the third device obtains the execution code issued by the second device, converts the execution code into device code that the GPU can recognize through the code converter corresponding to the GPU, and executes it through the GPU, thereby realizing the execution of the first operator task.
[0095] In summary, this application provides a method for issuing operator tasks, which improves the efficiency of operator task issuance and further enhances the computing power utilization of the execution device. The method is applied to the PyTorch framework, which includes a first device, a second device, and a third device. The method includes: the first device acquiring a first feature of a tensor corresponding to a first frame stack object; the first device adding the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue. The first frame stack object corresponds to a first operator task. If there are unprocessed frame stack objects in the preset queue, the second device acquires a second frame stack object and the first feature of the tensor corresponding to the second frame stack object from the preset queue, and acquires a second feature of the tensor corresponding to the second frame stack object; the second device obtains the tensor corresponding to the second frame stack object based on the first and second features of the tensor corresponding to the second frame stack object, and issues the second operator task to the third device. The second frame stack object corresponds to a second operator task. In this embodiment, the first device and the second device can operate simultaneously. The first device adds each frame stack object and the first feature of the tensor corresponding to each frame stack object to a preset queue according to the order in which the operator tasks are triggered. The second device retrieves and processes each frame stack object to be processed from the preset queue, obtaining the second feature of the tensor corresponding to the second frame stack object. By using the first and second devices, the operator task distribution process is segmented and processed asynchronously, shortening the operator task distribution time, improving distribution efficiency, and avoiding the waste of computing power caused by the execution device waiting for the operator distribution to complete, thereby improving the computing power utilization rate of the execution device.
[0096] The above mainly describes the solution of the embodiments of this application from a methodological perspective. It can be understood that the first device is designed to achieve the above... Figure 7 The functions described herein include at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art will readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0097] This application embodiment can divide the first device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.
[0098] For example, Figure 10 A schematic diagram of a first device structure provided in an embodiment of this application is shown. Figure 10 As shown, the first device can be applied in a computing device, and the first device 800 includes:
[0099] The acquisition module 810 is used to acquire the first feature of the tensor corresponding to the first frame stack object; the first frame stack object corresponds to the first operator task.
[0100] The processing module 820 is used to add the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue.
[0101] In one possible implementation, the acquisition module 810 is further configured to acquire a first feature of the tensor corresponding to the second frame stack object; the second frame stack object corresponds to the second operator task.
[0102] In one possible implementation, the acquisition module 810 is further configured to acquire the first feature of the tensor corresponding to the third frame stack object; the third frame stack object corresponds to the third operator task; the third operator task is triggered after the first operator task;
[0103] In one possible implementation, the acquisition module 810 is also used to parse the first frame stack object to obtain the first information; and to determine the first feature corresponding to the first frame stack object according to the preset mapping relationship.
[0104] In one possible implementation, the processing module 820 is further configured to add the first feature of the second frame stack object and the tensor corresponding to the second frame stack object to a preset queue.
[0105] In one possible implementation, the processing module 820 is further configured to add the first feature of the third frame stack object and the tensor corresponding to the third frame stack object to a preset queue.
[0106] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the operations described in the above embodiments and various feasible implementation methods.
[0107] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the operations of any of the above-described embodiments and various feasible implementation methods.
[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0109] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0110] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions, which, when loaded and executed on a computer, generate all or part of the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one network site, computer, server, or data center to another network site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or it can be a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape, etc.), an optical medium (e.g., DVD, etc.), or a semiconductor medium (e.g., solid-state drive), etc.
[0111] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for issuing operator tasks, characterized in that, The method is applied to the PyTorch framework, which includes a first device, a second device, and a third device. The method includes: The first device acquires a first feature of the tensor corresponding to the first frame stack object; the first frame stack object corresponds to the first operator task. The first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue; When there is a frame stack object to be processed in the preset queue, the second device obtains a second frame stack object and a first feature of the tensor corresponding to the second frame stack object from the preset queue, and obtains a second feature of the tensor corresponding to the second frame stack object; the second frame stack object corresponds to the second operator task. The second device obtains the tensor corresponding to the second frame stack object based on the first and second features of the tensor corresponding to the second frame stack object, and sends the second operator task to the third device.
2. The method according to claim 1, characterized in that, The first feature includes the shape of the tensor; the second feature includes the data type and memory address of the tensor.
3. The method according to claim 1 or 2, characterized in that, When the first frame stack object and the second frame stack object are different frame stack objects, before the second device obtains the second frame stack object and the first feature of the tensor corresponding to the second frame stack object from the preset queue, the method further includes: The first device acquires a first feature of the tensor corresponding to the second frame stack object; the second frame stack object corresponds to the second operator task. The first device adds the second frame stack object and the first feature of the tensor corresponding to the second frame stack object to the preset queue.
4. The method according to claim 1 or 2, characterized in that, After the first device adds the first frame stack object and the first feature of the tensor corresponding to the first frame stack object to a preset queue, the method further includes: The first device acquires a first feature of the tensor corresponding to the third frame stack object; the third frame stack object corresponds to a third operator task; the third operator task is triggered after the first operator task; The first device adds the third frame stack object and the first feature of the tensor corresponding to the third frame stack object to the preset queue.
5. The method according to any one of claims 1-4, characterized in that, The first device acquires the first feature of the tensor corresponding to the first frame stack object, including: The first device parses the first frame stack object to obtain the first information; The first device determines the first feature of the tensor corresponding to the first frame stack object according to a preset mapping relationship; The preset mapping relationship includes the mapping relationship between the first information in the frame stack object and the first feature of the tensor corresponding to the frame stack object.
6. The method according to any one of claims 1-5, characterized in that, The second device includes the native interpreter: The native interpreter parses the first feature of the tensor corresponding to the second frame stack object and obtains the second feature of the tensor corresponding to the second frame stack object from the preset queue; The native interpreter is also used to obtain the tensor corresponding to the second frame stack object based on the first and second features of the tensor corresponding to the second frame stack object.
7. The method according to claim 6, characterized in that, The second device also includes a dispenser: The dispatcher is used to obtain a dispatch identifier based on the second frame stack object and the tensor corresponding to the second frame stack object; and to determine the execution code of the second operator task corresponding to the second frame stack object based on the dispatch identifier. The execution code of the second operator task corresponding to the second frame stack object is then sent to the third device.
8. The method according to any one of claims 1-6, characterized in that, The third device includes a graphics processing unit (GPU) and a corresponding code converter for the GPU, or an embedded neural network processing unit (NPU) and a corresponding code converter for the NPU. The code converter corresponding to the GPU is used to convert the executable code into code adapted to the GPU; the code converter corresponding to the NPU is used to convert the executable code into code adapted to the NPU.
9. A computing device, characterized in that, The computing device includes: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instructions, causing the computing device to perform the operator task dispatching method as described in any one of claims 1-8.
10. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computing device, cause the computing device to perform the method of issuing operator tasks as described in any one of claims 1-8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer program instructions, which, when executed by a computing device, enable the computing device to perform the method of issuing operator tasks as described in any one of claims 1-8.