Operator execution method and device, equipment and medium
By storing and generating operator binary files when creating new operator streams in the deep learning processor, and packaging and sending operator groups to the driver module when conditions are met, the problem of long operator execution time is solved, and the performance of deep learning models is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGHAO XINYING (HANGZHOU) TECH CO LTD
- Filing Date
- 2025-10-24
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, operator execution schemes require the transmission of multiple operators separately, resulting in a large number of interactions between the driving modules, which takes a long time and reduces the performance of deep learning models.
When creating a new operator stream in the deep learning processor, the matching operator queue and file cache are determined, the operator is stored, and the operator binary file is generated. When obtaining an operator stream call request, it is checked whether the packaged execution conditions are met, and if the conditions are met, the operator group is sent to the driver module for execution.
By sending the operator group to the driver module at once, the number of interactions between the driver module and the driver module is reduced, the execution time of the operator is reduced, and the performance of the deep learning model is improved.
Smart Images
Figure CN122019068A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to an operator execution method, apparatus, device, and medium. Background Technology
[0002] With the development of artificial intelligence technology, more and more companies are starting to use deep learning models for image processing, text processing, or speech processing. When a deep learning model needs to complete a specific computational task during its operation, it typically calls the deep learning processor to execute the various operators in the operator stream corresponding to the specified computational task, thereby completing the specified computational task.
[0003] In related technologies, a common operator execution scheme involves sending each operator in the operator stream corresponding to a specified computational task to the driver module of the deep learning processor, and then executing each operator in the operator stream corresponding to the specified computational task through the driver module. This operator execution scheme requires the separate transmission of multiple operators, resulting in numerous interactions between the driver module and a prolonged execution time, thus degrading the performance of the deep learning model. Summary of the Invention
[0004] This invention provides an operator execution method, apparatus, device, and medium to solve the problem that in related technologies, operator execution schemes require the separate transmission of multiple operators, resulting in numerous interactions between driving modules, which leads to long execution times and reduced performance of deep learning models.
[0005] According to one aspect of the present invention, an operator execution method is provided, comprising:
[0006] When a new operator stream is detected in the deep learning processor, an operator queue and file cache matching the new operator stream are determined, each operator in the new operator stream is stored in the operator queue, and the operator binary file of the new operator stream is stored in the file cache.
[0007] When an operator stream call request is received, it is checked whether the target operator stream of the operator stream call request meets the packaged execution conditions. After determining that the target operator stream meets the packaged execution conditions, the operators in the operator queue that match the target operator stream are packaged to obtain an operator group. The operator group is then sent to the driver module so that the driver module can execute each operator in the target operator stream.
[0008] According to another aspect of the present invention, an operator execution apparatus is provided, comprising:
[0009] The operator storage unit is used to determine the operator queue and file cache that match the new operator stream when a new operator stream is detected in the deep learning processor, store each operator in the new operator stream in the operator queue, and store the operator binary file of the new operator stream in the file cache.
[0010] The operator execution unit is used to detect whether the target operator stream of the operator stream call request meets the packaging execution conditions when an operator stream call request is obtained. After determining that the target operator stream meets the packaging execution conditions, the unit packages the operators in the operator queue that match the target operator stream to obtain an operator group and sends the operator group to the driver module so that the driver module executes each operator in the target operator stream.
[0011] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0012] At least one processor;
[0013] and a memory communicatively connected to the at least one processor;
[0014] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to execute the operator execution method according to any embodiment of the present invention.
[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute the operator execution method according to any embodiment of the present invention.
[0016] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the operator execution method described in any embodiment of the present invention.
[0017] The technical solution of this invention, when a new operator stream is detected in a deep learning processor, determines the operator queue and file cache matching the new operator stream, stores each operator in the new operator stream in the operator queue, and stores the operator binary file of the new operator stream in the file cache; when an operator stream call request is obtained, it checks whether the target operator stream of the operator stream call request meets the packaging and execution conditions. After determining that the target operator stream meets the packaging and execution conditions, it packages the operators in the operator queue matching the target operator stream to obtain an operator group, and sends the operator group to the driver module so that the driver module executes each operator in the target operator stream. This solves the problem of operator in related technologies. The current execution scheme requires the transmission of multiple operators separately, resulting in numerous interactions between the driver module and prolonged operator execution time, thus degrading the performance of the deep learning model. To address this, when each operator stream is created, a queue of operators matching the stream is used to store each operator. Once the conditions for packaged execution are met, the queue automatically packages the operators from the required operator streams into a single operator group and sends it to the driver module. This allows all operators from the required operator streams to be sent to the driver module at once, effectively reducing the number of interactions between the driver module, decreasing operator execution time, and improving the performance of the deep learning model.
[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart of an operator execution method provided in Embodiment 1 of the present invention.
[0021] Figure 2 This is a schematic diagram of an operator execution process provided in Embodiment 1 of the present invention.
[0022] Figure 3 This is a flowchart of an operator execution method provided in Embodiment 2 of the present invention.
[0023] Figure 4 This is a schematic diagram of an operator execution device provided in Embodiment 3 of the present invention.
[0024] Figure 5 A schematic diagram of the structure of an electronic device for implementing the operator execution method of this embodiment of the invention. Detailed Implementation
[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0026] It should be noted that the terms "target," "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising," "including," and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0027] Example 1
[0028] Figure 1 This is a flowchart illustrating an operator execution method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where a deep learning processor is invoked to execute various operators in an operator stream corresponding to a specified computational task, thereby completing the specified computational task. This method can be executed by an operator execution device, which can be implemented in hardware and / or software, and can be configured within a deep learning processor set up in an enterprise. Figure 1 As shown, the method includes:
[0029] Step 101: When a new operator stream is detected in the deep learning processor, determine the operator queue and file cache that match the new operator stream, store each operator in the new operator stream into the operator queue, and store the operator binary file of the new operator stream into the file cache.
[0030] Optionally, the deep learning processor can be a processor used to handle the computational tasks required during the execution of the target deep learning model. For example, the deep learning processor can be a Tensor Processing Unit (TPU), a Neural Processing Unit (NPU), or a Graphics Processing Unit (GPU) used to handle the computational tasks required during the execution of the target deep learning model. The target deep learning model can be a deep learning model used for image processing, text processing, or speech processing. The computational task can be a data computation process required by the target deep learning model during image processing, text processing, or speech processing. A computational task consists of multiple data computation operations. Each data computation operation can be an operation that performs computation based on specified data. Whenever a new computational task is generated, the deep learning processor creates an operator stream corresponding to the computational task and sets the identification information of the operator stream corresponding to the computational task. The identification information of the operator stream can be information used to uniquely identify the operator stream. The operator stream corresponding to the computational task contains multiple operators. Each operator can be a program that, after execution, is used to complete a portion of the data computation operations in the computational task. For each computational task, the task can be completed by executing the operators in the operator stream corresponding to the task. When a specific computational task needs to be completed during the execution of the target deep learning model, the deep learning processor can be invoked to execute the operators in the operator stream corresponding to the specified computational task, thereby completing the specified computational task.
[0031] Optionally, the operator queue matching the operator stream can be a queue within the deep learning processor used to store the individual operators in the operator stream and information related to the operator stream. The file cache matching the operator stream can be a cache within the deep learning processor used to store the operator binary files of the operator stream. The operator binary files of the operator stream can be files generated during the creation of the operator stream, storing the individual operators in the operator stream in binary format.
[0032] Optionally, a new operator stream can refer to an operator stream created in the deep learning processor corresponding to a new computational task when a new computational task is generated. The system can detect whether a new operator stream is being created in the deep learning processor. Each time a new operator stream is detected, the system determines the operator queue and file cache matching the new operator stream, stores each operator in the new operator stream in the matching operator queue, and stores the operator binary file of the new operator stream in the matching file cache.
[0033] Optionally, determining the operator queue and file cache that match the new operator stream, storing each operator in the new operator stream into the operator queue, and storing the operator binary file of the new operator stream into the file cache includes: creating an operator queue that matches the new operator stream according to the dynamic queue length, storing each operator in the new operator stream into the operator queue; selecting an available cache from various available caches as the file cache that matches the new operator stream, and storing the operator binary file of the new operator stream into the file cache.
[0034] Optionally, the dynamic queue length can be the length of the operator queue matching the operator stream set in the deep learning processor. A queue with a length equal to the dynamic queue length can be created, its identification information set, and the created queue designated as the operator queue matching the new operator stream. The identification information of the new operator stream and the identification information of the operator queue matching the new operator stream are stored correspondingly in a queue information file. Then, each operator in the new operator stream is stored in the operator queue matching the new operator stream. The queue identification information can be information used to uniquely identify the queue. The queue information file can be a pre-set file used to store the identification information of each operator stream and the operator queue matching each operator stream. By storing the identification information of the new operator stream and the identification information of the operator queue matching the new operator stream in the queue information file, a binding relationship is established between the new operator stream and the operator queue matching the new operator stream.
[0035] Optionally, this also includes adjusting the dynamic queue length during the execution of the target deep learning model. Typically, setting a smaller dynamic queue length in the initial stage of the target deep learning model's execution ensures timely operator delivery, preventing the deep learning processor from idling and thus avoiding performance degradation. Setting a larger dynamic queue length in the later stages of the target deep learning model's execution can optimize its performance.
[0036] Optionally, during the execution of the target deep learning model, the dynamic queue length is adjusted, including: initially setting the dynamic queue length to a first length; wherein the first length is the length of the operator queue matching the operator stream suitable for use in the initial stage of the target deep learning model's execution; each time the target deep learning model completes a processing operation, the dynamic queue length is increased by a preset multiple until the dynamic queue length increases to a second length; wherein the second length is the length of the operator queue matching the operator stream suitable for use in the later stage of the target deep learning model's execution, and the second length is greater than the first length. The target deep learning model completing a processing operation can refer to the target deep learning model completing an image processing operation, a text processing operation, or a speech processing operation.
[0037] Optionally, each available cache can refer to multiple idle caches that can be used in the deep learning processor. An available cache can be randomly selected from these caches as the file cache matching the new operator stream. The identifier information of the new operator stream and the identifier information of the file cache matching the new operator stream are stored in a cache information file. Then, the operator binary file of the new operator stream is stored in the file cache. The cache identifier information can be pre-defined information used to uniquely identify the cache. The cache information file can be a pre-defined file used to store the identifier information of each operator stream and the file cache matching each operator stream. By storing the identifier information of the new operator stream and the identifier information of the file cache matching the new operator stream in the cache information file, a binding relationship is established between the new operator stream and the file cache matching the new operator stream.
[0038] Step 102: When an operator stream call request is obtained, it is checked whether the target operator stream of the operator stream call request meets the packaged execution conditions. After determining that the target operator stream meets the packaged execution conditions, the operators in the operator queue that match the target operator stream are packaged to obtain an operator group. The operator group is sent to the driver module so that the driver module can execute each operator in the target operator stream.
[0039] Optionally, an operator stream invocation request can be information used to invoke a deep learning processor to execute the various operators in a specified operator stream, thereby completing the computational task corresponding to the specified operator stream. The target operator stream of the operator stream invocation request is the specified operator stream that the deep learning processor is invoked to execute. The target operator stream can be any already created operator stream. The operator stream invocation request contains the identification information of the target operator stream.
[0040] Optionally, the target deep learning model can send operator stream call requests to the runtime interface of the deep learning processor. The runtime interface can be an interface set up in the deep learning processor for processing call information sent by the target deep learning model. It can detect whether the runtime interface has received an operator stream call request from the target deep learning model. Each time the runtime interface detects that it has received such a request, it retrieves the received request. When an operator stream call request is retrieved, it checks whether the target operator stream of the request meets the conditions for packaged execution. If the target operator stream meets the conditions, it packages the operators in the operator queue matching the target operator stream to obtain an operator group, and sends the operator group to the driver module so that the driver module executes each operator in the target operator stream. The driver module can be a software or hardware module set up in the deep learning processor for executing operator streams.
[0041] Optionally, the packaging execution condition is used to determine whether the operator stream has any anomalies and whether the individual operators in the operator stream can be packaged and sent to the driver module for execution. The packaging execution condition can be that the operator binary file of the operator stream is cached in the file cache matching the operator stream and the operator queue matching the operator stream is full. When the operator binary file of the operator stream is cached in the file cache matching the operator stream and the operator queue matching the operator stream is full, i.e., the operator stream meets the packaging execution condition, it can be determined that the operator stream has no anomalies, and the individual operators in the operator stream can be packaged and sent to the driver module for execution. When the operator binary file of the operator stream is not cached in the file cache matching the operator stream and the operator queue matching the operator stream is full, i.e., the operator stream does not meet the packaging execution condition, it can be determined that the operator stream has anomalies, and the individual operators in the operator stream cannot be packaged and sent to the driver module for execution.
[0042] Optionally, "the operator binary file of the operator stream is cached in the file cache matching the operator stream" can mean that the operator binary file of the operator stream is already stored in the file cache matching the operator stream. "The operator queue matching the operator stream is full" can mean that the total number of operators stored in the operator queue matching the operator stream equals the total number of operators in the operator stream. The total number of operators in the operator stream can be the total number of operators in the operator stream that is counted and stored after the operator stream is created. The identifier information of the operator stream and the total number of operators are stored correspondingly in a statistics file. The statistics file can be a pre-set file used to store the identifier information of the operator stream and the total number of operators.
[0043] Optionally, detecting whether the target operator stream of the operator stream call request meets the conditions for packaged execution includes: querying whether the operator binary file of the target operator stream of the operator stream call request is cached in the file cache matching the target operator stream; if it is determined that the operator binary file of the target operator stream is cached in the file cache matching the target operator stream, then checking whether the operator queue matching the target operator stream is full; if it is determined that the operator queue matching the target operator stream is full, then determining that the target operator stream meets the conditions for packaged execution.
[0044] Optionally, detecting whether the target operator stream of the operator stream call request meets the packaged execution conditions further includes: if it is determined that the operator binary file of the target operator stream is not cached in the file cache matching the target operator stream, then the target operator stream does not meet the packaged execution conditions; if it is determined that the operator queue matching the target operator stream is not full, then the target operator stream does not meet the packaged execution conditions.
[0045] Optionally, the target operator stream can be determined based on the identifier information of the target operator stream in the operator stream call request. The identifier information of the target operator stream can be used to retrieve the identifier information of the operator queue matching the target operator stream from the queue information file, and the matching operator queue can be determined based on this identifier information. The identifier information of the file cache matching the target operator stream can be used to retrieve the identifier information of the file cache matching the target operator stream from the cache information file, and the matching file cache can be determined based on this identifier information. The total number of operators in the target operator stream can be obtained from the statistics information file based on the identifier information of the target operator stream.
[0046] Optionally, it can be detected whether the operator binary file of the target operator stream is already stored in the file cache matching the target operator stream. If it is detected that the operator binary file of the target operator stream is already stored in the file cache matching the target operator stream, then it is determined that the operator binary file of the target operator stream is cached in the file cache matching the target operator stream. If it is detected that the operator binary file of the target operator stream is not stored in the file cache matching the target operator stream, then it is determined that the operator binary file of the target operator stream is not cached in the file cache matching the target operator stream.
[0047] Optionally, it can be detected whether the total number of operators stored in the operator queue matching the target operator stream is equal to the total number of operators in the target operator stream. If it is detected that the total number of operators stored in the operator queue matching the target operator stream is equal to the total number of operators in the target operator stream, then it is determined that the operator queue matching the target operator stream is full. If it is detected that the total number of operators stored in the operator queue matching the target operator stream is not equal to the total number of operators in the target operator stream, then it is determined that the operator queue matching the target operator stream is not full.
[0048] Optionally, when the target operator stream meets the packaged execution conditions, the operators in the operator queue matching the target operator stream are the individual operators in the target operator stream. After determining that the target operator stream meets the packaged execution conditions, the operators in the operator queue matching the target operator stream can be packaged to obtain an operator group, which is then sent to the driver module. An operator group can be a data packet obtained by packaging the operators in the operator queue matching the target operator stream after determining that the target operator stream meets the packaged execution conditions. The operator group contains all the operators in the target operator stream. Sending the operator group to the driver module sends all the operators in the target operator stream to the driver module at once. After receiving the operator group, the driver module can store each operator in the operator group in the underlying queue, and then read and execute each operator in the operator group through the on-chip scheduler, thereby executing each operator in the target operator stream. The underlying queue can be a queue set in the driver module for storing the operators to be executed. The on-chip scheduler can be a software module set in the driver module for executing operators.
[0049] Optionally, after sending the operator group to the driver module, the method further includes: generating a call record information based on the configuration information of the target operator stream and the operator parameters in the operator stream call request, and storing the call record information in the operator queue matching the target operator stream. The configuration information of the target operator stream may refer to information stored in the deep learning processor used to configure the target operator stream. The configuration information of the target operator stream may include the identifier information of the operator stream, the identifier information of the operator queue matching the target operator stream, and the identifier information of the file cache matching the target operator stream. The operator parameters in the operator stream call request may refer to information related to the operators in the target operator stream in the operator stream call request. The operator stream call request includes the identifier information of the target operator stream, the operator identifier, and the operator parameters. The operator identifier may be information used to identify the operators in the target operator stream. The operator parameters may be information used to describe the operators in the target operator stream. The call record information can be used to record information about the process of calling the deep learning processor to execute each operator in the operator stream.
[0050] Optionally, the configuration information of the target operator stream and the operator parameters in the operator stream call request can be determined as a call record information for recording the current call process, and the call record information can be stored in the operator queue that matches the target operator stream.
[0051] Optionally, it also includes: when multiple operator stream call requests are obtained, determining the call order of each operator stream call request; according to the call order of each target operator stream, sequentially checking whether the target operator stream of each operator stream call request meets the packaging execution condition; after determining that the target operator stream meets the packaging execution condition, packaging the operators in the operator queue that match the target operator stream to obtain an operator group, and sending the operator group to the driver module so that the driver module executes each operator in the target operator stream.
[0052] Optionally, multiple operator stream call requests can be a group of operator stream call requests that need to be processed sequentially. The calling order of each operator stream call request refers to the order in which they are processed. Each operator stream call request contains a numerical number. The numerical numbers in different operator stream call requests are different. The order of the operator stream call requests is the order in which they are arranged according to their numerical numbers in ascending order. The operator stream call request that is first in the queue needs to be processed first. The operator stream call request that is second in the queue needs to be processed second. And so on, with the operator stream call request that is last in the queue needing to be processed last.
[0053] Optionally, when multiple operator stream call requests are received, the requests are arranged in ascending order of their numerical numbers to obtain the call order. Then, starting with the first operator stream call request, processing operations are performed on each request sequentially according to the call order of the target operator streams, until the last request is processed. The processing operations for each operator stream call request include: detecting whether the target operator stream of the call request meets the packaging and execution conditions; after determining that the target operator stream meets the packaging and execution conditions, packaging the operators in the operator queue matching the target operator stream to obtain an operator group; and sending the operator group to the driver module so that the driver module executes each operator in the target operator stream.
[0054] Optionally, after determining that the target operator stream does not meet the conditions for packaged execution, an exception message is sent to the target user. The exception message may indicate that the operator stream has an error and cannot be packaged and sent to the driver module for execution. The target user may be a technician handling exceptions in the deep learning processor.
[0055] The technical solution of this invention, when a new operator stream is detected in a deep learning processor, determines the operator queue and file cache matching the new operator stream, stores each operator in the new operator stream in the operator queue, and stores the operator binary file of the new operator stream in the file cache; when an operator stream call request is obtained, it checks whether the target operator stream of the operator stream call request meets the packaging and execution conditions. After determining that the target operator stream meets the packaging and execution conditions, it packages the operators in the operator queue matching the target operator stream to obtain an operator group, and sends the operator group to the driver module so that the driver module executes each operator in the target operator stream. This solves the problem of operator in related technologies. The current execution scheme requires the transmission of multiple operators separately, resulting in numerous interactions between the driver module and prolonged operator execution time, thus degrading the performance of the deep learning model. To address this, when each operator stream is created, a queue of operators matching the stream is used to store each operator. Once the conditions for packaged execution are met, the queue automatically packages the operators from the required operator streams into a single operator group and sends it to the driver module. This allows all operators from the required operator streams to be sent to the driver module at once, effectively reducing the number of interactions between the driver module, decreasing operator execution time, and improving the performance of the deep learning model.
[0056] The technical solution of this invention significantly optimizes the overhead of calling the deep learning processor to execute each operator in the operator stream by packaging each operator in the operator stream into an operator group and sending it to the driver module at once. At the same time, it reduces the number of interactions with the driver module and comprehensively improves the overall performance of the deep learning framework based on the deep learning processor.
[0057] Optionally, in a specific instance, Figure 2This is a schematic diagram of an operator execution process provided in Embodiment 1 of the present invention. The operator execution process is divided into a process completed on the user side and a process completed on the driver side. The process completed on the user side includes: the target deep learning model sending an operator stream call request to the runtime interface of the deep learning processor; obtaining the operator stream call request received by the runtime interface; after determining that the target operator stream of the operator stream call request meets the packaged execution conditions, packaging the operators in the operator queue that match the target operator stream to obtain an operator group; and sending the operator group to the driver module so that the driver module can execute each operator in the target operator stream. The process completed on the driver side includes: the driver module storing each operator in the operator group in the underlying queue; and performing operator execution processing through the on-chip scheduler. The on-chip scheduler reads each operator in the operator group and executes each operator in the operator group, thereby executing each operator in the target operator stream. The on-chip scheduler can execute the operators in the operator group continuously, which can reduce the number of operator parameter loading and operator binary loading times. The technical solution of this invention reduces operator delivery overhead and optimizes the processing logic of the on-chip scheduler by packaging each operator in the operator stream into an operator group and sending it to the driver module at once.
[0058] Example 2
[0059] Figure 3 This is a flowchart illustrating an operator execution method provided in Embodiment 2 of the present invention. Embodiments of the present invention can be combined with various optional solutions from one or more of the above embodiments. For example... Figure 3 As shown, the method includes:
[0060] Step 201: When a new operator stream is detected in the deep learning processor, determine the operator queue and file cache that match the new operator stream, store each operator in the new operator stream into the operator queue, and store the operator binary file of the new operator stream into the file cache.
[0061] Step 202: When multiple operator stream call requests are obtained, the call order of each operator stream call request is determined. According to the call order of each target operator stream, the target operator stream of each operator stream call request is checked in turn to see if it meets the packaged execution conditions. After it is determined that the target operator stream meets the packaged execution conditions, the operators in the operator queue that match the target operator stream are packaged to obtain an operator group. The operator group is sent to the driver module so that the driver module can execute each operator in the target operator stream.
[0062] The technical solution of this invention can automatically package each operator in the operator stream into an operator group and send it to the driver module after determining that each operator stream meets the package execution conditions according to the calling order of each operator stream. This allows all operators in the operator stream to be executed to be sent to the driver module at once, which can effectively reduce the number of interactions with the driver module, reduce the time spent in the operator execution process, and improve the performance of the deep learning model.
[0063] Example 3
[0064] Figure 4 This is a schematic diagram of an operator execution device provided in Embodiment 3 of the present invention. The device can be configured in an electronic device. Figure 4 As shown, the device includes an operator storage unit 301 and an operator execution unit 302.
[0065] The operator storage unit 301 is used to determine the operator queue and file cache that match the new operator stream when a new operator stream is detected in the deep learning processor, store each operator in the new operator stream in the operator queue, and store the operator binary file of the new operator stream in the file cache. The operator execution unit 302 is used to detect whether the target operator stream of the operator stream call request meets the packaged execution conditions when an operator stream call request is obtained, and after determining that the target operator stream meets the packaged execution conditions, package the operators in the operator queue that match the target operator stream to obtain an operator group, and send the operator group to the driver module so that the driver module executes each operator in the target operator stream.
[0066] The technical solution of this invention, when a new operator stream is detected in a deep learning processor, determines the operator queue and file cache matching the new operator stream, stores each operator in the new operator stream in the operator queue, and stores the operator binary file of the new operator stream in the file cache; when an operator stream call request is obtained, it checks whether the target operator stream of the operator stream call request meets the packaging and execution conditions. After determining that the target operator stream meets the packaging and execution conditions, it packages the operators in the operator queue matching the target operator stream to obtain an operator group, and sends the operator group to the driver module so that the driver module executes each operator in the target operator stream. This solves the problem of operator in related technologies. The current execution scheme requires the transmission of multiple operators separately, resulting in numerous interactions between the driver module and prolonged operator execution time, thus degrading the performance of the deep learning model. To address this, when each operator stream is created, a queue of operators matching the stream is used to store each operator. Once the conditions for packaged execution are met, the queue automatically packages the operators from the required operator streams into a single operator group and sends it to the driver module. This allows all operators from the required operator streams to be sent to the driver module at once, effectively reducing the number of interactions between the driver module, decreasing operator execution time, and improving the performance of the deep learning model.
[0067] In an optional embodiment of the present invention, the operator storage unit 301, when performing the operations of determining the operator queue and file cache matching the new operator stream, storing each operator in the new operator stream into the operator queue, and storing the operator binary file of the new operator stream into the file cache, is specifically configured to: create an operator queue matching the new operator stream according to the dynamic queue length, store each operator in the new operator stream into the operator queue; select an available cache from each available cache as the file cache matching the new operator stream, and store the operator binary file of the new operator stream into the file cache.
[0068] In an optional embodiment of the present invention, the operator execution unit 302, when performing the operation of detecting whether the target operator stream of the operator stream call request meets the package execution conditions, is specifically configured to: query whether the operator binary file of the target operator stream of the operator stream call request is cached in the file cache matching the target operator stream; if it is determined that the operator binary file of the target operator stream is cached in the file cache matching the target operator stream, then check whether the operator queue matching the target operator stream is full; if it is determined that the operator queue matching the target operator stream is full, then determine that the target operator stream meets the package execution conditions.
[0069] In an optional embodiment of the present invention, the operator execution unit 302 may be further configured to: determine that the target operator stream does not meet the package execution conditions if it is determined that the operator binary file of the target operator stream is not cached in the file cache matching the target operator stream; and determine that the target operator stream does not meet the package execution conditions if it is determined that the operator queue matching the target operator stream is not full.
[0070] In an optional embodiment of the present invention, the operator execution unit 302 may be further configured to: generate a call record information based on the configuration information of the target operator stream and the operator parameters in the operator stream call request, and store the call record information in an operator queue that matches the target operator stream.
[0071] In an optional embodiment of the present invention, the operator execution device may further include: a sequential execution unit, configured to determine the calling order of each operator stream calling request when multiple operator stream calling requests are obtained, and according to the calling order of each target operator stream, sequentially detect whether the target operator stream of each operator stream calling request meets the packaged execution condition; after determining that the target operator stream meets the packaged execution condition, package the operators in the operator queue that match the target operator stream to obtain an operator group, and send the operator group to the driver module so that the driver module executes each operator in the target operator stream.
[0072] In an optional embodiment of the present invention, the operator execution device may further include: a length adjustment unit, used to adjust the dynamic queue length during the operation of the target deep learning model.
[0073] The operator execution device provided in the embodiments of the present invention can execute the operator execution method provided in any embodiment of the present invention, and has the corresponding functional units and beneficial effects of the execution method.
[0074] Example 4
[0075] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement the operator execution method of embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0076] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory 12 or a random access memory 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 12 or loaded from storage unit 18 into the random access memory 13. The random access memory 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, read-only memory 12, and random access memory 13 are interconnected via a bus 14. An input / output interface 15 is also connected to the bus 14.
[0077] Multiple components in electronic device 10 are connected to input / output interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0078] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as operator execution methods.
[0079] In some embodiments, the operator execution method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via read-only memory and / or a communication unit. When the computer program is loaded into random access memory and executed by a processor, one or more steps of the operator execution method described above may be performed. Alternatively, in other embodiments, the processor may be configured to execute the operator execution method by any other suitable means (e.g., by means of firmware).
[0080] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices (PLCs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0081] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or electronic device.
[0082] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0083] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).
[0084] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data electronic devices), or computing systems that include middleware components (e.g., application electronic devices), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0085] A computing system can include clients and electronic devices. Clients and electronic devices are generally geographically separated and typically interact via communication networks. The client-electronic device relationship is created by computer programs running on the respective computers and establishing a client-electronic device relationship between them. Electronic devices can be cloud electronic devices, also known as cloud computing electronic devices or cloud servers, which are host products within the cloud computing service system. These address the shortcomings of traditional physical hosts and virtual private server services, such as high management difficulty and weak business scalability.
[0086] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0087] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. An operator execution method, characterized in that, include: When a new operator stream is detected in the deep learning processor, an operator queue and file cache matching the new operator stream are determined, each operator in the new operator stream is stored in the operator queue, and the operator binary file of the new operator stream is stored in the file cache. When an operator stream call request is received, it is checked whether the target operator stream of the operator stream call request meets the packaged execution conditions. After determining that the target operator stream meets the packaged execution conditions, the operators in the operator queue that match the target operator stream are packaged to obtain an operator group. The operator group is then sent to the driver module so that the driver module can execute each operator in the target operator stream.
2. The operator execution method according to claim 1, characterized in that, Determine the operator queue and file cache that match the new operator stream, store each operator in the new operator stream into the operator queue, and store the operator binary file of the new operator stream into the file cache, including: Based on the dynamic queue length, create an operator queue that matches the new operator stream, and store each operator in the new operator stream into the operator queue; Select one available cache from the available caches as the file cache that matches the new operator stream, and store the operator binary file of the new operator stream into the file cache.
3. The operator execution method according to claim 1, characterized in that, Detecting whether the target operator stream of the operator stream invocation request meets the conditions for packaged execution includes: Check whether the operator binary file of the target operator stream in the operator stream call request is cached in the file cache that matches the target operator stream; If it is determined that the operator binary file of the target operator stream is cached in the file cache that matches the target operator stream, then check whether the operator queue that matches the target operator stream is full; If it is determined that the operator queue matching the target operator stream is full, then the target operator stream is determined to meet the conditions for packaged execution.
4. The operator execution method according to claim 3, characterized in that, Detecting whether the target operator stream of the operator stream invocation request meets the conditions for packaged execution further includes: If it is determined that the operator binary file of the target operator stream is not cached in the file cache that matches the target operator stream, then it is determined that the target operator stream does not meet the conditions for packaged execution. If it is determined that the operator queue matching the target operator stream is not full, then the target operator stream does not meet the conditions for packaged execution.
5. The operator execution method according to claim 1, characterized in that, After sending the operator group to the driver module, the following is also included: Based on the configuration information of the target operator stream and the operator parameters in the operator stream call request, a call record information is generated and stored in the operator queue that matches the target operator stream.
6. The operator execution method according to claim 1, characterized in that, Also includes: When multiple operator stream call requests are received, the call order of each operator stream call request is determined. Based on the call order of each target operator stream, the target operator stream of each operator stream call request is checked in turn to see if it meets the packaged execution conditions. After it is determined that the target operator stream meets the packaged execution conditions, the operators in the operator queue that match the target operator stream are packaged to obtain an operator group. The operator group is then sent to the driver module so that the driver module can execute each operator in the target operator stream.
7. The operator execution method according to claim 1, characterized in that, Also includes: Adjust the dynamic queue length during the execution of the target deep learning model.
8. An operator execution device, characterized in that, include: The operator storage unit is used to determine the operator queue and file cache that match the new operator stream when a new operator stream is detected in the deep learning processor, store each operator in the new operator stream in the operator queue, and store the operator binary file of the new operator stream in the file cache. The operator execution unit is used to detect whether the target operator stream of the operator stream call request meets the packaging execution conditions when an operator stream call request is obtained. After determining that the target operator stream meets the packaging execution conditions, the unit packages the operators in the operator queue that match the target operator stream to obtain an operator group and sends the operator group to the driver module so that the driver module executes each operator in the target operator stream.
9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the operator execution method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the operator execution method according to any one of claims 1-7.