Computing devices and task processing methods

Abstract

This application discloses a computing device and a task processing method, belonging to the field of computer technology. The computing device includes a neural network processor (NPU), a first memory, and a second memory. The NPU is directly connected to the first memory and connected to the second memory via a bus. The first memory stores task data for a first type of task; the second memory stores task data for a second type of task, wherein the processing latency required by the first type of task is lower than that required by the second type of task. The NPU is used to acquire task data from the first memory and task data from the second memory via the bus, and process tasks based on the acquired task data. This computing device can improve the utilization efficiency of the NPU.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a computing device and a task processing method. Background Technology

[0002] NPUs (Neural Network Processing Units) are characterized by high computing speed and low power consumption, making them ideal for processing data such as video and images. In related technologies, computing devices include multiple NPUs, each dedicated to handling different types of tasks. Because some types of tasks are fewer in number, the NPUs used for these tasks may remain idle, resulting in low NPU utilization efficiency. Summary of the Invention

[0003] This application provides a computing device and a task processing method that can improve the utilization efficiency of the NPU in the computing device. The technical solution is as follows:

[0004] According to one aspect of the embodiments of this application, a computing device is provided, the computing device including a neural network processor (NPU), a first memory and a second memory, wherein the NPU is directly connected to the first memory and connected to the second memory via a bus;

[0005] The first memory is used to store task data for the first type of task;

[0006] The second memory is used to store task data for the second type of task, wherein the processing latency required by the first type of task is lower than the processing latency required by the second type of task;

[0007] The NPU is used to acquire task data from the first memory and acquire task data from the second memory via the bus, and process tasks based on the acquired task data.

[0008] According to another aspect of the embodiments of this application, a task processing method is provided, the method being executed by a computing device, the computing device including a neural network processor (NPU), a first memory, and a second memory, the NPU being directly connected to the first memory and connected to the second memory via a bus; the method includes:

[0009] The first memory stores the task data for the first type of task;

[0010] The second memory stores the task data of the second type of task, where the processing latency required by the first type of task is lower than that required by the second type of task.

[0011] The NPU acquires task data from the first memory and acquires task data from the second memory via the bus, and processes the task based on the acquired task data.

[0012] In the computing device provided in this application embodiment, the NPU is not only directly connected to the first memory, but also connected to the second memory through a bus. In this way, the NPU can not only obtain task data of the first type of task from the first memory, but also obtain task data of the second type of task from the second memory through the bus, thereby enabling it to process multiple types of tasks. This avoids the NPU from being idle due to a small number of certain types of tasks, thereby improving the utilization efficiency of the NPU. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This invention provides a schematic diagram of the structure of a computing device according to an exemplary embodiment of the present application.

[0015] Figure 2 This application shows a schematic diagram of the structure of another computing device provided in an exemplary embodiment;

[0016] Figure 3 This illustration shows a process diagram of processing multiple tasks using time-division multiplexing, provided by an exemplary embodiment of this application.

[0017] Figure 4 This application shows a schematic diagram of the structure of an NPU provided in an exemplary embodiment;

[0018] Figure 5 This invention provides a schematic diagram of the structure of a processing unit in an NPU according to an exemplary embodiment of the present application.

[0019] Figure 6 This invention provides a schematic diagram of the structure of another processing unit in an NPU according to an exemplary embodiment of the present application.

[0020] Figure 7 A schematic diagram of a task queue provided in an exemplary embodiment of this application is shown;

[0021] Figure 8 A schematic diagram of a second memory provided in an exemplary embodiment of this application is shown;

[0022] Figure 9 A flowchart of a task processing method provided by an exemplary embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0024] In this article, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0025] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the task data involved in this application was obtained with full authorization.

[0026] Figure 1 This is a schematic diagram of the structure of a computing device 100 provided in an exemplary embodiment of this application. (Reference) Figure 1 The computing device 100 includes an NPU 101, a first memory 102, and a second memory 103. The NPU 101 is directly connected to the first memory 102 and is connected to the second memory 103 via a bus.

[0027] The first memory 102 is used to store task data for a first type of task. The second memory 103 is used to store task data for a second type of task, wherein the processing latency required for the first type of task is lower than that required for the second type of task. The NPU 101 is used to obtain task data from the first memory 102 and task data from the second memory 103 via the bus, and process tasks based on the obtained task data.

[0028] Optionally, the computing device 100 is a terminal, including various types of terminals such as mobile phones, cameras, desktop computers, laptops, and tablets. Optionally, the computing device 100 is a server; this embodiment of the application does not impose any limitations on this.

[0029] The first type of task requires lower processing latency, meaning it has higher real-time requirements. Examples of first-type tasks include image processing and video processing. In contrast, the second type of task requires higher processing latency, meaning it has lower real-time requirements. Examples of second-type tasks include audio processing.

[0030] The task data includes the data to be processed and the processing parameters used to process the data. For example, the task data includes the input data of a neural network and the weight parameters corresponding to multiple operators in the neural network. The input data of the neural network is the data to be processed, and the weight parameters corresponding to the multiple operators in the neural network are the processing parameters for the data to be processed.

[0031] Optionally, NPU 101 acquires task data from the first memory 102 and acquires task data from the second memory 103 via the bus. The implementation of processing tasks based on the acquired task data includes: NPU 101 acquires task data from the first memory 102, processes the task based on the task data, and in response to the completion of the task processing, NPU 101 acquires task data from the second memory 103 via the bus and processes the task based on the task data. Alternatively, NPU 101 acquires task data from the first memory 102 and acquires task data from the second memory 103 via the bus, and processes multiple tasks using a time-division multiplexing approach based on the acquired task data. Of course, NPU 101 can also use other methods to process tasks, and this embodiment does not limit this.

[0032] In the computing device 100 provided in this application embodiment, the NPU 101 is not only directly connected to the first memory 102, but also connected to the second memory 103 through a bus. In this way, the NPU 101 can not only obtain task data of the first type of task from the first memory 102, but also obtain task data of the second type of task from the second memory 103 through the bus, thereby being able to process multiple types of tasks, so as to avoid the NPU 101 being idle due to the small number of certain types of tasks, thereby improving the utilization efficiency of the NPU 101. Furthermore, since the NPU101 is directly connected to the first memory 102 and connected to the second memory 103 via a bus, the NPU101 is more efficient at acquiring data from the first memory 102 than from the second memory 103. Therefore, storing task data for the first type of task with lower processing latency in the first memory 102 and task data for the second type of task with higher processing latency in the second memory 103 ensures that the NPU101 can acquire task data for the first type of task more efficiently, thereby maximizing the processing efficiency of the first type of task and minimizing its processing latency, thus meeting the requirement of lower processing latency for the first type of task.

[0033] Optionally, refer to Figure 2 The computing device 100 further includes a CPU (Central Processing Unit) 104, and the NPU 101 further includes a processing module. The CPU 104 is used to send processing instructions to the NPU 101 to instruct the processing of at least one task. The NPU 101 is used to receive processing instructions through the processing module, and in response to the processing instructions, acquire task data and process the task based on the acquired task data. Optionally, the processing instructions carry information such as the task name, identifier, task type, priority, and storage location of the task data; this embodiment of the application does not limit this.

[0034] Optionally, the NPU101 also includes a scheduler for scheduling the processing modules within the NPU101 to handle tasks. Accordingly, the NPU101 receives processing instructions through the scheduler, forwards these instructions to the processing modules, and then the processing modules receive the instructions. These processing modules can be of any type, such as those within the Tensor Engine, Vector Engine, or accumulator of the NPU101.

[0035] In one possible implementation, NPU101 includes a storage module, which loads the acquired task data into the storage module of each task respectively; and processes each task based on the task data in the storage module of each task. Optionally, in response to a processing instruction, NPU101 determines the storage module allocated to each task carried by the processing instruction, and then loads the task data of each task into the storage module of each task respectively. The storage module includes any type of storage module, such as the storage module in the tensor processor, the storage module in the vector processor, the storage module in the accumulator, etc. in NPU101.

[0036] Optionally, the NPU101 determines the storage modules allocated to each task by: the NPU101 determines the number of storage modules allocated to each task based on the ratio of the total number of storage modules to the number of tasks, and then allocates that number of storage modules to each task. Alternatively, considering that different types of tasks have different computational loads, and tasks with different computational loads require different storage spaces, to ensure sufficient storage space is prepared for tasks, the storage modules allocated to each task are determined according to the task type to which each task belongs, wherein tasks with higher computational loads are allocated more storage modules.

[0037] Optionally, the processing instructions carry the storage location of the task data for each task. Accordingly, the neural network processor retrieves the task data for each task based on its storage location and loads it into the allocated storage module for each task. For example, the processing instructions carry the storage locations of the task data for a first task and a second task. The first task is a first-type task and is stored in the first memory 102, while the second task is a second-type task and is stored in the second memory 103. The NPU 101 retrieves the task data for the first task from the first memory 102 and loads it into the allocated storage module for the first task. It then retrieves the task data for the second task from the second memory 103 via the bus and loads it into the allocated storage module for the second task.

[0038] In this embodiment, NPU101 allocates its own storage module to each task, and the task data of each task is stored in its own storage module. In this way, NPU101 can distinguish the task data of different tasks through different storage modules, so that the task data in the storage module of each task does not affect each other during the processing of each task.

[0039] In one possible implementation, CPU 104 identifies multiple tasks whose computational complexity is less than a computational complexity threshold, and sends a first processing instruction carrying the multiple tasks to NPU 101. NPU 101, in response to the first processing instruction, retrieves task data for the multiple tasks from at least one of the first memory 102 and the second memory 103, and processes the multiple tasks using time-division multiplexing. The computational complexity threshold can be set to any value as needed, and this embodiment does not impose any limitations on it.

[0040] This method employs time-division multiplexing to process multiple tasks, meaning multiple tasks are processed alternately. For example, multiple tasks are arranged sequentially, and each time a task is processed, only a portion of that task is processed before moving on to the next task's portion. After processing a portion of the last task, the unprocessed portion of the first task is processed, followed by the unprocessed portion of the next task after the current task, and so on. The current task is any one of these multiple tasks.

[0041] Optionally, the order in which multiple tasks are processed alternately can be flexibly set as needed. Optionally, the NPU101 processes multiple tasks based on the processing latency required by the multiple tasks, for example, prioritizing the processing of first-type tasks with lower required processing latency. Alternatively, the NPU101 determines the order in which the multiple tasks are processed alternately according to their priority, for example, starting with the higher-priority tasks. Optionally, tasks with lower required processing latency have higher priority. Alternatively, the NPU101 determines the order in which the multiple tasks are processed alternately according to the order of the multiple tasks carried in the processing instructions; this embodiment of the application does not limit this.

[0042] Figure 3 This is a schematic diagram illustrating a process for handling multiple tasks using time-division multiplexing, as provided in an embodiment of this application. (Reference) Figure 3 The NPU101 switches tasks over time. For example, it processes task A in time period T, task B in time period T+1, task A in time period T+2, task B in time period T+3, and so on. Specifically, if the task switching condition is met while currently processing task A, it switches to processing task B; conversely, if the task switching condition is met while currently processing task B, it switches back to processing task A. Here, T is a natural number greater than 0.

[0043] In one possible implementation, the NPU101 uses time-division multiplexing to process multiple tasks, including: when a task switching condition is met, the NPU101 processes the next task of the current task. For example, the NPU101 processes the current task based on the task data in the storage module of the current task, and when the task switching condition is met, processes the next task based on the task data in the storage module of the next task.

[0044] In this embodiment, time-division multiplexing is used to process multiple tasks, and task switching is controlled by task switching conditions, so that NPU101 is not limited to processing only one task at a time, which greatly improves the flexibility of NPU101 in processing tasks.

[0045] The task switching conditions can be flexibly set as needed. Optionally, the task switching condition is that the processing time of the current task reaches a reference time. Accordingly, when the processing time of the current task reaches the reference time, the NPU101 will process the next task of the current task. Optionally, the task switching condition is that the computational load of the current task reaches a reference computational load. Taking a reference computational load of one-tenth of the total computational load corresponding to the current task as an example, the task switching condition is that the computational load of the current task reaches one-tenth of the total computational load.

[0046] In one possible implementation, when the task switching condition is met, the NPU101 first stores the intermediate processing result of the current task in the current task's storage module, and then processes the next task. When the task switching condition is met, since the current task has not yet finished processing, the result obtained is not the final result, but rather an intermediate processing result. Storing the intermediate processing result of the current task in the current task's storage module avoids the loss of the intermediate processing result, ensuring that when switching back to process the same task, the intermediate processing result can be retrieved from the task's storage module, allowing the task to continue processing based on the intermediate processing result, thus guaranteeing the smooth processing of the task.

[0047] In one possible implementation, the task data includes the input data of the neural network and the weight parameters corresponding to multiple operators in the neural network. The input data includes multiple unit input data. Accordingly, the NPU101 processes the next task of the current task when the task switching conditions are met, including: after the NPU101 has completed processing one unit input data of the current task based on the weight parameters corresponding to one operator in the neural network, it processes the next task of the current task.

[0048] For example, if the input data is an image, the image consists of multiple image patches of unit size; correspondingly, each unit of input data is an image patch of unit size. If the input data is video, the video consists of multiple video frames; correspondingly, each unit of input data is one video frame. If the input data is audio, the audio consists of multiple audio segments of unit length; correspondingly, each unit of input data is an audio segment of that unit length.

[0049] The neural network can be of any type. For example, if the input data is an image, the neural network can be an image processing network. Or, if the input data is video, the neural network can be a video processing network. Or, if the input data is audio, the neural network can be an audio processing network. The multiple operators in the neural network include operators of any type, such as convolution operators, pooling operators, activation operators, normalization operators, scaling operators, etc., and this application embodiment does not impose any limitations on this.

[0050] Optionally, data dependencies exist between the multiple operators included in the neural network. For example, the feature data output by the first operator serves as the input data for the second operator. Therefore, for the operators other than the first operator in the neural network, completing the processing of a unit of input data for the current task means completing the processing of the feature data output by the previous operator that corresponds to that unit of input data.

[0051] In this embodiment, the input data of the neural network is divided into multiple unit input data. When the processing of one unit input data of the current task is completed based on the weight parameters corresponding to an operator in the neural network, the next task of the current task is processed. This task switching scheme makes the NPU101 no longer limited to processing the current task before switching to process the next task, which greatly improves the flexibility of task processing.

[0052] It should be noted that after the NPU101 stores the intermediate processing result of the current task into the storage module of the current task, it needs to perform the following steps (1) or (2) depending on whether the next task of the current task has been processed.

[0053] (1) If the NPU101 has not processed the next task of the current task, it starts processing the next task based on the task data in the storage module of the next task.

[0054] Because multiple tasks are processed in an alternating manner, if the next task of the current task has not been processed, there is no corresponding intermediate processing result. The next task is processed directly based on the task data in the storage module of the next task of the current task, which ensures the correct processing of the task.

[0055] (2) When the NPU101 has already processed the next task of the current task, it processes the next task based on the task data and intermediate processing results in the storage module of the next task.

[0056] For ease of description, let's take the current task as task A and the next task as task B as an example. The way in which the intermediate processing results of task B are stored in the storage module of task B is the same as the way in which the intermediate processing results of task A are stored in the storage module of task A, and will not be repeated here.

[0057] For example, if the NPU101 has already processed the input data of the neural network based on the first operator in the neural network corresponding to task B, obtained the intermediate processing result, and stored the intermediate processing result in the storage module of task B, then when processing task B this time, the NPU101 will continue to process the intermediate processing result based on the second operator in the neural network.

[0058] In this embodiment, since time-division multiplexing is used to process multiple tasks, the next task of the current task may have already been partially processed. In this case, there are intermediate processing results of the next task. Therefore, processing the next task based on the task data and intermediate processing results in the storage module of the next task can avoid repeatedly processing the parts of the next task that have already been processed, thereby ensuring the correct processing of the task.

[0059] In one possible implementation, the CPU 104 is further configured to send a third processing instruction to the NPU 101 during the processing of multiple tasks. The multiple tasks include a first task and a second task, where the first task is a first-type task, the second task is a second-type task, and the third processing instruction carries a third task, which is also a first-type task. The NPU 101 is further configured to, in response to the third processing instruction, transfer the associated data of the second task out of the second task's storage module and reallocate the storage module to the third task. The associated data includes the task data and intermediate processing results of the second task. Then, the NPU 101 loads the task data of the third task into the storage module. The first and third tasks are processed using time-division multiplexing. Optionally, the computational load of the third task is less than the aforementioned computational load threshold.

[0060] During the alternating processing of the first and second tasks, the neural network processor receives a processing instruction carrying the third task. Since the third task is a first-type task, its required processing latency is lower than that of the second task. Therefore, the processing of the second task is interrupted. That is, the second task is excluded from the multiple tasks processed in a time-division multiplexing manner, and the first and third tasks are processed in a time-division multiplexing manner instead. The implementation method includes: transferring the associated data of the second task out of the allocated storage module and reallocating the storage module to the third task. In other words, while keeping the storage modules allocated to other tasks other than the second task unchanged, the storage module previously allocated to the second task is transferred to the third task. Then the task data of the third task can be loaded into the storage module, thereby allowing the third task to join the time-division multiplexing process.

[0061] In this embodiment, since the processing latency required by the third task is lower than that of the second task, the processing of the second task is interrupted when a processing instruction carrying the third task is received. The third task inherits the storage module of the second task and replaces the second task in the time-division multiplexing process, ensuring that the task with the lower processing latency is processed in a timely manner.

[0062] In one possible implementation, the NPU101 is further configured to, upon completion of either the first or third task, reallocate the storage module of the completed task to the second task; load the associated data of the second task into the storage module; and process the second task and any unfinished tasks from the first and third tasks using time-division multiplexing. This ensures that the third task, with its lower processing latency, is processed promptly, while also enabling the second task, with its higher processing latency, to be processed as quickly as possible, thereby reducing overall task processing latency.

[0063] In this embodiment, the number of the first task, second task, and third task is not limited. Taking the example of multiple first tasks and multiple second tasks, and one third task, the NPU101 interrupts the processing module of any second task, and the third task inherits the storage module of the second task and replaces the second task in the time-division multiplexing process.

[0064] Optionally, the NPU 101 transfers the associated data of the second task from the allocated storage module to a target location, for example, to DDR (Double Data Rate) in the computing device 100. Subsequently, after processing either the first or third task and reallocating the storage module allocated to that task to the second task, the associated data of the second task is loaded from the DDR into that storage module for further processing. Optionally, the DDR is connected to the NPU 101 via a bus.

[0065] Figure 4 This is a schematic diagram of an NPU400 provided as an embodiment of this application. (Reference) Figure 4 The NPU 400 includes a tensor processor 401, a vector processor 402, a memory 403, a scheduler 404, and an accumulator 405. The tensor processor 401 includes multiple multiply-accumulate units 406, each including a processing module and a storage module. Similarly, the vector processor 402 and the accumulator 405 also include processing and storage modules. Therefore, in this embodiment, the processing modules in the NPU 400 include those in the tensor processor 401, vector processor 402, and accumulator 405, while the storage modules in the NPU 400 include those in the tensor processor 401, vector processor 402, and accumulator 405. Optionally, the storage modules in the tensor processor 401, vector processor 402, and accumulator 405 can only be used by each of the three processors individually, while the memory 403 is shared by the tensor processor 401, vector processor 402, and accumulator 405. Optionally, the memory 403 is SRAM (Static Random-Access Memory). In this embodiment, task data for multiple tasks is first loaded from outside the NPU 400 into the memory 403, and then loaded from the memory 403 into each storage module. For example, the task data includes the input data of the neural network and the weight parameters corresponding to multiple operators of the neural network. Since different processors are suitable for processing different operators, for example, the tensor processor 401 is suitable for processing convolution operators, and the vector processor 402 is suitable for processing pooling operators and scaling operators, the input data of the neural network and the weight parameters corresponding to multiple operators of the neural network are loaded into the storage modules of the corresponding processors.

[0066] Figure 5 This is a schematic diagram of a processing unit provided in an embodiment of this application. The processing unit is as described above. Figure 4 The multiply-accumulate unit 406 in the reference. Figure 5The multiply-accumulate unit 406 includes a multiply-accumulate tree and four storage modules. The multiply-accumulate tree is the processing module within the multiply-accumulate unit 406, used to perform multiplication and addition operations. In this embodiment, taking the NPU 400 alternately processing tasks A and B as an example, storage modules 501 and 502 are assigned to task A to store the weight parameters corresponding to the operators of the neural network in task A, as well as the data to be processed. Storage modules 503 and 504 are assigned to task B to store the weight parameters corresponding to the operators of the neural network in task B, as well as the data to be processed. Assuming that the multiply-accumulate tree, based on the data in the storage module allocated to task A, meets the task switching condition when processing task A, stores the currently obtained intermediate processing result in the storage module allocated to A. Then, task B is processed based on the data in the storage module allocated to task B. Optionally, the two storage modules allocated to either task include a working register and a weight parameter storage module. The weight parameter storage module stores the weight parameters corresponding to the operators in the neural network, while the working register stores the current working status, control signals, and intermediate processing results of the multiply-accumulate unit 406. The working status indicates the current task's processing progress and whether processing is in progress. Control signals include enable signals; the multiply-accumulate unit 406 can only process data when the scheduler in the NPU400 provides an enable signal. Figure 6 As shown, the multiply-accumulate unit 600 in the related art includes a multiply-accumulate tree and two storage modules. Compared to this multiply-accumulate unit 600, the multiply-accumulate unit 406 in this embodiment has more storage modules. Therefore, when processing multiple tasks alternately, the associated data of multiple tasks can have sufficient storage space. It should be noted that in other embodiments, the multiply-accumulate unit 406 includes not only four storage modules, but also more.

[0067] The above Figure 4 The tensor processor 401 works in conjunction with the accumulator 405 to perform convolution operations, that is, to process the input data of the convolution operator using the weight parameters corresponding to the convolution operator in the neural network. (Continue to refer to...) Figure 4The tensor processor 401 includes multiple multiply-accumulate units 406 arranged in an array. When the input data and weight parameters corresponding to the convolution operator are loaded from the memory 403 in the NPU 400 into the storage module of each multiply-accumulate unit 406, taking the first column of multiply-accumulate units 406 as an example, the bottom multiply-accumulate unit 406 reads the weight parameters and the data to be processed from its own storage module, multiplies the data with the weight parameters, and inputs the resulting data into the next multiply-accumulate unit 406 in that column. Then, the next multiply-accumulate unit 406 multiplies the data to be processed in its storage module with the weight parameters, adds the result of the multiplication to the data input from the previous multiply-accumulate unit 406, and inputs the resulting data into the next multiply-accumulate unit 406 in that column, and so on. It should be noted that the multi-column multiply-accumulate unit 406 processes the data in parallel and outputs the final data to the accumulator 405. The accumulator 405 adds the multiple data to complete the convolution operation.

[0068] Continue to refer to Figure 2 In the above embodiments, considering the limited number of storage modules in NPU101, in order to ensure that there are enough storage modules, CPU104 determines multiple tasks with computational loads less than the computational load threshold. Since these multiple tasks require fewer storage modules during processing, the storage modules in NPU101 can be allocated to these multiple tasks to ensure that each task has enough storage modules. This avoids the loss of task data and intermediate processing results due to insufficient storage modules during the processing of multiple tasks using time-division multiplexing, thereby ensuring the correct processing of multiple tasks.

[0069] In one possible implementation, the CPU 104 is further configured to determine a task whose computational complexity is not less than a computational complexity threshold, and send a second processing instruction carrying the task to the NPU 101; the NPU 101 is further configured to receive the second processing instruction and, in response to the second processing instruction, execute the task separately.

[0070] Optionally, if the processing latency required for a task with a computational load not less than the computational load threshold is lower than the processing latency required for each task in the first processing instruction, even if the NPU 101 is currently processing multiple tasks in the first processing instruction using time-division multiplexing, the CPU 104 will still send a second processing instruction to the NPU 101. The NPU 101 will then transfer the associated data of each task being processed from its allocated storage module, and load the task data carried by the second processing instruction into the storage module of the NPU 101 for separate processing. Optionally, after the task is completed, the NPU 101 will load the associated data of the multiple tasks carried by the first processing instruction into the allocated storage module for further processing.

[0071] Optionally, if the CPU 104 has completed processing multiple tasks in the first processing instruction, it sends the second processing instruction to the NPU 101. Since the NPU 101 has not processed any tasks at this time, it can execute the tasks in the second processing instruction alone upon receiving the second processing instruction.

[0072] In this embodiment, considering that the storage space corresponding to the storage module in NPU101 is limited, and the task with a computational workload greater than the computational workload threshold requires a large amount of storage space during processing, for tasks with a computational workload greater than the computational workload threshold, NPU101 executes the task alone. In this way, the task can occupy all the storage modules in NPU101, thereby preparing sufficient storage space for the task and avoiding the loss of task data and intermediate processing results due to insufficient storage space during the processing of the task, thus ensuring the correct processing of the task.

[0073] In addition to being able to process multiple tasks using time-division multiplexing, the NPU101 in this embodiment can also process a single computationally intensive task, demonstrating ample flexibility.

[0074] It should be noted that, in practice, the number of second-type tasks with higher processing latency is greater than that of first-type tasks requiring lower processing latency. Therefore, in one possible implementation, CPU 104 is also used to generate a task queue including at least one second-type task, and store the task data of each task in the task queue in the second memory 103. This allows NPU 101 to retrieve the task data of each task in the task queue from the second memory 103, ensuring that each task in the task queue can be processed. Optionally, CPU 104 generates processing instructions according to the order of tasks in the task queue, so that tasks at the beginning of the task queue are processed first, ensuring the orderly processing of tasks.

[0075] Figure 7 This is a schematic diagram of a task queue provided in an embodiment of this application. (Reference) Figure 7 The task queue contains three second-type tasks, which are arranged in order from front to back as task A, task B, and task C. Figure 8 This is a schematic diagram of a second memory 103 provided in an embodiment of this application. (See reference...) Figure 8 The second memory 103 stores the task data of task A, task B, and task C in the order of the tasks in the task queue.

[0076] Optionally, in this embodiment, the first memory 102 is directly connected to both the first processing unit and the NPU 101, and the computing device 100 further includes a third memory, which is directly connected to both the NPU 101 and the second processing unit. The first processing unit stores the processed task data of the first type of task in the first memory 102. The NPU 101 retrieves the task data from the first memory 102, processes the task data, and stores the result in the third memory. The second processing unit then retrieves the result from the third memory and processes it further. That is, a pipelined approach is used to process the first type of task to reduce processing latency. Additionally, the NPU 101 is connected to the second memory 103 via a bus. The second memory 103 stores task data of multiple second type tasks in the order of multiple second type tasks in the task queue. The NPU 101 retrieves the task data from the second memory 103 via the bus and processes the retrieved task data; that is, a bus approach is used for reading and processing task data. Optionally, the second memory 103 is an OCM (On-Chip Memory).

[0077] Optionally, the NPU101 in this embodiment has more processing modules and storage modules than the NPU101 in related technologies, and its task processing efficiency is higher. Therefore, even if multiple tasks are processed alternately, the processing time of each task will not be extended compared to the NPU101 in related technologies. Moreover, from an overall perspective, multiple tasks are processed in the same amount of time, so the task processing efficiency is higher.

[0078] It's important to note that in related technologies, computing devices contain a first NPU and a second NPU, whose processing and storage modules are not shared. The first NPU is dedicated to processing the first type of task, and the second NPU is dedicated to processing the second type of task. Therefore, when a task requiring NPU processing is generated, the CPU can only determine which NPU to assign the task to based on its type; it cannot call upon both the first and second NPUs to process a single task. Optionally, the number of processing modules in the NPU101 in this embodiment is the sum of the number of processing modules in the first NPU and the second NPU, and the number of storage modules in the NPU101 in this embodiment is the sum of the number of storage modules in the first NPU and the second NPU. Using one NPU101 in this embodiment to replace the first NPU and the second NPU in the computing device will not increase the area occupied by the NPU101 in the computing device. On the other hand, the NPU101 can call all the internal processing modules and storage modules (equivalent to the processing modules and storage modules of the first NPU and the second NPU) to process tasks with large computational loads, which greatly improves task processing efficiency.

[0079] Continue to refer to Figure 1 and Figure 2 Optionally, the first memory 102 and the second memory 103 in this embodiment include random access memory (RAM), read-only memory (ROM), non-transitory computer-readable storage medium, etc. In addition to the first memory 102 and the second memory 103 described above, the computing device 100 may also include other memories, such as memories for storing program code and the aforementioned task data.

[0080] Optionally, the computing device 100 may also include a graphics processing unit (GPU) and / or a modem. The GPU is responsible for rendering and drawing the content that the display screen needs to show. The modem is used for handling wireless communication.

[0081] Optionally, the computing device 100 also includes a display screen. The display screen is a display component used to display a user interface. Optionally, the display screen is a touch-enabled display screen, through which the user can perform touch operations on the display screen using a finger, stylus, or any suitable object.

[0082] In addition, those skilled in the art will understand that the structure of the computing device 100 shown in the accompanying drawings of the embodiments of this application does not constitute a limitation on the computing device 100. The computing device 100 may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the computing device 100 may also include components such as a microphone, speaker, input unit, sensor, audio circuit, module, power supply, and Bluetooth module, which will not be described in detail here.

[0083] Figure 9 A flowchart illustrating a task processing method provided in an exemplary embodiment of this application is shown. The method is executed by a computing device, which includes an NPU, a first memory, and a second memory. The NPU is directly connected to the first memory and connected to the second memory via a bus. See also... Figure 9 The method includes:

[0084] 901. The computing device stores task data for a first type of task through a first memory.

[0085] 902. The computing device stores task data for the second type of task through a second memory, where the processing latency required for the first type of task is lower than that required for the second type of task.

[0086] 903. The computing device obtains task data from the first memory through the NPU and obtains task data from the second memory through the bus, and processes the task based on the obtained task data.

[0087] The implementation methods for each step are described in the above embodiments and will not be repeated here.

[0088] The method provided in this application can be applied to video processing scenarios. For example, after a user clicks the shooting control, the CPU in the computing device calls the camera to capture video footage and the microphone to capture audio. The video footage is stored in a first memory directly connected to the NPU, and the audio is stored in a second memory connected to the NPU via a bus. Then, a processing instruction is sent to the NPU to instruct it to process the video footage and the audio. In response to the processing instruction, the NPU loads the video footage and the audio into its internal storage modules, and then processes the video footage and the audio using a time-division multiplexing method.

[0089] Of course, the task processing method provided in this application embodiment can also be applied to other scenarios, such as in the scenario of processing multiple audio files, and this application embodiment does not limit this.

[0090] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0091] Those skilled in the art will understand that all or part of the steps in the task processing method of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. The above descriptions are merely optional embodiments of this application and are not intended to limit the application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A computing device, characterized in that, The computing device includes a neural network processor (NPU), a first memory, and a second memory. The NPU is directly connected to the first memory and connected to the second memory via a bus. The NPU is more efficient at acquiring data from the first memory than at acquiring data from the second memory. The first memory is used to store task data for the first type of task; The second memory is used to store task data for the second type of task, wherein the processing latency required by the first type of task is lower than the processing latency required by the second type of task; The NPU is used to acquire task data from the first memory and acquire task data from the second memory via the bus, and process tasks based on the acquired task data.

2. The computing device according to claim 1, characterized in that, The NPU includes a storage module, and the NPU is used for: The acquired data for each task is loaded into the storage module of each task. Each task is processed based on the task data within the storage module for each task.

3. The computing device according to claim 1, characterized in that, The computing device also includes a central processing unit (CPU); The CPU is used to determine multiple tasks whose computational workload is less than a computational workload threshold, and send a first processing instruction carrying the multiple tasks to the NPU. The NPU is configured to, in response to the first processing instruction, retrieve task data of the plurality of tasks from at least one of the first memory and the second memory, and process the plurality of tasks in a time-division multiplexing manner.

4. The computing device according to claim 3, characterized in that, The NPU is used for: If the task switching conditions are met, proceed with the next task from the current task.

5. The computing device according to claim 4, characterized in that, The NPU is used for: If the task switching conditions are met, the intermediate processing results of the current task are stored in the storage module of the current task.

6. The computing device according to claim 4, characterized in that, The NPU is used for: If the task switching conditions are met and the next task has not been processed, the next task will be processed based on the task data in the storage module of the next task. If the task switching conditions are met and the next task has already been processed, the next task is processed based on the task data and intermediate processing results in the storage module of the next task.

7. The computing device according to claim 4, characterized in that, The task data includes the input data of the neural network and the weight parameters corresponding to multiple operators in the neural network. The input data includes multiple unit input data. The NPU is used to process the next task of the current task after processing one unit of input data of the current task based on the weight parameters corresponding to an operator in the neural network.

8. The computing device according to claim 3, characterized in that, The CPU is also used to determine a task whose computational workload is not less than the computational workload threshold, and send a second processing instruction carrying the task to the NPU; The NPU is also used to process the task individually in response to the second processing instruction.

9. The computing device according to claim 1, characterized in that, The computing device also includes a central processing unit (CPU); The CPU is configured to send a third processing instruction to the NPU during the NPU's processing of multiple tasks. The multiple tasks include a first task and a second task. The first task is a first type of task, and the second task is a second type of task. The third processing instruction carries a third task, which is a first type of task. The NPU is also used for: In response to the third processing instruction, the associated data of the second task is transferred out of the storage module of the second task, and the storage module is reassigned to the third task, wherein the associated data includes the task data and intermediate processing results of the second task; Load the task data of the third task into the storage module; The first task and the third task are processed using time-division multiplexing.

10. The computing device according to claim 9, characterized in that, The NPU is also used for: If either the first task or the third task is completed, the storage module of the completed task is reassigned to the second task. Load the associated data of the second task into the storage module; The second task, as well as any unfinished tasks from the first and third tasks, are processed using time-division multiplexing.

11. The computing device according to claim 1, characterized in that, The computing device also includes a central processing unit (CPU); The CPU is used to generate a task queue, the task queue including at least one task of the second type, and to store the task data of each task in the task queue into the second memory.

12. A task processing method, characterized in that, The method is executed by a computing device, which includes a neural network processor (NPU), a first memory, and a second memory. The NPU is directly connected to the first memory and connected to the second memory via a bus. The NPU's efficiency in retrieving data from the first memory is higher than its efficiency in retrieving data from the second memory. The method includes: The first memory stores the task data for the first type of task; The second memory stores the task data of the second type of task, where the processing latency required by the first type of task is lower than that required by the second type of task. The NPU acquires task data from the first memory and acquires task data from the second memory via the bus, and processes the task based on the acquired task data.

Images