A chip, an acceleration card, an electronic device and a data processing method

Abstract

Embodiments of the present disclosure provide a chip, an acceleration card, an electronic device, and a data processing method and apparatus. A control unit in the chip controls scheduling of each subnetwork of a neural network to a first processing unit for AI inference. When the first processing of a group of subnetworks is completed, the control unit schedules network parameters of the next group of subnetworks to the first processing unit for the next stage of first processing, so that the multiple groups of subnetworks included in the neural network realize pipelining processing, thereby improving processing efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of chip technology, and more particularly to a chip, an accelerator card, an electronic device, and a data processing method. Background Technology

[0002] In recent years, neural networks have been widely used in various fields such as image processing, fault diagnosis, and video security. In most of these applications, multiple neural networks may need to work together to solve a complex problem. However, in practical applications, the terminal typically needs to serially schedule each neural network to the GPU or accelerator card via the host CPU, making it difficult to achieve pipelined processing of each neural network and resulting in low processing efficiency. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide a chip, the chip comprising: a control unit, configured to schedule network parameters of one group of sub-networks from a plurality of sub-networks included in a neural network to a first processing unit; a first processing unit, configured to perform a first processing on a video frame to be processed in a video based on the network parameters of the sub-network scheduled to the processing unit, to obtain a first video frame; the control unit is further configured to, upon completion of the first processing, schedule network parameters of the next group of sub-networks from the plurality of sub-networks to the first processing unit.

[0004] In some embodiments, the first processing unit is configured to send an interrupt signal to the control unit when the first processing of the video frame to be processed based on the network parameters of the sub-network currently scheduled to the processing unit is completed; the control unit is configured to schedule the network parameters of the next set of sub-networks to the first processing unit upon receiving the interrupt signal.

[0005] In some embodiments, the number of the first processing units is greater than 1, and each first processing unit is used to perform the first processing on the video frame to be processed based on the network parameters scheduled to the processing unit; the control unit is used to schedule the network parameters of two adjacent sub-networks to different first processing units.

[0006] In some embodiments, each of the plurality of first processing units is configured to: upon completion of the first processing in its own processing unit, send an interrupt signal to the control unit so that the control unit schedules the network parameters of the next set of sub-networks to other first processing units.

[0007] In some embodiments, the control unit is further configured to: send an enable signal to the first processing unit, wherein the first processing unit performs the first processing upon receiving the enable signal.

[0008] In some embodiments, the chip further includes a second processing unit, configured to perform a second processing on the first video frame to obtain a second video frame and output it.

[0009] In some embodiments, the chip further includes: a third processing unit, configured to perform a third processing on the video frame to be processed and then output it to the first processing unit, so that the first processing unit performs a first processing on the video frame to be processed after the third processing.

[0010] Secondly, embodiments of this disclosure provide an accelerator card, the accelerator card comprising: a memory unit for storing network parameters of each sub-network in a plurality of sub-networks included in a neural network; and a chip as described in any embodiment of this disclosure.

[0011] Thirdly, embodiments of this disclosure provide an electronic device, the electronic device comprising: the accelerator card described in the second aspect; and an external processing unit for outputting network parameters of each sub-network in the multiple sub-networks included in the neural network to the memory unit.

[0012] Fourthly, embodiments of this disclosure provide a data processing method, which is applied to a control unit in a chip according to any embodiment of this disclosure; the method includes: scheduling network parameters of one group of sub-networks in a neural network to a first processing unit; and, after the first processing is completed, scheduling network parameters of the next group of sub-networks in the multiple groups of sub-networks to the first processing unit.

[0013] In some embodiments, scheduling the network parameters of the next group of sub-networks in the plurality of sub-networks to the first processing unit when the first processing is completed includes: receiving an interrupt signal sent by the first processing unit, wherein the first processing unit sends the interrupt signal when the first processing of the video frame to be processed based on the network parameters of the sub-network currently scheduled to the processing unit is completed; and responding to the interrupt signal, scheduling the network parameters of the next group of sub-networks to the first processing unit.

[0014] In some embodiments, the number of the first processing units is greater than 1, and each first processing unit is used to perform the first processing on the video frame to be processed based on the network parameters scheduled to the processing unit; the network parameters of two adjacent sub-networks are scheduled to different first processing units.

[0015] In some embodiments, scheduling the network parameters of the next group of sub-networks in the plurality of sub-networks to the first processing unit when the first processing is completed includes: receiving an interrupt signal sent by the first processing unit, wherein the first processing unit sends the interrupt signal when the first processing of the video frame to be processed based on the network parameters of the sub-network currently scheduled to the processing unit is completed; and responding to the interrupt signal, scheduling the network parameters of the next group of sub-networks to a first processing unit other than the first processing unit.

[0016] In some embodiments, the method further includes: sending an enable signal to the first processing unit, wherein the first processing unit performs the first processing upon receiving the enable signal.

[0017] Fifthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any embodiment of this disclosure.

[0018] This disclosure provides a chip, an accelerator card, an electronic device, and a data processing method and apparatus. The chip controls and schedules each sub-network of the neural network to a first processing unit for AI operations (including training and / or inference) through a control unit within the chip. When the first processing of a group of sub-networks is completed, the control unit schedules the network parameters of the next group of sub-networks to the first processing unit for the next stage of first processing, thereby enabling the multiple groups of sub-networks included in the neural network to achieve pipelined processing, thereby improving processing efficiency.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to illustrate the technical solutions of this disclosure.

[0021] Figure 1 This is a schematic diagram of a chip according to an embodiment of this disclosure;

[0022] Figure 2 This is a schematic diagram of a data processing pipeline according to an embodiment of the present disclosure;

[0023] Figure 3 This is a schematic diagram of a data processing pipeline according to another embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram of the data flow between multiple processing units according to an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of an accelerator card according to an exemplary embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure;

[0027] Figure 7 This is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure;

[0028] Figure 8 This is a block diagram of a data processing apparatus according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0030] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0031] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0032] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0033] In recent years, neural networks have been widely used in various fields such as image processing, fault diagnosis, and video security. In most application scenarios, multiple neural networks may need to work together to solve a complex problem. For example, when performing image quality assessment tasks, it may be necessary to combine several neural networks that can extract features of the image in different quality dimensions and neural networks that can score the features of the image in different quality dimensions to obtain a score value that represents the image quality.

[0034] In practical applications, terminals typically need to serially schedule various neural networks to GPUs or accelerator cards via a host CPU, making it difficult to achieve pipelined processing of each neural network. Taking video face recognition using multiple neural networks as an example, it often requires multiple steps, such as video decoding, video frame analysis, and facial local feature analysis, to extract facial features from the video before achieving final face recognition. These steps, such as video frame analysis and facial local feature analysis, can be implemented using multiple neural networks. In this scenario, the terminal generally uses GPUs or accelerator cards to perform AI training (i.e., the process of training neural networks using sample data) or AI inference (inference is the process of deriving conclusions from existing facts and knowledge according to a certain strategy; AI inference is the process of processing input data using artificial intelligence algorithms). In related technologies, the data of each neural network is generally stored in the host main memory. The host CPU then serially schedules each neural network to the GPU or accelerator card during use. For example, the host CPU first schedules the video frame to be processed, along with the network parameters of the neural network used to detect face regions within the video frame, to the GPU or accelerator card. After the GPU or accelerator card completes model inference, the host CPU then schedules the network parameters of the neural network used for local facial feature analysis back to the GPU or accelerator card, and so on. This makes it difficult to achieve pipelined processing of the individual neural networks, which can easily affect processing efficiency.

[0035] Based on this, embodiments of this disclosure provide a chip to solve the above-mentioned problems. For example... Figure 1 The diagram shown is a schematic representation of a chip according to an exemplary embodiment of this disclosure. The chip includes:

[0036] Control unit 102 is used to schedule the network parameters of one set of subnetworks in the multiple sets of subnetworks included in the neural network to the first processing unit 101;

[0037] The first processing unit 101 is used to perform a first processing on the video frame to be processed in the video based on the network parameters of the sub-network scheduled to this processing unit, so as to obtain the first video frame.

[0038] The control unit 102 is further configured to schedule the network parameters of the next group of sub-networks in the plurality of sub-networks to the first processing unit after the first processing is completed.

[0039] The aforementioned chip can be an AI chip or a graphics processing unit (GPU) chip, or other data processing chip. This chip can be used in accelerator cards. The following discussion uses an AI accelerator card as an example. An AI accelerator card is a processor product specifically designed to accelerate the execution of AI algorithms. This AI accelerator card can be a circuit board including hardware modules such as a chip for AI computation and a communication interface. The chip for AI computation can be any one of a GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit), or other types of chips; this disclosure does not limit its application. The AI ​​accelerator card can be inserted into a slot and communicate with the host CPU through a communication interface, such as a PCIe (Peripheral Component Interconnect Express) interface. Of course, in other embodiments, the AI ​​accelerator card can also use other types of interfaces, such as a QPI (Quick Path Interconnect) interface, as an interaction channel with the host CPU.

[0040] The AI ​​accelerator card may include memory units for storing network parameters of each sub-network in a neural network comprising multiple sub-networks. These memory units are integrated memory on the AI ​​accelerator card and are a component of its memory. The memory units can be DRAM (Dynamic Random Access Memory) or SDRAM (Synchronous Dynamic Random Access Memory), or other types of memory; this disclosure does not limit the types. Each sub-network can be an independent neural network or may include one or more network layers of that neural network. The network parameters of each sub-network may include operators (e.g., addition operators, multiplication operators, etc.) and the parameters required for the operators to perform operations. A group of sub-networks can map inputs to outputs through their corresponding network parameters. A group of sub-networks can implement a function, wherein at least one sub-network is included in the group. In the case where the neural network comprises multiple groups of sub-networks, the number of sub-networks included in each group may be the same or different. Different subnetworks can perform different functions. For example, a set of neural networks used to perform face recognition tasks in videos may include subnetworks for face detection and subnetworks for local facial feature analysis. It should be noted that a subnetwork can be any type of CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), etc. Different subnetworks can be of the same or different types; this disclosure does not impose any restrictions on this.

[0041] In some examples, the aforementioned memory unit can be configured with multiple partitions, with network parameters for different sub-networks stored in different partitions. That is, the storage space of this memory unit can adopt a hierarchical partition structure to achieve partitioned storage management of network parameters for each sub-network. Furthermore, to facilitate the reading of network parameters for a specific sub-network, a partition table can be created. This partition table can record information such as the starting address, size, and corresponding sub-network identifier for each partition. Other units can then determine the storage address of a specific sub-network by looking up this partition table, and subsequently read the network parameters of that specific sub-network based on that storage address. Alternatively, different partition identifiers can be set for different sub-networks. The memory unit determines the partition path based on the partition identifier of the sub-network and stores the network parameters of that sub-network in the partition corresponding to that partition path. Correspondingly, other units can also find the network parameters of a specific sub-network from the memory unit based on the partition identifier of that specific sub-network. Of course, in other embodiments, other storage management methods can be set according to the needs of specific scenarios.

[0042] The aforementioned first processing unit 101 is a functional module in the AI ​​accelerator card that performs AI computation tasks. This first processing unit can be referred to as a Computing Unit (CU). The first processing unit may include multiple sub-units, such as sub-units performing addition, multiplication, and convolution multiplication. These sub-units can be called reconfigurable clusters, which can be combined to form different computational pathways to perform different tasks. The first processing unit performing the first processing on the video frame to be processed based on network parameters means that the first processing unit uses the corresponding sub-networks to perform corresponding processing on the video frame. The specific content of this first processing can be determined based on the functions implemented by the sub-networks. For example, one set of sub-networks can be used to extract features from the video frame to obtain a feature map corresponding to the video frame; another set of sub-networks can be used to perform target detection on the target object in the video frame based on the feature map to obtain the target detection result corresponding to the video frame; and yet another set of sub-networks can be used to identify the target object in the video frame based on the target detection result to obtain the category of the target object in the video frame. It should be noted that in this embodiment, the AI ​​computing task performed by the first processing unit can be either AI training or AI inference. For the sake of brevity, the following content will be described as AI inference.

[0043] In this embodiment, the first processing unit's operation of performing first processing on the video frame to be processed based on the network parameters of the sub-network is executed under the control and scheduling of the control unit. Through the control and scheduling of the control unit, the first processing unit can orderly use appropriate sub-networks to process the video frame. Optionally, the control unit can be an MCU (Microcontroller Unit). That is, unlike the AI ​​accelerator card that simply acts as a slave device in related technologies, this embodiment adds an MCU to the AI ​​accelerator card to replace the host CPU for controlling and scheduling each sub-network. Of course, in other embodiments, the control unit can also be a processor chip with architectures such as ARM, RISC-V, or PowerPC. In some examples, the control unit can control and schedule the sub-networks downloaded to the first processing unit by sending an enable signal to the first processing unit. This enable signal is similar to a trigger signal, which can trigger the first processing unit to perform the operation of reading network parameters and then performing first processing on the video frame to be processed based on the read network parameters. Optionally, after enabling the first processing unit, the control unit can configure the storage address of the network parameters to be read for the first processing unit, so that the first processing unit can read the corresponding data from the storage address.

[0044] To ensure the control unit's efficient scheduling of multiple sub-networks, in some examples, the first processing unit can send an interrupt signal to the control unit upon completion of the first processing of the video frame to be processed based on the network parameters of the sub-network currently scheduled to this processing unit. Upon receiving the interrupt signal, the control unit then schedules the network parameters of the next sub-network to the first processing unit. In other words, after completing the processing of the video frame to be processed based on one sub-network, the first processing unit can notify the control unit by sending an interrupt signal, enabling the control unit to schedule the network parameters of the next sub-network. This allows the first processing unit to process the video frame based on the network parameters of the next sub-network. Thus, the AI ​​accelerator card can immediately start the AI ​​inference of the second sub-network after completing the AI ​​inference of the first sub-network, and so on, until the AI ​​inference of the last sub-network is completed, thereby achieving pipelined processing of each sub-network and improving processing efficiency.

[0045] The chip includes N first processing units, where N can be 1, meaning one CU can complete AI inference for all subnetworks included in the neural network; or N can be greater than or equal to 2, meaning multiple CUs can complete AI inference for multiple sets of subnetworks. When the chip includes multiple first processing units, each first processing unit can perform initial processing on the video frame to be processed based on the network parameters scheduled to its unit. The control unit can schedule the network parameters of two adjacent sets of subnetworks to different first processing units. Similarly, each first processing unit can send an interrupt signal to the control unit after completing its initial processing, causing the control unit to schedule the network parameters of the next set of subnetworks to other first processing units. In this way, other first processing units can continue processing the video frames processed by their respective units based on the network parameters of the next set of subnetworks. This achieves ordered scheduling of the subnetworks. In some examples, the value of N can correspond to the number of subnetworks included in the neural network. For example, if the neural network includes two sets of subnetworks, the AI ​​accelerator card can also include two first processing units. Each first processing unit processes the video frames to be processed based on the network parameters of one set of subnetworks, and different first processing units correspond to different subnetworks. In this way, after the first first processing unit has processed a set of video frames based on the first set of subnetworks, it can pass this set of video frames to the second first processing unit, which then processes this set of video frames based on the second set of subnetworks. At this point, the first first processing unit begins to process the next set of video frames based on the first set of subnetworks, thereby realizing pipelined processing of multiple sets of video frames.

[0046] See Figure 2 With both the number of first processing units and the number of sub-network groups being two, the HostCPU can transmit the video stream to be processed to the memory unit of the AI ​​accelerator card via the PCIe interface. The video is decoded by the video codec to obtain the video frames to be processed. The MCU can schedule these video frames to the first processing unit (CU1). Simultaneously, the MCU can also schedule the network parameters of the first sub-network group (model1) to CU1, allowing CU1 to process the video frames based on model1. The processing result obtained by CU1 can be output to the second processing unit (CU2). Furthermore, the MCU can schedule the network parameters of the second sub-network group (model2) to CU2, allowing CU2 to process the processing result output by CU1 based on model2, obtaining the final output result. In the above embodiment, CU1 and CU2 implement pipelined processing of model1 and model2.

[0047] Furthermore, if the number of subnetwork groups is greater than the number of first processing units, the network parameters of a portion of the subnetworks can be sequentially scheduled to the first processing units. Once one or more of the first processing units have completed the first processing, the network parameters of the remaining subnetworks can be scheduled to the first processing unit that has completed the first processing. In some embodiments, the number of video frames n processed at one time can be set. If a first processing unit completes processing n video frames, it is determined that the first processing unit has completed the first processing. If a first processing unit has completed processing the set number of video frames based on a set of subnetworks, the network parameters of another set of subnetworks can be scheduled to that first processing unit.

[0048] like Figure 3 The diagram shown is a schematic representation of a process for scheduling subnetworks to CUs to process video frames, according to an embodiment of this disclosure. The AI ​​accelerator card has three CUs: CU1, CU2, and CU3. The neural network comprises six subnetworks: subnetwork 1, subnetwork 2, subnetwork 3, subnetwork 4, subnetwork 5, and subnetwork 6 (referred to as M1, M2, M3, M4, M5, and M6 in the diagram), assuming each subnetwork forms a group. Figure 2 It can be seen that the steps of each CU processing a set of video frames to be processed in the video can be represented as follows:

[0049] Phase 1: Subnetwork 1 is scheduled to CU1. CU1 processes video frame 1 based on subnetwork 1. At this time, CU2 and CU3 are in an idle state. Figure 2 In the text, "M1:V1" indicates that video frame 1 is processed based on sub-network 1, and "empty" indicates that it is in an idle state, i.e., no processing is performed.

[0050] Second stage: After CU1 finishes processing video frame 1, it passes video frame 1 to CU2 and schedules subnetwork 2 to CU2. At this time, CU1 processes video frame 2 based on subnetwork 1, CU2 processes video frame 1 based on subnetwork 2, and CU3 is in an idle state.

[0051] Third stage: After CU2 finishes processing video frame 1, it passes video frame 1 to CU3 and schedules subnetwork 3 to CU3. After CU1 finishes processing video frame 2, it passes video frame 2 to CU2. At this time, CU1 processes video frame 3 based on subnetwork 1, CU2 processes video frame 2 based on subnetwork 2, and CU3 processes video frame 1 based on subnetwork 3.

[0052] Fourth stage: After CU3 finishes processing video frame 1, it passes video frame 1 to CU1 and schedules subnetwork 4 to CU1. After CU2 finishes processing video frame 2, it passes video frame 2 to CU3. After CU1 finishes processing video frame 3, it passes video frame 3 to CU2. At this time, CU1 processes video frame 1 based on subnetwork 4, CU2 processes video frame 3 based on subnetwork 2, and CU3 processes video frame 2 based on subnetwork 3.

[0053] Fifth stage: After CU3 finishes processing video frame 2, it passes video frame 2 to CU1. After CU2 finishes processing video frame 3, it passes video frame 3 to CU3. After CU1 finishes processing video frame 1, it passes video frame 1 to CU2. Sub-network 5 is scheduled to CU2. At this time, CU1 processes video frame 2 based on sub-network 4, CU2 processes video frame 1 based on sub-network 5, and CU3 processes video frame 3 based on sub-network 3.

[0054] Phase 6: After CU3 finishes processing video frame 3, it passes video frame 3 to CU1. After CU2 finishes processing video frame 1, it passes video frame 1 to CU3. Subnet 6 is scheduled to CU3. After CU1 finishes processing video frame 2, it passes video frame 2 to CU2. At this time, CU1 processes video frame 3 based on subnet 4, CU2 processes video frame 2 based on subnet 5, and CU3 processes video frame 1 based on subnet 6.

[0055] Phase 7: After CU3 finishes processing video frame 1, video frame 1 completes model inference for all sub-networks. At this point, video frame 1 is stored in memory. After CU2 finishes processing video frame 2, it passes video frame 2 to CU3. After CU1 finishes processing video frame 3, it passes video frame 3 to CU2. Sub-network 1 is rescheduled to CU1. At this point, CU1 processes video frame 4 based on sub-network 1, CU2 processes video frame 3 based on sub-network 5, and CU3 processes video frame 2 based on sub-network 6.

[0056] Phase 8: Continue in this manner until model inference for all sub-networks is completed for all video frames to be processed.

[0057] Video frames 1, 2, and 3 are all groups of video frames, and the number of video frames in each group is the set number of video frames. Each CU, after completing the processing of the set number of video frames based on the sub-network currently scheduled to it, begins processing based on the next sub-network scheduled to it. Furthermore, if model inference for all sub-networks is completed in a set number of video frames to be processed in one go, and there is an idle CU, then that idle CU will begin processing the set number of video frames to be processed next. This demonstrates that a pipeline is formed between the sub-networks, enabling parallel processing of multiple video frames and improving processing efficiency.

[0058] In practical applications, it is often necessary to process video frames processed by a set of sub-networks (referred to as post-processing). In related technologies, because GPUs or AI accelerator cards are limited by hardware conditions and do not support general-purpose processing operations, this processing is generally implemented by the host CPU. This requires the host CPU to read a set of video frames processed by a set of sub-networks from the GPU / AI accelerator card's memory, process them, and then store them back in the GPU / AI accelerator card's memory so that the GPU / AI accelerator card can process the processed video frames based on the next set of sub-networks. This makes it difficult to pipelining between different sub-networks, affecting the pipelining process between them. Therefore, in some examples, the chip disclosed herein may further include a second processing unit, which performs a second processing on the first video frame to obtain a second video frame, and outputs the second video frame to a memory unit for storage. The second processing, i.e., the post-processing mentioned above, may include at least one of the following processing methods: cropping, sharpening, rotation, scaling, and transparency. In other words, this disclosure achieves better pipelined processing by adding a second processing unit to the AI ​​accelerator card to replace the host CPU in performing model post-processing tasks.

[0059] In an optional embodiment, the second processing unit can be used to perform image segmentation on the first video frame to obtain an image including the target object in the first video frame, and output the image to the memory unit. The target object can be a complete object, such as a person, a basketball, or a tree, or a partial part of an object, such as a face, eyes, or tree branches. Optionally, the target object can be determined based on the processing object of the next set of sub-networks. For example, if the next set of sub-networks is a neural network for facial local feature analysis, the target object can be a face. In this case, the second processing unit can perform image segmentation on the first video frame, crop to obtain an image including the face region in the first video frame, and output this image to the memory unit for use by the next set of sub-networks.

[0060] Correspondingly, in practical applications, it is often necessary to preprocess the decoded video frames before inputting them into the neural network for AI inference. Similarly, in related technologies, this data preprocessing is generally implemented by the host CPU. To better achieve pipelined operation of each sub-network, in some examples, the chip disclosed herein may include a third processing unit. This third processing unit performs a third processing on the video frame to be processed and outputs it to the first processing unit, so that the first processing unit can perform a first processing on the third-processed video frame. The third processing can also be called preprocessing or pre-processing, and may include at least one of image data preprocessing such as adjusting brightness, adjusting contrast, adjusting size, image segmentation, and normalization. In other words, by adding a third processing unit to the AI ​​accelerator card to replace the host CPU in performing model preprocessing tasks, this disclosure allows the "model preprocessing - model inference - model postprocessing" process to be executed within the AI ​​accelerator card, achieving better pipelined operation. Simultaneously, it reduces the consumption of host CPU resources and the frequency of host CPU access to the AI ​​accelerator card's memory.

[0061] See Figure 4 In embodiments including preprocessing and postprocessing, the chip of this disclosure embodiment can implement a pipelined process of preprocessing, model processing, and postprocessing. In some embodiments, the video frame pic1 obtained after video decoding can be output to CU1, where it undergoes preprocessing, model processing based on a first set of sub-networks, and postprocessing in sequence. The video frame pic2 output by the postprocessing unit in CU1 can be further output to the preprocessing unit of CU2. Similarly, CU2 can perform preprocessing, model processing based on a second set of sub-networks, and postprocessing in sequence, and output the processing result. This process continues until the final output result is obtained. It should be noted that a video frame output by the postprocessing unit in one CU can be used as a video frame to be processed by the preprocessing unit in the next CU.

[0062] In addition to pre- and post-processing tasks, the chip disclosed herein can also reduce the resource consumption of the host CPU in video decoding. In some examples, the chip disclosed herein may also include a video decoding unit, which can be used to decode video under the control of the control unit to obtain video frames to be processed, and output the video frames to be processed to the memory unit. The video decoding unit may include a video codec, that is, a program or device capable of compressing or decompressing video. With this configuration, the host CPU only needs to transmit the video to be processed to the memory of the AI ​​accelerator card, and the control unit in the AI ​​accelerator card can control the video decoding unit to decode the video and output the decoded video frames to be processed to the memory unit for subsequent processing by the first processing unit, the second processing unit, and the third processing unit, etc. In this way, the video decoding business is offloaded to the AI ​​accelerator card, reducing the access frequency between the host CPU and the memory of the AI ​​accelerator card. It should be noted that, in an optional embodiment, the video decoding unit may send a first interrupt signal to the control unit after the video decoding is completed, so that the control unit responds to the first interrupt signal by scheduling network parameters to the first processing unit. In other words, the video decoding unit can notify the control unit of the completion of video decoding in the form of an interrupt signal, thereby improving the processing efficiency inside the AI ​​accelerator card.

[0063] In some examples, the control unit can immediately control the video decoding unit to begin video decoding after the AI ​​accelerator card is powered on; in other examples, the control unit can control the video decoding unit to begin video decoding in response to receiving a control command from an external processing unit. This external processing unit can be a Host CPU. In other embodiments, the external processing unit can also be a device other than the AI ​​accelerator card, such as an external device connected to the terminal where the AI ​​accelerator card is located. The external processing unit sends a control command to the control unit to trigger the control unit to control the video decoding unit to begin video decoding. This allows the video decoding function of the AI ​​accelerator card to be enabled when needed, thereby triggering the subsequent AI inference function of the AI ​​accelerator card. It should be noted that in this disclosure, "Host CPU" refers to the main processor on the Host side. In practical applications, the main processor can also be other types of processors besides a CPU, such as an MPU (Microprocessor Unit), etc., and this disclosure does not impose any limitations on this.

[0064] The chip provided in this embodiment is applied to an accelerator card. The memory unit of the accelerator card stores the network parameters of each sub-network in the multiple sub-networks included in the neural network. The chip controls and schedules each sub-network of the neural network to the first processing unit for AI inference through the control unit in the chip. In this way, the task of controlling and scheduling each sub-network is decentralized to the accelerator card. The data transmission between each sub-network no longer needs to rely on the control and scheduling of the host CPU. The multiple sub-networks included in the neural network realize pipelined processing, thereby improving processing efficiency.

[0065] In some practical applications, the neural network in this disclosure embodiment can be a neural network for face detection, which may include a sub-network for target tracking and a sub-network for attribute analysis of face images. Of course, besides the above-mentioned application scenarios, the solutions in this disclosure embodiment can also be used in other application scenarios, which will not be listed here.

[0066] To provide a more detailed description of the chip in this embodiment, a specific embodiment will be described below:

[0067] The chip in this embodiment is applied to an AI accelerator card, which is inserted into the motherboard slot of the terminal. The scenario in this embodiment is that the terminal uses neural network technology to perform face recognition on received video. In this scenario, multiple steps are required to extract features, including video decoding, face region recognition, and local face feature analysis, before the final face recognition can be achieved. This involves N (N is a positive integer greater than 1) sub-networks. Taking sub-network 1 and sub-network 2 as an example, each sub-network is a group. Sub-network 1 is a neural network used for face region recognition, and sub-network 2 is a neural network used for local face feature analysis.

[0068] In related technologies, AI accelerator cards are generally used as pure slave devices to assist the host CPU in its work. The process of the terminal executing AI inference includes:

[0069] The host CPU decodes the video to obtain frame data;

[0070] The host CPU downloads the frame data and subnet 1 into the memory of the AI ​​accelerator card;

[0071] The AI ​​accelerator card performs model inference on frame data through sub-network 1 and stores the results in memory;

[0072] The host CPU reads the output of subnet 1 from the memory of the AI ​​accelerator card;

[0073] The host CPU processes the output of subnet 1;

[0074] The host CPU downloads the processed data and subnet 2 into the memory of the AI ​​accelerator card;

[0075] The AI ​​accelerator card performs model inference on the processed data through sub-network 2 and stores the results in memory;

[0076] The host CPU reads the inference results stored in the memory of the AI ​​accelerator card.

[0077] The above process involves a large number of memory access operations between the host and the accelerator card, which requires a lot of host CPU resources. Different sub-networks cannot be processed in a pipelined manner, resulting in low efficiency of AI inference.

[0078] In this embodiment, improvements have been made to the internal structure of the AI ​​accelerator card. The improved AI accelerator card includes an MCU (i.e., the aforementioned control unit), DRAM (i.e., the aforementioned memory unit), a Video Codec (i.e., the aforementioned video decoding unit), a PCIe, and several CUs (which may include the aforementioned first processing unit). The DRAM is the memory of the AI ​​accelerator card, used to store the network parameters of the neural network and the video frames before and after processing. The MCU is used to schedule the model's network parameters into the CUs and to control and schedule the Video Codec and JPEG Codec. Each CU is used for model inference based on the neural network on the video frames. Each CU also includes a pre-processing unit and a post-processing unit. The pre-processing unit is responsible for the model's pre-processing tasks, and the post-processing unit is responsible for the model's post-processing tasks. The Video Codec is a video codec used to decode the video. The PCIe is the interaction path between the AI ​​accelerator card and the host CPU, used to implement command issuance and data transmission between the AI ​​accelerator card and the host CPU. The video stream can also be transmitted to the AI ​​accelerator card's memory through the PCIe. Furthermore, within the AI ​​accelerator card, the above modules are connected via NoC (Network On...) Interconnection is achieved through interconnect buses such as chip (network on chip) and memory.

[0079] In this embodiment, the process of the terminal performing AI inference includes:

[0080] The host CPU transmits the video to be processed, the network parameters of subnetwork 1, and the parameters of subnetwork 2 to the memory of the AI ​​accelerator card;

[0081] The MCU inside the AI ​​acceleration card controls the Video Codec to decode the video in memory, and controls the preprocessing unit to perform data preprocessing on the decoded video frames (e.g., resizing, image segmentation, and data normalization) before storing the resulting large frame image in memory.

[0082] The MCU in the AI ​​acceleration card controls the CU to read the decoded frame image and transmit it to sub-network 1. Then, sub-network 1 is enabled to start the first stage of model inference, and the result data is transmitted to the post-processing unit.

[0083] The MCU in the AI ​​acceleration card controls the post-processing unit to perform image segmentation on the output data of sub-network 1, obtain frame small images that meet the requirements (e.g., remove background features and retain local facial features, etc.), and store the frame small images in memory.

[0084] The MCU in the AI ​​acceleration card controls the CU to read the frame image and transmit it to sub-network 2. Then, sub-network 2 is enabled to start the second stage of model inference, and the resulting data is transmitted to the post-processing unit.

[0085] The post-processing unit of subnetwork 2 performs post-processing and obtains the final processing result.

[0086] It should be noted that this embodiment uses subnetwork 1 and subnetwork 2 as examples, but in actual applications, the number of subnetwork groups can be greater than or equal to 3, that is, there can be multiple neural networks. The model inference process for subnetworks after subnetwork 2 is similar to the model inference process for subnetwork 2, and will not be described again in this embodiment. Moreover, in this embodiment, different CUs are responsible for different subnetworks, which constitute a hardware pipeline operation, and the MCU is responsible for the control and scheduling of the pipeline operation.

[0087] The solution in this embodiment has at least the following advantages:

[0088] (1) By offloading video decoding, image encoding, model pre- and post-processing and other services to the AI ​​acceleration card, the host CPU only needs to transmit the relevant models and video streams to the AI ​​acceleration card, which greatly reduces the workload of the host CPU.

[0089] (2) All models are downloaded to the AI ​​accelerator card at once. The MCU in the AI ​​accelerator card controls and schedules the process, realizing the pipeline between multiple models. Moreover, the "model preprocessing - model inference - model postprocessing" is executed inside the AI ​​accelerator card, achieving better pipeline processing. Furthermore, different video frames can be processed in a pipelined manner between multiple CUs.

[0090] (3) Data transfer between models no longer needs to rely on Host CPU control and scheduling, which reduces the frequency of Host CPU memory access to AI accelerator card, thus further improving processing efficiency.

[0091] Experiments have shown that the terminal display results obtained by the solution in this embodiment effectively avoid frame drops compared to the terminal display results obtained by related technical solutions, and the terminal display is smoother.

[0092] Corresponding to the aforementioned chip embodiments, this disclosure also provides embodiments of AI accelerator cards and corresponding devices. For example... Figure 5 The diagram shown is a schematic representation of an accelerator card according to an exemplary embodiment of this disclosure. The accelerator card includes:

[0093] Memory unit 501 is used to store the network parameters of the neural network; and

[0094] Chip 502.

[0095] The chip 502 described above can be any of the chips described in the foregoing embodiments.

[0096] like Figure 6 The diagram shown is a schematic of an electronic device according to an exemplary embodiment of the present disclosure. The electronic device includes an accelerator card 601 and an external processing unit 602. The accelerator card 601 may be an accelerator card as described in any of the foregoing embodiments. The external processing unit 602 may be a Host CPU as described in the foregoing embodiments, used to output the network parameters of the neural network to the memory unit.

[0097] The specific implementation process of the functions and roles of each component in the aforementioned accelerator card and electronic device is detailed in the implementation process of the corresponding components in the aforementioned chip, and will not be repeated here. It should be noted that other improvements corresponding to the aforementioned chip also apply to the accelerator card and electronic device.

[0098] See Figure 7 This disclosure also provides a data processing method, which is applied to a control unit in a chip according to any embodiment of this disclosure; the method includes:

[0099] Step 701: Schedule the network parameters of one set of subnetworks from the multiple sets of subnetworks included in the neural network to the first processing unit;

[0100] Step 702: After the first processing is completed, schedule the network parameters of the next group of sub-networks in the multiple groups of sub-networks to the first processing unit.

[0101] In some embodiments, scheduling the network parameters of the next group of sub-networks in the plurality of sub-networks to the first processing unit when the first processing is completed includes: receiving an interrupt signal sent by the first processing unit, wherein the first processing unit sends the interrupt signal when the first processing of the video frame to be processed based on the network parameters of the sub-network currently scheduled to the processing unit is completed; and responding to the interrupt signal, scheduling the network parameters of the next group of sub-networks to the first processing unit.

[0102] In some embodiments, the number of the first processing units is greater than 1, and each first processing unit is used to perform the first processing on the video frame to be processed based on the network parameters scheduled to the processing unit; the network parameters of two adjacent sub-networks are scheduled to different first processing units.

[0103] In some embodiments, scheduling the network parameters of the next group of sub-networks in the plurality of sub-networks to the first processing unit when the first processing is completed includes: receiving an interrupt signal sent by the first processing unit, wherein the first processing unit sends the interrupt signal when the first processing of the video frame to be processed based on the network parameters of the sub-network currently scheduled to the processing unit is completed; and responding to the interrupt signal, scheduling the network parameters of the next group of sub-networks to a first processing unit other than the first processing unit.

[0104] In some embodiments, the method further includes: sending an enable signal to the first processing unit, wherein the first processing unit performs the first processing upon receiving the enable signal.

[0105] The steps performed by the control unit in the above method embodiment are detailed in the functions performed by the control unit in the aforementioned chip embodiment, and will not be repeated here.

[0106] See Figure 8 This disclosure also provides a data processing apparatus, which is applied to a control unit in a chip according to any embodiment of this disclosure; the apparatus includes:

[0107] The first scheduling module 801 is used to schedule the network parameters of one set of sub-networks from the multiple sets of sub-networks included in the neural network to the first processing unit.

[0108] The second scheduling module 802 is used to schedule the network parameters of the next group of sub-networks in the multiple groups of sub-networks to the first processing unit after the first processing is completed.

[0109] In some embodiments, the second scheduling module is configured to: receive an interrupt signal sent by the first processing unit, wherein the first processing unit sends the interrupt signal after completing the first processing of the video frame to be processed based on the network parameters of the sub-network currently scheduled to the processing unit; and in response to the interrupt signal, schedule the network parameters of the next set of sub-networks to the first processing unit.

[0110] In some embodiments, the number of the first processing units is greater than 1, and each first processing unit is used to perform the first processing on the video frame to be processed based on the network parameters scheduled to the processing unit; the network parameters of two adjacent sub-networks are scheduled to different first processing units.

[0111] In some embodiments, the second scheduling module is configured to: receive an interrupt signal sent by the first processing unit, wherein the first processing unit sends the interrupt signal after completing the first processing of the video frame to be processed based on the network parameters of the sub-network currently scheduled to the processing unit; and in response to the interrupt signal, schedule the network parameters of the next set of sub-networks to a first processing unit other than the first processing unit.

[0112] In some embodiments, the apparatus further includes: a sending module, configured to send an enable signal to the first processing unit, wherein the first processing unit performs the first processing upon receiving the enable signal.

[0113] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0114] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any embodiment of this disclosure. Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0115] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0116] The modules or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0117] The embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims

1. A chip, characterized by The chip comprises: a control unit configured to schedule network parameters of a group of sub-networks included in a neural network to a first processing unit; the first processing unit is configured to perform first processing on a to-be-processed video frame in a video based on the network parameters of the sub-network scheduled to the first processing unit, to obtain a first video frame; the control unit is further configured to, in a case where the first processing is completed, schedule network parameters of a next group of sub-networks in the plurality of groups of sub-networks to the first processing unit; the number of the first processing units is greater than 1, and the control unit is configured to schedule network parameters of two adjacent groups of sub-networks to different first processing units; after the first processing unit performs the first processing on the to-be-processed video frame based on the network parameters scheduled to the first processing unit to obtain the first video frame, the first processing unit is further configured to pass the first video frame to another first processing unit, wherein a sub-network scheduled to the another first processing unit is adjacent to a sub-network scheduled to the first processing unit.

2. The chip according to claim 1, characterized in that, the first processing unit is configured to, in a case where the first processing based on the network parameters of the sub-network currently scheduled to the first processing unit is completed, send an interrupt signal to the control unit; the control unit is configured to, in a case where the interrupt signal is received, schedule the network parameters of the next group of sub-networks to the first processing unit.

3. The chip of claim 1, wherein Each first processing unit is configured to: in a case where the first processing of the first processing unit is completed, send an interrupt signal to the control unit, so that the control unit schedules the network parameters of the next group of sub-networks to another first processing unit.

4. The chip according to any one of claims 1 to 3, characterized in that the control unit is further configured to: send an enable signal to the first processing unit, and the first processing unit performs the first processing in a case where the enable signal is received.

5. The chip according to any one of claims 1 to 3, characterized in that The chip further comprises: a second processing unit configured to perform second processing on the first video frame to obtain a second video frame and output the second video frame.

6. The chip according to any one of claims 1 to 3, wherein The chip further comprises: a third processing unit configured to perform third processing on the to-be-processed video frame and output the to-be-processed video frame after the third processing to the first processing unit, so that the first processing unit performs first processing on the to-be-processed video frame after the third processing.

7. An accelerator card, comprising: The accelerator card comprises: a memory unit configured to store network parameters of each group of sub-networks included in a neural network; and the chip of any one of claims 1 to 6.

8. An electronic device, comprising: The electronic device comprises: the accelerator card of claim 7; and an external processing unit configured to output the network parameters of each group of sub-networks included in the neural network to the memory unit.

9. A data processing method, characterized by, The method is applied to a control unit in the chip of any one of claims 1 to 6, and the number of first processing units in the chip is greater than 1; the method comprises: scheduling network parameters of a group of sub-networks included in a neural network to a first processing unit; in a case where the first processing is completed, scheduling network parameters of a next group of sub-networks in the plurality of groups of sub-networks to another first processing unit.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the steps in the method of claim 9.

Images