Chip, accelerator card, electronic device, and data processing method

Abstract

The embodiments of the present disclosure provide a chip, an accelerator card, an electronic device, and a data processing method, which implement the scheduling of network parameters through a control unit inside the chip, and implement the post-processing of the first video frame after neural network processing through a second processing sub-unit inside the chip. In this way, the scheduling and post-processing of the Host CPU are moved down to the chip, effectively reducing the frequency of the Host CPU's memory access to the accelerator card and reducing the data processing pressure on the Host CPU. In addition, the present disclosure uses an on-chip cache unit to cache the first video frame after neural network processing, eliminating the need to store the first video frame in the memory unit of the accelerator card, thereby reducing the memory usage of the accelerator card, achieving memory optimization of the accelerator card, and reducing the memory pressure on the accelerator card.

Description

Technical Field

[0001] The present disclosure relates to the field of chip technology, and in particular to a chip, an accelerator card, an electronic device, and a data processing method. Background Art

[0002] In recent years, neural networks have been widely used in various fields, including image processing, fault diagnosis, and video security. Related technologies typically require the host's main processor (e.g., the host CPU) to work with an accelerator card to train and reason about neural networks. Consequently, during AI training and reasoning, the host CPU frequently accesses the accelerator card's memory, and large amounts of data need to be transferred between the accelerator card's memory and the host CPU. This places significant pressure on both the accelerator card's memory and the host CPU's data processing. Summary of the Invention

[0003] In a first aspect, an embodiment of the present disclosure provides a chip, comprising: a control unit, an on-chip cache unit, and a processing unit, wherein the processing unit comprises a first processing sub-unit and a second processing sub-unit; the control unit is used to schedule network parameters of a neural network to the first processing sub-unit; the first processing sub-unit is used to perform a first processing on a video frame to be processed based on the network parameters scheduled to the processing unit to obtain a first video frame, and output the first video frame to the on-chip cache unit for caching; the second processing sub-unit is used to read the first video frame from the on-chip cache unit, perform a second processing on the read first video frame, obtain a second video frame, and output it.

[0004] In some embodiments, the processing unit further includes a third processing sub-unit, which is used to perform a third processing on the video frame to be processed and then output it to the on-chip cache unit for caching; the first processing sub-unit is used to read the video frame to be processed that has undergone the third processing from the on-chip cache unit, and perform a first processing on the video frame to be processed that has undergone the third processing.

[0005] In some embodiments, the chip further includes: a video decoding unit, which is used to perform video decoding on the input video under the control of the control unit to obtain the video frame to be processed.

[0006] In some embodiments, the control unit is further used to filter out a first target video frame from the video frames to be processed based on a first preset condition; the chip also includes: a first image encoding unit, used to perform image compression on the first target video frame, and the compressed first target video frame is used to output to an external processing unit.

[0007] In some embodiments, the control unit is further used to filter out a second target video frame from the second video frame based on a second preset condition; the chip also includes: a second image encoding unit, used to perform image compression on the second target video frame, and the compressed second target video frame is used to output to an external processing unit.

[0008] In some embodiments, the number of processing units is greater than 1, and the first subunit in the i+1th processing unit is used to read the second video frame output by the second processing subunit in the i-th processing unit, and perform the first processing on the second video frame output by the second processing subunit in the i-th processing unit as the video frame to be processed, where i is a positive integer.

[0009] In some embodiments, the neural network includes multiple sub-networks, and the network parameters of each sub-network are stored in a memory unit of the accelerator card; the control unit is used to schedule the network parameters of the neural network from the memory unit to the first processing sub-unit.

[0010] In some embodiments, the i-th processing unit uses the i-th sub-network included in the neural network to perform the first processing.

[0011] In some embodiments, the memory unit includes a plurality of partitions; and network parameters of at least two sub-networks among the plurality of sub-networks are stored in different partitions.

[0012] In a second aspect, an embodiment of the present disclosure provides an acceleration card, comprising: a memory unit for storing network parameters of a neural network; and a chip as described in any embodiment.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: the acceleration card described in the second aspect; and an external processing unit for outputting network parameters of the neural network to the memory unit.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a data processing method, which is applied to the chip described in any embodiment of the present disclosure, and the method includes: the control subunit dispatches the network parameters of the neural network to the first processing subunit; the first processing subunit performs a first processing on the video frame to be processed based on the network parameters dispatched to this processing unit to obtain a first video frame, and outputs the first video frame to the on-chip cache unit for caching; the second processing subunit reads the first video frame from the on-chip cache unit, performs a second processing on the read first video frame, obtains a second video frame, and outputs it.

[0015] In some embodiments, the processing unit also includes a third processing sub-unit, which is used to perform a third processing on the video frame to be processed and then output it to the on-chip cache unit for caching; the method also includes: the first processing sub-unit reads the video frame to be processed that has undergone the third processing from the on-chip cache unit, and performs a first processing on the video frame to be processed that has undergone the third processing.

[0016] In some embodiments, the method further includes: the control unit controlling the video decoding unit to perform video decoding on the input video to obtain the video frame to be processed.

[0017] In some embodiments, the method also includes: the control unit filters out a first target video frame from the video frames to be processed based on a first preset condition, and outputs the first target video frame to a first image encoding unit for image compression, and the compressed first target video frame is used to output to an external processing unit.

[0018] In some embodiments, the method also includes: the control unit filters out a second target video frame from the second video frame based on a second preset condition, and outputs the second target video frame to a second image encoding unit for image compression, and the compressed second target video frame is used to output to an external processing unit.

[0019] In some embodiments, the number of processing units is greater than 1, and the first subunit in the i+1th processing unit is used to read the second video frame output by the second processing subunit in the i-th processing unit, and perform the first processing on the second video frame output by the second processing subunit in the i-th processing unit as the video frame to be processed, where i is a positive integer.

[0020] In some embodiments, the neural network includes multiple sub-networks, and the network parameters of each sub-network are stored in a memory unit of the accelerator card; the method also includes: the control unit dispatching the network parameters of the neural network from the memory unit to the first processing sub-unit.

[0021] In some embodiments, the i-th processing unit uses the i-th sub-network included in the neural network to perform the first processing.

[0022] In some embodiments, the memory unit includes a plurality of partitions; and network parameters of at least two sub-networks among the plurality of sub-networks are stored in different partitions.

[0023] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method executed by the control unit in the chip described in any embodiment of the present disclosure.

[0024] The embodiment of the present disclosure implements the scheduling of network parameters through the control unit inside the chip, and implements the post-processing of the first video frame after the neural network processing through the second processing sub-unit inside the chip. In this way, the scheduling and post-processing of the Host CPU are sunk to the chip, effectively reducing the frequency of the Host CPU's memory access to the accelerator card and reducing the data processing pressure of the Host CPU. In addition, the present disclosure adopts an on-chip cache unit to cache the first video frame after the neural network processing, without the need to store the first video frame in the memory unit of the accelerator card, thereby reducing the memory usage of the accelerator card, realizing the memory optimization of the accelerator card, and reducing the memory pressure of the accelerator card.

[0025] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into the specification and constitute a part of the present disclosure. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0027] Figure 1 is a schematic diagram of a chip according to an embodiment of the present disclosure;

[0028] Figure 2 This is a schematic diagram of the interaction process between the host CPU and the accelerator card in the related art;

[0029] Figure 3 is a schematic diagram of the interaction process between the Host CPU and the accelerator card according to an embodiment of the present disclosure;

[0030] Figure 4 It is a timing diagram of the AI ​​reasoning process in related technologies;

[0031] Figure 5 Schematic diagram of the hardware structure of the accelerator card according to an embodiment of the present disclosure;

[0032] Figure 6 is a timing diagram of the AI ​​reasoning process of an embodiment of the present disclosure;

[0033] Figure 7 is a schematic diagram of an accelerator card according to an exemplary embodiment of the present disclosure;

[0034] Figure 8 is a schematic diagram illustrating a data transmission process of an accelerator card according to an exemplary embodiment of the present disclosure;

[0035] Figure 9 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure;

[0036] Figure 10 It is a schematic diagram of a data processing method according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0038] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0039] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0040] In recent years, neural networks have been widely used in various fields such as image processing, fault diagnosis, and video security. In the field of neural network technology, the process of training a neural network model through sample data can be called AI training, and the process of processing an object using a neural network algorithm can be called AI reasoning. In related technologies, terminals generally use accelerator cards that are more suitable for performing computing tasks to complete AI training / reasoning. However, during the AI ​​training / reasoning process, there are a large number of memory access operations of the Host CPU to the accelerator card, and a large amount of data needs to be transferred between the memory of the accelerator card and the Host CPU, resulting in high memory pressure on the accelerator card and data processing pressure on the Host CPU.

[0041] Based on this, an embodiment of the present disclosure provides a chip for solving the above-mentioned problem.

[0042] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure are further described in detail below with reference to the accompanying drawings.

[0043] like Figure 1 FIG. 1 is a schematic diagram of a chip according to an embodiment of the present disclosure. The chip includes:

[0044] A control unit 101, an on-chip cache unit 102, and a processing unit 103, wherein the processing unit 103 includes a first processing sub-unit 1031 and a second processing sub-unit 1032;

[0045] The control unit 101 is used to dispatch the network parameters of the neural network to the first processing sub-unit 1031;

[0046] The first processing sub-unit 1031 is configured to perform a first process on the video frame to be processed based on the network parameters scheduled to the processing unit to obtain a first video frame, and output the first video frame to the on-chip cache unit 102 for caching;

[0047] The second processing sub-unit 1032 is configured to read the first video frame from the on-chip cache unit 102 , perform second processing on the read first video frame, obtain a second video frame, and output the second video frame.

[0048] In an embodiment of the present disclosure, the network parameters of the neural network can be stored in the memory unit of the accelerator card, and the control unit 101 can dispatch the network parameters of the neural network from the memory unit to the first processing sub-unit 1031. The subsequent embodiments are all described by taking the AI ​​accelerator card as an example. The AI ​​accelerator card is a processor product specially designed to accelerate the execution of AI algorithms. The AI ​​accelerator card can be a circuit board including hardware modules such as chips for performing AI calculations and communication interfaces. The chip for performing AI calculations can be any one of a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit), or other types of chips. The present disclosure does not limit this. The AI ​​accelerator card can be inserted into a slot and communicate with the host CPU through a communication interface, such as a PCIe (Peripheral Component Interconnect Express) interface. Of course, in other embodiments, the AI ​​accelerator card can also use other types of communication interfaces, such as a QPI (QuickPath Interconnect) interface, as an interaction channel with the host CPU.

[0049] The memory unit may be DRAM (Dynamic Random Access Memory) or SDRAM (Synchronous Dynamic Random Access Memory), or other types of memory, which are not limited in this disclosure. In this embodiment, the memory unit is used to store network parameters of the neural network, which may include operators (e.g., addition operators, multiplication operators, etc.) and parameters required for the operators to perform operations.

[0050] The processing unit 103 is a functional module in the AI ​​accelerator card that performs AI computing tasks and can be called a computing unit (CU). The processing unit may include a first processing subunit and a second processing subunit. The first processing subunit may include multiple subunits such as a subunit for performing addition operations, a subunit for performing multiplication operations, and a subunit for performing convolution operations. These subunits can be called reconfigurable clusters. These reconfigurable clusters can be combined to form different computing paths to perform different tasks through reconstruction. The first processing subunit 1031 performs a first processing on the video frame to be processed based on the network parameters of the neural network scheduled to this processing unit. The specific content of the first processing can be determined according to the function implemented by the neural network. For example, a neural network can be used to perform face segmentation on the video frame to be processed. The first processing performed based on the neural network can refer to segmenting a face image from the video frame to be processed, and the first video frame can be the segmented face image. It should be noted that the solution of this embodiment can be used in the training phase of the neural network, and can also be used in the reasoning phase of the neural network. The subsequent content is described using the reasoning phase of the neural network as an example.

[0051] In this embodiment, the operation of the first processing sub-unit 1031 performing the first processing on the video frame to be processed based on the network parameters of the neural network is performed under the control and scheduling of the control unit. Optionally, the control unit may be an MCU (Microcontroller Unit). That is to say, unlike the AI ​​accelerator card that simply serves as a slave device in the related art, this embodiment adds an MCU to the AI ​​accelerator card to replace the Host CPU to control and schedule the neural network, thereby reducing the memory access frequency of the Host CPU to the AI ​​accelerator card. Of course, in other embodiments, the control unit may also be a processor chip of an architecture such as ARM, RISC-V, or PowerPC. In some embodiments, the chip further includes a video decoding unit, which is used to perform video decoding on the input video under the control of the control unit to obtain the video frame to be processed.

[0052] In practical applications, it is often necessary to process video frames processed by neural networks, such as cropping, sharpening, and scaling. These processing operations are called post-processing. In related technologies, because GPUs or AI accelerator cards are limited by hardware conditions and do not support universal processing operations, the post-processing process is generally implemented by the host CPU. In this process, the video frames processed by the neural network need to be output to the memory unit of the AI ​​accelerator card for storage. The host CPU then reads the processed video frames from the memory unit of the AI ​​accelerator card and performs post-processing. The post-processed video frames are then returned to the memory of the AI ​​accelerator card, and the AI ​​accelerator card continues to perform other processing. It can be seen that the above process generates a large number of memory access operations. In cases where the neural network includes multiple sub-networks and the output results of each sub-network need to be post-processed, or in cases where the neural network needs to process the input video frames multiple times and the output results of each processing need to be post-processed, as the number of sub-networks increases or the total number of times the neural network processes the input video frames increases, the number of memory accesses and the number of post-processing operations performed by the host CPU will also increase, which will increase the memory pressure of the AI ​​accelerator card and the data processing pressure of the host CPU.

[0053] However, this embodiment adds a second processing sub-unit to the AI ​​accelerator card to replace the Host CPU to perform model post-processing tasks, so that the post-processing operations performed on the Host CPU can be sunk to the chip, thereby reducing the data processing pressure of the Host CPU. At the same time, it also reduces the frequency of the Host CPU's access to the AI ​​accelerator card memory, reducing the data transmission pressure of the memory unit of the AI ​​accelerator card. In addition, the present disclosure uses an on-chip cache unit to cache the first video frame after neural network processing. There is no need to store the first video frame in the memory unit of the AI ​​accelerator card, thereby reducing the memory usage of the AI ​​accelerator card, realizing the memory optimization of the AI ​​accelerator card, and reducing the data storage pressure of the memory of the AI ​​accelerator card. The second processing may include at least one of the image processing such as cropping processing, sharpening processing, rotation processing, scaling processing, transparency processing, NMS (Non-Maximum Suppression, non-maximum suppression) and the like.

[0054] In order to facilitate understanding of the technical effects of the embodiments of the present disclosure, the following describes the related technologies and the interaction process between the Host CPU and the accelerator card in the embodiments of the present disclosure in conjunction with the accompanying drawings. Figure 2As shown, taking the AI ​​accelerator card as an example, it is a schematic diagram of the interaction process between the Host CPU and the AI ​​accelerator card in the related art. Assuming that the input video frame needs to be processed N times (N is a positive integer) by the neural network, the Host CPU decodes the video stream through the video decoding module and passes the decoded video frame pic1 to the memory unit of the AI ​​accelerator card; the model inference module in the AI ​​accelerator card processes the video frame pic1 based on the neural network, and stores the processed video frame pic1A in the memory unit; the Host CPU reads the video frame pic1A from the memory unit of the AI ​​accelerator card to perform model post-processing through the model post-processing unit; after that, the Host CPU passes the video frame pic1B that has been post-processed by the model to the memory of the AI ​​accelerator card, and the AI ​​accelerator card processes the video frame pic1B again based on the neural network; and so on, until the Host CPU obtains the desired result (denoted as pic1N in the figure). In this process, the more times the neural network is processed, the more memory access operations are performed on the AI ​​accelerator card.

[0055] Taking the AI ​​accelerator card as an example, Figure 3 The figure shows a schematic diagram of the interaction process between the host CPU and the AI ​​accelerator card according to an embodiment of the present disclosure. The host CPU transmits the video stream to the memory unit of the AI ​​accelerator card via a PCIe or Ethernet interface. The video decoding unit reads the video stream from the memory unit and decodes it to obtain video frame pic2. The first processing subunit of the AI ​​accelerator card processes video frame pic2 based on a neural network and transmits the processed video frame pic2A to the on-chip cache unit. The second processing subunit obtains video frame pic2A from the on-chip cache unit, performs model post-processing on video frame pic2A, and transmits the model-post-processed video frame pic2B to the first processing subunit. The above process is repeated until the second processing subunit outputs the required video frame pic2N and outputs video frame pic2N to the memory unit, which then transmits video frame pic2N back to the host CPU. In this process, no matter how many times the neural network processes, the host CPU only needs to transmit the video frame to be processed to the memory of the AI ​​accelerator card and read the final AI inference result from the memory of the AI ​​accelerator card. The memory access frequency is low, and the processing pressure on the host CPU and the data transmission pressure between the host CPU and the memory of the AI ​​accelerator card are relatively small. At the same time, since the intermediate processing results of the first processing sub-unit and the second processing sub-unit can be cached by the on-chip cache unit, the data storage pressure of the memory unit is reduced.

[0056] In some application scenarios, multiple neural networks may need to work together to perform a specific task. For example, in the beautification scenario, it is necessary to detect the face from the video frame through the face detection network, and then detect the position of the facial features from the face image through the facial features detection network, and then beautify the detected facial features through the facial features beautification network. Therefore, in some examples, the neural network may include multiple sub-networks, each of which can be an independent neural network or one or more network layers of the neural network. In this case, the memory unit of the AI ​​accelerator card can be used to store the network parameters of each sub-network in the multiple sub-networks included in the neural network. A sub-network can map the input to the output through its corresponding network parameters. Different sub-networks can achieve different functions. For example, in the above-mentioned beautification scenario, the sub-networks included in the neural network are the face detection network, the facial features detection network and the facial features beautification network. It should be noted that the type of a sub-network can be any one of CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), etc. The types of different sub-networks can be the same or different, and this disclosure does not limit this.

[0057] In some examples, the memory unit can be configured with multiple partitions, with network parameters for different subnetworks stored in different partitions. In other words, the memory unit's storage space can adopt a hierarchical partition structure to implement partitioned storage management of the network parameters for each subnetwork. Furthermore, to facilitate access to the network parameters of a specified subnetwork, a partition table can be created. This partition table can record information such as the starting address, size, and corresponding subnetwork identifier of each partition. Other units can then determine the storage address of a specified subnetwork by searching the partition table and, based on this storage address, read the network parameters for the specified subnetwork. Alternatively, different partition identifiers can be set for different subnetworks. The memory unit determines a partition path based on the subnetwork's partition identifier and stores the subnetwork's network parameters in the partition corresponding to the partition path. Correspondingly, other units can also retrieve the network parameters for the specified subnetwork from the memory unit based on the subnetwork's partition identifier. Of course, in other embodiments, other storage management methods can also be configured based on the needs of specific scenarios.

[0058] In the case where the neural network includes multiple sub-networks, the first processing sub-unit can perform a first processing on the video frame to be processed based on the network parameters of the sub-network dispatched to this processing unit. The specific content of the first processing can be determined according to the function implemented by the sub-network. In some examples, the control unit can dispatch the network parameters to the first processing sub-unit by sending an enable signal to the first processing sub-unit. The enable signal is similar to a trigger signal, which can trigger the first processing sub-unit to read the network parameters and then perform the first processing on the video frame to be processed based on the read network parameters of the sub-network. Optionally, after enabling the first processing sub-unit, the control unit can configure the storage address of the network parameters of the sub-network to be read for the first processing sub-unit, so that the first processing sub-unit reads the corresponding data under the storage address.

[0059] In an optional embodiment, if the neural network includes multiple sub-networks connected in series, that is, the function of the neural network is realized based on the functions of multiple sub-networks, and the output of one sub-network is the input of the next sub-network, the control unit can first dispatch the network parameters of the first sub-network to the first processing sub-unit, and then dispatch the network parameters of the second sub-network to the first processing sub-unit, and so on, to ensure the orderly operation of each sub-network. In another optional embodiment, when only one neural network is included, the control unit can also dispatch the network parameters of the neural network to the first processing sub-unit and enable the first processing sub-unit multiple times, so that the first processing sub-unit processes the video frames to be processed multiple times based on the neural network, thereby improving the effect of AI reasoning.

[0060] In order to ensure the reasonable scheduling of the network parameters of multiple sub-networks by the control unit, in some examples, the first processing sub-unit may send an interrupt signal to the control unit when the current first processing is completed; the control unit, upon receiving the interrupt signal, schedules the network parameters of the next sub-network to the first processing sub-unit. That is, when the first processing sub-unit completes the processing of the video frame to be processed based on one sub-network, it may notify the control unit by sending an interrupt signal to the control unit so that the control unit schedules the network parameters of the next sub-network. In this way, the first processing sub-unit can process the video frame to be processed based on the network parameters of the next sub-network. In this way, the AI ​​accelerator card can immediately enable the AI ​​reasoning of the second sub-network after completing the AI ​​reasoning of the first sub-network, and so on, until the AI ​​reasoning of the last sub-network is completed, thereby improving processing efficiency.

[0061] In some embodiments, the processing unit further includes a third processing sub-unit for performing a third processing on the video frame to be processed and then outputting the processed video frame to the on-chip cache unit for caching. The first processing sub-unit is used to read the video frame to be processed that has undergone the third processing from the on-chip cache unit, and perform a first processing on the video frame to be processed that has undergone the third processing. The third processing sub-unit may also be referred to as a pre-processing unit or a pre-processing unit, and accordingly, the third processing may be referred to as pre-processing or pre-processing. Specifically, the third processing may include but is not limited to at least one of resizing, image segmentation, and data normalization. In related technologies, the pre-processing process is generally also performed by the Host CPU, while the embodiment of the present disclosure reduces the processing pressure on the Host CPU by sinking the pre-processing to the chip end.

[0062] In the above embodiment, the first processing subunit, the second processing subunit, and the third processing subunit can be functionally divided subunits, that is, the first processing subunit is used to process the video frames to be processed based on the network parameters of the neural network, the second processing subunit is used to perform post-processing, and the third processing subunit is used to perform pre-processing. In practical applications, the number of each subunit can be set to a value greater than or equal to 1 according to actual needs, and this disclosure does not impose any restrictions on this. For example, pre-processing can be performed in sequence by two third processing subunits, and then the video frames finally output by the two third processing subunits are used as the video frames to be processed and input into the first processing subunit for processing.

[0063] In some embodiments, the number of processing units is greater than one, and the first subunit in the (i+1)th processing unit is configured to read the second video frame output by the second processing subunit in the (i)th processing unit from the memory unit, and perform the first processing on the second video frame output by the second processing subunit in the (i)th processing unit as the video frame to be processed, where i is a positive integer. The (i)th processing unit may use the (i)th subnetwork included in the neural network to perform the first processing.

[0064] The number of processing units 103 may be greater than or equal to the number of sub-networks included in the neural network, or may be less than the number of sub-networks included in the neural network. When the number of processing units 103 is greater than or equal to the number of sub-networks included in the neural network, the network parameters of each sub-network may be dispatched to a different processing unit. When the number of processing units 103 is less than the number of sub-networks included in the neural network, the network parameters of some sub-networks may be dispatched to the processing units first, and when a processing unit completes processing, the network parameters of other sub-networks may be dispatched to the processing unit.

[0065] In the above embodiment, the i-th processing unit may be a fixed processing unit, for example, the i-th processing unit is the processing unit numbered XXXXXYi. Alternatively, the i-th processing unit may be the i-th processing unit in an idle state (i.e., not performing any data processing). For example, the processing unit numbered XXXXXY3 is the first processing unit in an idle state, and the processing unit numbered XXXXXY6 is the second processing unit in an idle state, then the processing unit numbered XXXXXY3 is determined as the first processing unit, and the processing unit numbered XXXXXY6 is determined as the second processing unit. However, in other cases, the first processing unit and the second processing unit may also be processing units with other numbers.

[0066] Furthermore, each processing unit 103 may include at least one third processing sub-unit. The third processing sub-unit in the i-th processing unit 103 is configured to pre-process the video frame to be processed and output the pre-processed video frame to be processed to the on-chip cache unit 102. The first processing sub-unit in the i-th processing unit 103 may read the pre-processed video frame to be processed from the on-chip cache unit 102 and perform the first processing. In this way, since the memory unit of the AI ​​accelerator card does not need to store the intermediate data output by the third processing sub-unit, the storage pressure of the memory unit is reduced.

[0067] It should be noted that in the embodiments of the present disclosure, the unprocessed video frame processed by the first processing sub-unit in a processing unit may be a video frame processed by the second processing sub-unit in the previous processing unit. The unprocessed video frame processed by the first processing sub-unit in the first processing unit may be a video frame decoded from a video stream and processed by the third processing sub-unit in the first processing unit.

[0068] Considering that the memory unit of the AI ​​accelerator card needs to store the network parameters of multiple sub-networks, the memory space occupied by the AI ​​accelerator card may be large. Therefore, in order to reduce the memory space occupied, in some examples, the control unit of the chip can be used to filter out a first target video frame from the video frame to be processed based on a first preset condition; the chip can also include a first image encoding unit, which is used to compress the first target video frame and output it to the memory unit, and the compressed first target video frame is used to output it to the external processing unit. The first preset condition is a selection condition for filtering the video frame to be processed. Optionally, the first preset condition can be related to the timestamp, sequence number or object included in the video frame to be processed. For example, the first target video frame can be filtered out from the video frame to be processed at a preset time, or it can be filtered out from the video frame to be processed at a preset number of frames, or it can be a video frame to be processed whose image content includes a specific target. In other words, the chip can filter and compress the decoded video frame to be processed, which can avoid the problem of high overhead caused by uncompressed video frames occupying more storage space. Among them, the first image encoding unit may include an image codec, that is, a program or device that can compress or decompress an image; the external processing unit may be a Host CPU, so that the Host CPU performs corresponding processing on the compressed first target video frame and outputs it to a display component for displaying the video, or the external processing unit may also include a display component for displaying the video, that is, the compressed first target video frame is directly output to the display component for display.

[0069] In an optional embodiment, the control unit may control the first image encoding unit to start image compression in response to an interrupt signal sent by the first processing subunit; the interrupt signal may be sent by the first processing subunit to the control unit when the first processing is completed. In other words, after completing the first processing of the video frame to be processed, the first processing subunit may notify the control unit of the completion of the first processing in the form of an interrupt signal, so that the control unit immediately controls the first image encoding unit to start image compression, thereby improving the processing efficiency within the AI ​​accelerator card.

[0070] In other examples, the chip's control unit can be configured to filter out a second target video frame from the second video frame based on a second preset condition. The chip can also include a second image encoding unit, which compresses the second target video frame and outputs it to a memory unit. The compressed first target video frame is then output to an external processing unit. The second preset condition is a selection condition for filtering the second video frame. Similarly, the second preset condition can be related to the timestamp, sequence number, or object included in the second video frame. For example, the second target video frame can be filtered out from the second video frame at a preset interval, at a preset number of frames, or when the image content includes a specific object. In other words, the chip can also filter and compress video frames after AI inference is completed, thus avoiding the problem of high storage overhead caused by uncompressed video frames occupying more space. Correspondingly, the first image encoding unit can also include an image codec; the external processing unit can be a host CPU or a display component for displaying the video. Similarly, the control unit can control the second image encoding unit to start image compression in response to the interrupt signal sent by the second processing subunit; the interrupt signal can be sent by the second processing subunit to the control unit when the second processing is completed. That is to say, when the second processing subunit completes the second processing of the first video frame, it can notify the control unit of the completion of the second processing in the form of an interrupt signal, so that the control unit immediately controls the second image encoding unit to start image compression, thereby improving the processing efficiency inside the AI ​​accelerator card. The above-mentioned first image encoding unit and the second image encoding unit can be the same image encoding unit, or they can be two independent and different image encoding units.

[0071] After the external processing unit obtains the decoded video frames and / or the video frames after AI inference, it can use these video frames to output the display screen. After improving the processing efficiency of AI inference, the smoothness of the display screen can be improved.

[0072] It should be noted that the first and second preset conditions described above can also be determined based on preset image quality parameters. For example, the first target video frame can be a video frame whose signal-to-noise ratio exceeds a preset signal-to-noise ratio threshold among the video frames to be processed; the second target video frame can be a video frame whose resolution exceeds a preset resolution threshold among the second video frames. Of course, the selected image quality parameters and thresholds can be set based on the needs of the specific scenario, and the first and second preset conditions can be the same or different, and this disclosure is not limited thereto.

[0073] In order to provide a more detailed description of the chip according to the embodiment of the present disclosure, a specific embodiment is introduced below:

[0074] The chip of this embodiment is applied to an AI accelerator card, which is inserted into the motherboard slot of the terminal. The scenario of this embodiment is that the terminal uses neural network technology to perform face recognition on the received video. In this scenario, it is necessary to extract features through multiple steps such as video decoding, face area recognition, and face local feature analysis to achieve the final face recognition, which involves N neural networks. Taking these N neural networks including sub-network 1 and sub-network 2 as an example, sub-network 1 is a neural network for face area recognition, and sub-network 2 is a neural network for face local feature analysis. The network parameters of these two neural networks are stored in the host main memory.

[0075] In related technologies, accelerator cards are generally used as pure slave devices to assist the host CPU. Taking AI accelerator cards as an example, the process of executing AI reasoning on the terminal is as follows: Figure 4 As shown, the process includes:

[0076] S401, Host CPU decodes the video to obtain frame data;

[0077] S402: The host CPU downloads the frame data and network parameters of subnetwork 1 to the memory of the AI ​​accelerator card.

[0078] S403: The AI ​​accelerator card performs model inference on the frame data through sub-network 1 and stores the results in memory.

[0079] S404: The host CPU reads the output of sub-network 1 from the memory of the AI ​​accelerator card.

[0080] S405, the host CPU performs post-processing on the output of sub-network 1;

[0081] S406: The Host CPU downloads the processed data and sub-network 2 to the memory of the AI ​​accelerator card.

[0082] S407: The AI ​​accelerator card performs model inference on the processed data through sub-network 2 and stores the results in memory.

[0083] S408. The Host CPU reads the inference result stored in the memory of the AI ​​accelerator card.

[0084] The above process involves a large number of memory accesses between the host and the accelerator card, which consumes a large amount of host CPU resources and reduces AI inference efficiency. In related technologies, terminals can also use GPUs to perform AI training / inference tasks. However, GPUs often need to wait for the host CPU to perform memory access operations and are therefore idle, resulting in wasted resources and similarly low efficiency.

[0085] In this embodiment, hardware improvements are made inside the accelerator card. Taking the AI ​​accelerator card as an example, Figure 5 This is a hardware structure diagram of an AI accelerator card shown in an embodiment of the present disclosure, wherein the AI ​​accelerator card includes an MCU (i.e. the above-mentioned control unit), DRAM (i.e. the memory unit of the AI ​​accelerator card), a Video Codec (i.e. the above-mentioned video decoding unit), a JPEG Codec (i.e. the above-mentioned first image encoding unit and the second image encoding unit), PCIe and several CUs; ​​wherein, DRAM is the memory unit of the AI ​​accelerator card, which is used to store the network parameters of the neural network and the video frames before and after processing; MCU is used to schedule the network parameters of the model to the CU, and to control and schedule the Video Codec and JPEG Codec; each CU is used to perform model inference on the video frame based on the neural network, and the CU includes a pre-processing unit and a post-processing unit, the pre-processing unit is responsible for the pre-processing tasks of the model, and the post-processing unit is responsible for the post-processing tasks of the model; Video Codec is a video codec for decoding the video; JPEG Codec is an image codec for compressing the video frame; PCIe is an interactive path between the AI ​​accelerator card and the Host CPU, which is used to realize the communication between the AI ​​accelerator card and the Host CPU. Commands are sent between CPUs, data is transmitted, and video streams can be transmitted to the memory of the AI ​​accelerator card through PCIe. In addition, within the AI ​​accelerator card, the above modules are interconnected through interconnection buses such as NoC (Network on Chip) and memory.

[0086] In this embodiment, the process of the terminal performing AI reasoning is as follows: Figure 6 As shown, the process includes:

[0087] S601: The host CPU transfers the video to be processed, the network parameters of subnetwork 1, and the network parameters of subnetwork 2 to the memory of the AI ​​accelerator card.

[0088] S602: The MCU in the AI ​​accelerator card controls the Video Codec to decode the video in the memory, and controls the pre-processing unit to perform data preprocessing (for example, resizing, image segmentation, and data normalization) on the decoded video frames, and then stores the resulting large frame image in the memory.

[0089] S603: The MCU in the AI ​​accelerator card controls the CU to read the decoded large image frame and transmit it to sub-network 1. Sub-network 1 is then enabled to start the first stage of model inference. The resulting data is transmitted to the post-processing unit. Simultaneously, the MCU selects a portion of the large image frame based on the first preset condition and transmits it to the JPEG Codec for image compression. The compressed large image frame is stored in memory and waits for the Host CPU to read it.

[0090] S604: The MCU in the AI ​​accelerator card controls the post-processing unit to perform image segmentation on the output data of model 1, obtain a frame thumbnail that meets the requirements (for example, removing background features and retaining local facial features), and store the frame thumbnail in memory.

[0091] S605: The MCU in the AI ​​accelerator card controls the CU to read the frame thumbnails and transmit them to sub-network 2. Sub-network 2 is then enabled to start the second stage of model inference. The resulting data is transmitted to the post-processing unit. Simultaneously, the MCU selects some frame thumbnails based on the second preset condition and transmits them to the JPEG Codec for image compression. The compressed frame thumbnails are stored in memory and wait for the Host CPU to read them.

[0092] S606: Transmit the compressed large frame image and the compressed small frame image stored in the memory of the AI ​​accelerator card to the host CPU for terminal display. The compressed large frame image and the compressed small frame image can be transmitted in different steps. For example, the compressed large frame image can also be transmitted immediately after the image compression is completed in step S603.

[0093] It should be noted that this embodiment uses subnetwork 1 and subnetwork 2 as examples, but in actual applications, the number of neural networks can be greater than or equal to 3, that is, there can be multiple neural networks. For the inference process of the subnetworks after subnetwork 2, please refer to steps S604 and S605, which will not be repeated in this embodiment. In this embodiment, some parameter settings of the AI ​​accelerator card for video face recognition tasks are shown in Table 1.

[0094] Table 1 Parameter settings

[0095]

[0096] Based on this parameter setting, the comparison of memory usage of the AI ​​accelerator card when using image compression and when not using image compression is shown in Table 2.

[0097] Table 2 Comparison of memory usage

[0098]

[0099] Among them, nodes i1, fb1, fb2, o1, o2, and o3 refer to the video to be processed, the large frame image obtained after data preprocessing, the small frame image obtained after image segmentation, the compressed large frame image, the compressed small frame image, and the image obtained by sub-network 2 model reasoning mentioned in the previous steps respectively; at the same time, the JPG column refers to the memory usage when image compression is used, and the RGB24 column refers to the memory usage when image compression is not used. By comparison, it can be seen that for a video with the same bit rate of 4MB / s, without image compression, the image needs to occupy 2100MB of memory space during the AI ​​reasoning process, resulting in a very slow speed of AI reasoning. During the experiment, only 1 to 2 images can be processed per second; after using image compression technology, the memory space occupied by the image during the AI ​​reasoning process is only 104.8MB, and the speed of AI reasoning is faster. It can be seen that the solution of this embodiment can effectively reduce memory usage.

[0100] The solution of this embodiment has at least the following advantages:

[0101] (1) Services such as video decoding, image encoding, and model pre- and post-processing are moved to the AI ​​accelerator card. The host CPU only needs to transmit the relevant models and video streams to the AI ​​accelerator card, which greatly reduces the workload of the host CPU.

[0102] (2) All network parameters are downloaded to the AI ​​accelerator card at one time. The MCU in the AI ​​accelerator card controls and schedules the pipeline between multiple models. In addition, “model pre-processing-model inference-model post-processing” is executed inside the AI ​​accelerator card, achieving better pipeline processing.

[0103] (3) Data transfer between models no longer needs to rely on the host CPU control scheduling, reducing the host CPU's memory access frequency to the AI ​​accelerator card, thereby further improving processing efficiency;

[0104] (4) Image compression technology is used to select a subset from the large frame image and the small frame image for compression, which reduces the memory space occupied and reduces the PCIe data volume and interaction times, thereby improving processing efficiency.

[0105] It has been found through experiments that the terminal display result obtained by the solution of this embodiment effectively avoids frame drops, and the terminal display image is smoother than the terminal display result obtained by the related technical solution.

[0106] Corresponding to the aforementioned chip embodiments, the present disclosure also provides embodiments of an accelerator card and its corresponding device. Figure 7 FIG. 1 is a schematic diagram of an accelerator card according to an exemplary embodiment of the present disclosure, wherein the accelerator card includes:

[0107] A memory unit 702 for storing network parameters of the neural network; and

[0108] Chip 702.

[0109] The chip 702 can be any of the chips described in the above embodiments. Figure 8 As shown. Model 1 and Model 2 are two sub-networks in the neural network in the aforementioned embodiment. For the sake of simplicity, the above embodiment is described with the number of sub-networks being 2. In other embodiments, the number of sub-networks can also be other numbers, which will not be repeated here.

[0110] like Figure 9 , is a schematic diagram of an electronic device according to an embodiment of the present disclosure, wherein the electronic device includes an accelerator card 901 and an external processing unit 902, wherein the accelerator card 901 can be the accelerator card in any of the aforementioned embodiments; the external processing unit 902 can be the Host CPU in the aforementioned embodiment, which is used to output the network parameters of the neural network to the memory unit.

[0111] The implementation process of the functions and effects of each component in the above-mentioned accelerator card and electronic device is detailed in the implementation process of the corresponding components in the above-mentioned chip, which will not be repeated here. It should be noted that other improvements corresponding to the above-mentioned chip are also applicable to the accelerator card and electronic device.

[0112] like Figure 10 As shown, an embodiment of the present disclosure further provides a data processing method, which is applied to the chip described in any embodiment of the present disclosure, and the method includes:

[0113] Step 1001: the control subunit dispatches the network parameters of the neural network to the first processing subunit;

[0114] Step 1002: The first processing sub-unit performs a first processing on the to-be-processed video frame based on the network parameters dispatched to the processing unit to obtain a first video frame, and outputs the first video frame to the on-chip cache unit for caching;

[0115] Step 1003: The second processing sub-unit reads the first video frame from the on-chip cache unit, performs second processing on the read first video frame, obtains a second video frame, and outputs it.

[0116] In some embodiments, the processing unit also includes a third processing sub-unit, which is used to perform a third processing on the video frame to be processed and then output it to the on-chip cache unit for caching; the method also includes: the first processing sub-unit reads the video frame to be processed that has undergone the third processing from the on-chip cache unit, and performs a first processing on the video frame to be processed that has undergone the third processing.

[0117] In some embodiments, the method further includes: the control unit controlling the video decoding unit to perform video decoding on the input video to obtain the video frame to be processed.

[0118] In some embodiments, the method also includes: the control unit filters out a first target video frame from the video frames to be processed based on a first preset condition, and outputs the first target video frame to a first image encoding unit for image compression, and the compressed first target video frame is used to output to an external processing unit.

[0119] In some embodiments, the method also includes: the control unit filters out a second target video frame from the second video frame based on a second preset condition, and outputs the second target video frame to a second image encoding unit for image compression, and the compressed second target video frame is used to output to an external processing unit.

[0120] In some embodiments, the number of the processing units is greater than 1, and the first subunit in the i+1th processing unit is used to read the second video frame output by the second processing subunit in the i-th processing unit from the memory unit, and perform the first processing on the second video frame output by the second processing subunit in the i-th processing unit as the video frame to be processed, where i is a positive integer.

[0121] In some embodiments, the neural network includes multiple sub-networks, and the network parameters of each sub-network are stored in a memory unit of the accelerator card; the method also includes: the control unit dispatching the network parameters of the neural network from the memory unit to the first processing sub-unit.

[0122] In some embodiments, the i-th processing unit uses the i-th sub-network included in the neural network to perform the first processing.

[0123] In some embodiments, the memory unit includes a plurality of partitions; and network parameters of at least two sub-networks among the plurality of sub-networks are stored in different partitions.

[0124] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method executed by the control unit in the chip described in any embodiment of the present disclosure.

[0125] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0126] Through the description of the above implementation methods, it can be seen that those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the embodiments of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments of the present disclosure.

[0127] The modules or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0128] The embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate. When implementing the embodiments of the present disclosure, the functions of each module may be implemented in the same or multiple software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement the present invention without inventive effort.

[0129] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the scope of protection of the embodiment of the present disclosure.

Claims

1. A chip, characterized in that: The chip is applied to an accelerator card, and the chip includes: A control unit, an on-chip cache unit, and a processing unit, wherein the processing unit includes a first processing sub-unit and a second processing sub-unit; the first processing unit and the second processing unit are communicatively connected to the memory unit of the accelerator card, and the memory unit is communicatively connected to the host CPU; The control unit is configured to dispatch network parameters of the neural network to the first processing subunit; the network parameters are downloaded by the host CPU to the memory unit of the accelerator card, and dispatched by the control unit from the memory unit of the accelerator card to the first processing subunit; A video decoding unit, configured to decode an input video under the control of the control unit to obtain a video frame to be processed; The first processing sub-unit is configured to perform a first process on the video frame to be processed based on the network parameters scheduled to the processing unit to obtain a first video frame, and output the first video frame to the on-chip cache unit for caching; the video frame to be processed is sent by the host CPU to the memory unit, and is retrieved from the memory unit by the first processing sub-unit; The second processing sub-unit is configured to read the first video frame from the on-chip cache unit, perform second processing on the read first video frame, obtain a second video frame, and output the second video frame; the second processing sub-unit sends the second video frame to the memory unit, and the host CPU obtains the second video frame from the memory unit; The control unit is further configured to filter out a first target video frame from the to-be-processed video frames based on a first preset condition and to filter out a second target video frame from the second video frames based on a second preset condition; the chip further comprises: a first image encoding unit, configured to perform image compression on the first target video frame, and output the compressed first target video frame to an external processing unit; The second image encoding unit is used to perform image compression on the second target video frame, and the compressed second target video frame is used to output to an external processing unit.

2. The chip according to claim 1, characterized in that The processing unit further includes a third processing sub-unit, configured to perform a third processing on the video frame to be processed and then output the processed video frame to the on-chip cache unit for caching; The first processing sub-unit is configured to read the to-be-processed video frame that has undergone the third processing from the on-chip cache unit, and perform the first processing on the to-be-processed video frame that has undergone the third processing.

3. The chip according to claim 1, characterized in that The number of processing units is greater than 1, and the first subunit in the i+1th processing unit is used to read the second video frame output by the second processing subunit in the i-th processing unit, and use the second video frame output by the second processing subunit in the i-th processing unit as the video frame to be processed for the first processing, where i is a positive integer.

4. The chip according to claim 3, characterized in that The neural network includes multiple sub-networks, and the network parameters of each sub-network are stored in the memory unit of the accelerator card; The control unit is used to dispatch the network parameters of the neural network from the memory unit to the first processing subunit.

5. The chip according to claim 4, characterized in that The i-th processing unit uses the i-th sub-network included in the neural network to perform the first processing.

6. The chip according to claim 4 or 5, characterized in that: The memory unit includes a plurality of partitions; network parameters of at least two sub-networks among the plurality of sub-networks are stored in different partitions.

7. An accelerator card, characterized in that: The accelerator card includes: Memory units, used to store the network parameters of the neural network; and The chip according to any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device comprises: The accelerator card of claim 7; and An external processing unit is used to output the network parameters of the neural network to the memory unit.

9. A data processing method, characterized in that: Applied to the chip according to any one of claims 1 to 6, the method comprising: The control subunit dispatches the network parameters of the neural network to the first processing subunit; The first processing sub-unit performs a first process on the video frame to be processed based on the network parameters scheduled to the processing unit to obtain a first video frame, and outputs the first video frame to the on-chip cache unit for caching; The second processing sub-unit reads the first video frame from the on-chip cache unit, performs second processing on the read first video frame, obtains a second video frame, and outputs the second video frame.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method executed by the control unit in the chip according to any one of claims 1 to 6 is implemented.

Images