Interface controller, data transmission method and system on chip

Abstract

Embodiments of the present application provide an interface controller, a data transmission method and a system on chip, which relate to the technical field of interfaces and can improve the overall performance of the system. The data transmission method comprises the following steps: obtaining a plurality of data slices in a data frame, wherein the data frame is generated by a first processor; writing the plurality of data slices into a buffer; obtaining a first data slice according to a target data slice in the plurality of data slices, the first data slice comprising at least the target data slice; and transmitting the first data slice to a second processor, wherein the second processor is configured to process the first data slice.

Description

Technical Field

[0001] This application relates to the field of interface technology, and in particular to an interface controller, a data transmission method, and a system-on-a-chip. Background Technology

[0002] In a system-on-a-chip (SoC), the general process of processing data frames using a certain algorithm is as follows: the data frames are processed jointly by different processors (typically, a processor can contain one or more intellectual property cores (IP cores), each of which is a logic unit) to form an algorithm pipeline. The processors used in the algorithm pipeline can be purely hardware processors or software-programmable processors, such as: central processing unit (CPU), digital signal processor (DSP), image signal processor (ISP), video encoder, neural processing unit (NPU), graphics processing unit (GPU), or display subsystem (DSS), etc.

[0003] Generally, data exchange between processors is usually achieved through memory sharing. For example, refer to... Figure 1 As shown, after the preceding processor finishes processing the current data frame, it places it in shared memory (at least one of a buffer or DDR) and notifies the following processor to continue processing. The following processor reads the data frame from the shared memory and performs subsequent processing. It is evident that the algorithm chain is executed serially between the preceding and following processors in the above process. Therefore, for periodic real-time processing and latency-sensitive algorithm chains (e.g., 30 FPS (30 frames per second) game super-resolution), the processing of one frame of data must be completed within a unit period. Figure 2As shown, the processing and output of a data frame by the previous processor occupies one frame's processing time T1 within a unit cycle, during which the subsequent processors are idle. When a data frame is processed, the remaining time period T2 within the unit cycle is processed by the subsequent processor. Since the remaining time period T2 may be very short, this method places high demands on the performance of the subsequent processor. Furthermore, when the data frame is typically too large, it cannot be placed in the on-chip cache (SoC cache) when using shared memory, and therefore needs to be placed in external memory (e.g., double data rate dynamic random access memory, DDR), which leads to overall performance limitations due to memory bandwidth and power consumption specifications. In summary, in current SoC architectures, the data transmission method affects the overall performance improvement when processors collaboratively execute algorithms. Summary of the Invention

[0004] This application provides an interface controller, a data transmission method, and a system-on-a-chip, which can improve the overall system performance.

[0005] Firstly, a data transmission method is provided. This data transmission method is applied to an interface controller, which connects to a first processor and a second processor. The data transmission method includes: acquiring multiple data slices from a data frame, wherein the data frame is generated by the first processor; writing the multiple data slices into a buffer; acquiring a first data slice based on a target data slice among the multiple data slices, wherein the first data slice at least includes the target data slice; and transmitting the first data slice to the second processor, wherein the second processor processes the first data slice. In this scheme, the interface controller can store multiple data slices from the data frame processed by the first processor in the buffer, and can acquire the first data slice based on the target data slice among the multiple data slices, wherein the first data slice at least includes the target data slice, and then send the first data slice to the second processor, whereby the second processor processes the corresponding first data slice. Because the data frame is divided into multiple data slices in the above process, the size of each data transmission is reduced compared to the data frame. Furthermore, by transmitting the data to the second processor through the interface controller's buffer, the use of shared memory for data transmission between the first and second processors can be avoided. Additionally, since the interface controller can transmit and notify the second processor to process each first data slice acquired, the method is efficient. In this way, while the second processor processes the first data slice, the interface controller is further used to transmit other data slices to the second processor. This effectively forms a computational pipeline. Since the data frame is divided into data slices, the data of each data slice can be transmitted to the second processor through the interface controller after the first processor has finished processing it, and the second processor can start processing it. This gives the second processor more processing time. Compared with the prior art in which algorithm data frames are executed serially in each processor, in this scheme, as long as the first processor has finished processing the data of a data slice, it can be transmitted to the second processor through the interface controller for processing. Therefore, the performance requirements of the second processor are lower, and the latency of the entire computational pipeline can be reduced, thereby improving the overall system performance.

[0006] In one possible implementation, dividing the data frame into multiple data slices can be performed by a first processor or an interface controller. For example, when the first processor is a GPU-like processor, since GPUs typically render images in data slices, the first processor divides the data frame into multiple data slices and acquires these multiple data slices; specifically, this includes receiving the multiple data slices transmitted by the first processor. Alternatively, for a first processor that does not divide the data frame into data slices, this division can be performed by the interface controller. For example, before acquiring the multiple data slices, the process may include receiving the data frame transmitted by the first processor; acquiring the multiple data slices includes dividing the data frame into multiple data slices.

[0007] In one possible implementation, obtaining a first data slice based on a target data slice from multiple data slices includes: reading the target data slice and adjacent data slices from a buffer; generating a first data slice based on the target data slice and adjacent data slices, wherein the first data slice covers the target data slice and the adjacent data slices have overlapping areas. For example, the second processor can be a neural network processor (NPU); the data transmitted between the GPU and the NPU is an image. Furthermore, since the width and height of the data slices on the NPU input side are shrunk according to the number of layers and stride of the neural network when the neural network in the NPU performs calculations on the data slices, the width (height) of the data slices on the NPU output side is typically smaller than the width (height) of the data slices on the input side. Therefore, to ensure that the data slices on the NPU output side can completely cover the entire data frame, the data slices input to the NPU need to have a larger width and height. Therefore, in order to ensure that the data slice processed by the second processor on the first data slice has the same size as the target data slice, the first data slice needs to have a larger size than the target data slice. That is, in addition to covering the target data slice, the first data slice also needs to have an overlapping area with the data slices adjacent to the edge of the target data slice. The width of the overlapping area is determined by the number of layers and stride of the convolutional network of the NPU.

[0008] In one possible implementation, before obtaining the first data slice from a target data slice among multiple data slices, the method further includes: determining and storing edge-adjacent data slices of the target data slice in a buffer. Specifically, when it is necessary to calculate the first data slice based on the target data slice and its edge-adjacent data slices, it is essential to ensure that the edge-adjacent data slices of the target data slice are stored in the buffer.

[0009] In one possible implementation, the method further includes: generating a bitmap; after determining that the data slice is to be acquired, setting the bit at the corresponding position of the data slice in the bitmap to 1; determining that the data slices adjacent to the edge of the target data slice are stored in a buffer, including: determining that all bits around the corresponding position of the target data slice in the bitmap are 1. The bitmap can record that the corresponding data slice for each bit has been stored in the buffer.

[0010] In one possible implementation, the target data slice comprises 2n data slices, where n is a positive integer. If the granularity of the target data slice is 1x1, i.e., it contains only one data slice, then calculating for each target data slice requires obtaining at least eight other data slices adjacent to the edge of the target data slice (excluding target data slices located at the edge of the data frame); thus, data in the overlapping area between the first data slice and these other data slices will be repeatedly transmitted. To reduce the repeated transmission of data in the overlapping area, the target data slice provided in the embodiments of this application may include 2n data slices. For example, calculating for a 2x2 granularity target data slice requires at most 12 data slices adjacent to the edge of the target data slice. In this way, for each data slice in a 2x2 granularity target data slice, it is equivalent to providing an average of three data slices adjacent to its edge for the calculation of the first data slice, reducing the repeated transmission of data in the overlapping area.

[0011] In one possible implementation, the interface controller needs to acquire information about the data frame in order to divide the data frame into multiple data slices or to determine the position of a data slice within the data frame. Therefore, the implementation also includes: acquiring information about the data frame, including its base address, width, height, and number of channels; and dividing the data frame into multiple data slices based on this information.

[0012] In one possible implementation, the interface controller is also connected to the on-chip system's cache; when it is determined that the buffer is full, data slices from the buffer are stored in the cache. Thus, when the buffer is full, the interface controller can store data slices from the buffer into the system's cache, and also store data slices from the buffer into DDR; when there is remaining space in the buffer, it is read from the on-chip system's cache or DDR. In this way, if the buffer has sufficient space, data transfer only occurs within the buffer; if the buffer is insufficient, to avoid data transmission interruptions, data slices can be stored in the on-chip system's cache or DDR.

[0013] In a second aspect, an interface controller is provided, the interface controller being connected to a first processor and a second processor, the interface controller comprising: an acquisition unit for acquiring multiple data slices in a data frame, wherein the data frame is generated by the first processor; a processing unit for writing the multiple data slices acquired by the acquisition unit into a buffer; the processing unit for acquiring a first data slice based on a target data slice among the multiple data slices in the buffer, the first data slice including at least the target data slice; and a sending unit for transmitting the first data slice acquired by the processing unit to the second processor, wherein the second processor is used to process the first data slice.

[0014] In one possible implementation, the acquisition unit is specifically used to receive the plurality of data slices transmitted by the first processor, wherein the first processor is used to divide the data frame into the plurality of data slices.

[0015] In one possible implementation, the acquisition unit is further configured to receive the data frame transmitted by the first processor; specifically, the acquisition unit is configured to divide the data frame into the plurality of data slices.

[0016] In one possible implementation, the processing unit is specifically configured to read a target data slice from the buffer, and data slices adjacent to the edge of the target data slice; generate a first data slice based on the target data slice and the data slices adjacent to the edge of the target data slice, wherein the first data slice covers the target data slice, and the first data slice has an overlapping area with the data slices adjacent to the edge of the target data slice.

[0017] In one possible implementation, the second processor is a neural network processor (NPU).

[0018] In one possible implementation, the processing unit is further configured to determine that data slices adjacent to the edge of the target data slice are stored in a buffer.

[0019] In one possible implementation, the processing unit is further configured to generate a bitmap, and when it is determined that the data slice is to be acquired, set the bit at the corresponding position of the data slice in the bitmap to 1; the processor is specifically configured to determine that all bits around the corresponding position of the target data slice in the bitmap are 1.

[0020] In one possible implementation, the target data slice comprises 2n data slices, where n is a positive integer.

[0021] In one possible implementation, the acquisition unit is further configured to acquire information about the data frame, including the base address, width, height, and number of channels; the processing unit is specifically configured to divide the data frame into multiple data slices based on the information of the data frame.

[0022] In one possible implementation, the interface controller is further connected to a buffer of the system on-chip; the processing unit is specifically used to store data slices from the buffer into the buffer via the sending unit when it is determined that the buffer is full.

[0023] Thirdly, an interface controller is provided, including a buffer and one or more processors, the processors being used to invoke computer instructions to perform the methods described above.

[0024] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when run on a computer, causes the computer to perform the methods described above.

[0025] Fifthly, a system-on-a-chip is provided, including a first processor, a second processor, and an interface controller as described above, wherein the interface controller is connected to the first processor and the second processor, and the interface controller is used to execute the data transmission method described above.

[0026] In one possible implementation, while the second processor processes the first data slice, the interface controller is further configured to transfer other data slices to the second processor. Optionally, the first processor is further configured to process other data while the second processor processes the first data slice, to form a pipeline.

[0027] The technical effects of any of the possible implementations of aspects two through five can be found in the technical effects of the different implementations of aspect one above, and will not be repeated here. Attached Figure Description

[0028] Figure 1 A schematic diagram of the structure of a system-on-a-chip provided for an embodiment of this application;

[0029] Figure 2 A schematic diagram of an algorithm link pipeline between processors is provided for embodiments of this application;

[0030] Figure 3 A schematic diagram of a system-on-a-chip provided for another embodiment of this application;

[0031] Figure 4a A schematic diagram of a system-on-a-chip provided for another embodiment of this application;

[0032] Figure 4b A schematic flowchart illustrating a data transmission method provided for an embodiment of this application;

[0033] Figure 5 A schematic diagram of the structure of a data slice array for a data frame provided in an embodiment of this application;

[0034] Figure 6 A schematic diagram of a data slice structure provided for an embodiment of this application;

[0035] Figure 7 A schematic diagram illustrating the processing sequence of data slices in a data frame, provided for an embodiment of this application;

[0036] Figure 8 A schematic diagram illustrating the data slice division order of a data frame, provided for an embodiment of this application;

[0037] Figure 9 A schematic diagram of a system-on-a-chip provided for another embodiment of this application;

[0038] Figure 10 A schematic diagram showing the dimensions of the data slices on the NPU output side and the NPU input side, provided for embodiments of this application;

[0039] Figure 11 A schematic diagram of the structure of a data slice array for a data frame is provided for another embodiment of this application;

[0040] Figure 12 A schematic diagram of a bitmap structure provided for an embodiment of this application;

[0041] Figure 13 This is a schematic diagram of the structure of a bitmap provided in another embodiment of this application;

[0042] Figure 14 A schematic diagram of a bitmap structure provided for yet another embodiment of this application;

[0043] Figure 15 A schematic diagram of a bitmap structure provided for another embodiment of this application;

[0044] Figure 16 A schematic diagram of a bitmap structure is provided for another embodiment of this application;

[0045] Figure 17 A schematic diagram of a bitmap structure provided for yet another embodiment of this application;

[0046] Figure 18a A schematic diagram of a bitmap structure provided for another embodiment of this application;

[0047] Figure 18b A schematic diagram of a bitmap structure is provided for another embodiment of this application;

[0048] Figure 19 A schematic diagram of an algorithm link pipeline between processors is provided for another embodiment of this application;

[0049] Figure 20 A schematic flowchart of a data transmission method provided for another embodiment of this application;

[0050] Figure 21 This is a schematic diagram of the structure of an interface controller provided for an embodiment of this application. Detailed Implementation

[0051] In this embodiment, the terms "first," "second," etc., are used for descriptive convenience only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units refer to two or more processing units.

[0052] Furthermore, in the embodiments of this application, "upper," "lower," "left," and "right" are not limited to the orientation of the components schematically placed in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings. In the accompanying drawings, for clarity, the thickness of layers and regions is exaggerated, and the dimensional proportions between the parts in the drawings do not reflect the actual dimensional proportions.

[0053] In the embodiments of this application, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium.

[0054] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] The artificial neural network (ANN) mentioned in the embodiments of this application is simply referred to as a neural network (NN) or a neural network-like system. In the fields of machine learning and cognitive science, it is a mathematical or computational model that mimics the structure and function of biological neural networks (such as the central nervous system of animals, especially the brain) and is used to estimate or approximate functions. Artificial neural networks can include convolutional neural networks (CNN), deep neural networks (DNN), time-delay neural networks (TDNN), and multilayer perceptrons (MLP), among other neural networks.

[0056] The technical solutions provided in this application can be applied to electronic devices including various types of chips, such as systems-on-a-chip (SoCs), such as mobile phones, mobile terminals, personal computers (PCs), servers, laptops, tablets, in-vehicle computers, smart cameras, smartwatches, embedded devices, etc. The embodiments of this application do not impose any special limitations on the specific form of the electronic device.

[0057] The following describes in detail, with reference to the accompanying drawings, an interface controller, a data transmission method, and a system-on-a-chip provided in the embodiments of this application. Figure 3 This is a schematic diagram of a system-on-a-chip (SoC) provided in an embodiment of this application. It should be understood that the following description uses an SoC as an example only; the actual solution can also be applied to other types of chips or devices. Figure 3 As shown, the system-on-a-chip 100 may include a processor 110, a processor 111, an interface controller 112, a memory 120, a communication line 130, a register 140, and at least one communication interface 150 and peripheral circuitry (not shown in the figure). It should be noted that, in addition to... Figure 3 In addition to the devices shown, the system-on-chip 100 may also include a communication interface 150. It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the system-on-chip 100. In other embodiments of this application, the system-on-chip 100 may include more or fewer components than illustrated, or some components may be combined, or some components may be separated, or there may be different component arrangements. The illustrated components may be implemented in hardware or a combination of software and hardware.

[0058] Processor 110 is the computing and control core of System-on-Chip 100. Processor 110 may include a central processing unit (CPU), digital signal processor (DSP), image signal processor (ISP), video encoder, neural processing unit (NPU), graphics processing unit (GPU), DSS, or application-specific integrated circuit (ASIC). In embodiments of this application, the processor may be purely hardware or hardware capable of loading software programs. In a specific implementation, as one embodiment, processor 110 may include one or more processor cores, such as... Figure 3 Cores 0 and 1 in the chip. In a specific implementation, as one embodiment, the system-on-a-chip 100 may include multiple processors, such as... Figure 1 The processors 110, 111, and 112 are included. Each of these processors can be a single-core processor (i.e., the processor includes one core) or a multi-core processor (i.e., the processor includes multiple cores).

[0059] The memory 120 may exist independently of the processor 110, but is connected to the processor 110 via a communication line 130. In one example, the memory 120 may be used to store instructions and data, including instructions for executing the scheme of this application; the memory 120 may be, for example, static random access memory (SRAM). The instructions are controlled and executed by the processor 110. In the embodiments of this application, the interface controller 112 may execute the instructions stored in the memory 120 to implement the data transmission method provided in the following embodiments of this application. Optionally, the instructions in the embodiments of this application may also be referred to as application code, and this application does not specifically limit this.

[0060] Communication line 130 may include a path for transmitting information between the aforementioned components. Communication interface 150, using any transceiver-like device, is used for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0061] Cache 140 is a high-speed memory located between processor 110 and memory 120; its capacity is smaller than memory 120, but its exchange speed with processor 110 is faster. Cache 140 can be used to store instructions or data that processor 110 has just used or that are used repeatedly. If processor 110 needs to use the instruction or data again, it can retrieve it directly from cache 140. For example, the instructions or data stored in cache 140 may be a small portion of those in memory 120, but this small portion of instructions or data is something processor 110 will access in a short time; when processor 110 retrieves the instructions or data, it can bypass memory 120 and retrieve it directly from cache 140, thereby speeding up the read speed.

[0062] As described above, the interface controller 112 can be a single-core or multi-core processor. Furthermore, in the embodiments of this application, the interface controller 112 is also provided with a buffer. The buffer, also known as a buffer register, is mainly used to compensate for the data accumulation caused by the difference in data processing speed between the devices at both ends of the buffer when data is transmitted through the buffer between two devices.

[0063] Based on the above-described system-on-a-chip, embodiments of this application provide a data transmission method applied to an interface controller 112. (Refer to...) Figure 4a As shown, the interface controller 112 is connected to the first processor 110 and the second processor 111, wherein, in conjunction with the above-mentioned... Figure 3 The illustrated on-chip system description shows that the interface controller 112 can be connected to the first processor 110 and the second processor 111 via communication lines. (Refer to...) Figure 4b The data transmission method shown includes the following steps:

[0064] 101. Interface controller 112 acquires multiple data slices from a data frame, wherein the data frame is generated by a first processor. In step 101, the data frame can be in the form of an image, text, audio, or video, etc. The multiple data slices (slices or tiles) in the data frame can be in the form of a data slice array. Taking a GPU as an example, the data frame can be a single image frame, and the image data frame can be a three-dimensional data array (3D datacube) including pixel data from each channel. Figure 5As shown, in the xyz rectangular coordinate system, the x-direction of the data frame represents the width W (where the width can be the number of pixels in a row in the x-direction), and the y-direction represents the height H (where the height can be the number of pixels in a column in the y-direction). Taking a 4K image as an example, its resolution is 4096×2160, so the width W of the image data frame is 4096, and the height H is 2160. The z-direction of the data frame represents the channel C (for example, the data frame of an image using RGB (red, green, blue) as the three primary colors contains three channels: R channel, G channel, and B channel). The image data frame can be divided into multiple data slices on the xy plane. Embodiments of this application also provide a data slice structure, such as... Figure 6 As shown, for the data slices divided into data frames of an image, each data slice also contains data from three channels. The difference between a data slice and the entire frame is that the height and width of a data slice are only fractions of the entire frame's data. It can be understood that each channel of a data slice contains pixel data arranged in an array of rows and columns. Furthermore, other data can also be presented in the form of two-dimensional data arrays. Of course, a color image can also be converted to grayscale (or binarized) and transformed into a single-channel grayscale image; the data frame of such a grayscale image can also be in the form of a two-dimensional data array.

[0065] In the embodiments of this application, the data frame can be divided into multiple data slices by the first processor 110 or by the interface controller 112 provided in the embodiments of this application. For example, if the first processor 110 divides the data frame into multiple data slices, then in step 101, the interface controller 112 can directly receive the multiple data slices transmitted by the first processor 110. The first processor 110 can divide the data frame into multiple data slices, process each data slice, and then send the data slices sequentially to the interface controller 112 in a certain order, for example, referring to... Figure 7As shown, data slices in the data slice array can be processed in a zigzag scanning manner and then sequentially sent to the interface controller 112; alternatively, the first processor 110 can send the processed data slices to the interface controller 112 out of order. In another approach, the interface controller 112 can directly receive the data frame transmitted by the first processor 110 and then divide the data frame into multiple data slices. It should be noted that in a SoC system, the CPU can configure registers for the first processor 110 and the interface controller 112 to notify whether the first processor 110 or the interface controller 112 will divide the data frame into multiple data slices. In some examples, the interface controller 112 needs to obtain information about the data frame in order to divide it into multiple data slices or to determine the position of a data slice within the data frame. This information includes the base address, width, height, and number of channels of the data frame. This information can be provided to the interface controller 112 by the CPU configuring registers for it. Specifically, the first processor 110 can send data frames to the interface controller 112 via burst events. A single burst event can send a data frame, a data slice, or data of any other arbitrary length. Thus, if the interface controller 112 divides the data frame into multiple data slices, when a burst event sends data of any length (e.g., a data frame or other data of any other arbitrary length), the burst event can simultaneously carry the write address and data length. Therefore, as long as the length of the data to be sent exceeds one data slice based on the write address and data length, the interface controller 112 will divide the received data into multiple data slices according to the information of the data frame. For example, refer to... Figure 8 As shown, for the first processor 110 that outputs data frames line by line in a zigzag scan, the interface controller 112 can divide several consecutive lines of data into a data slice according to the base address, width, height, and number of channels of the data frame. If the first processor 110 divides the data frame into multiple data slices, when a burst event sends a data slice, the burst event can simultaneously carry the write address and data length of the data slice; then the interface controller 112 can determine the position of the data slice in the data frame according to the write address, data length, and the aforementioned information of the data frame.

[0066] 102. Interface controller 112 writes multiple data slices to the buffer. Specifically, when the buffer is full, interface controller 112 can store the data slices in the buffer into the system's cache. Additionally, interface controller 112 can store the data slices in the buffer into DDR. When there is remaining space in the buffer, the data is read from the on-chip system's cache or DDR. Thus, if the buffer has sufficient space, data transmission only occurs within the buffer; if the buffer is insufficient, to avoid data transmission interruption, data slices can be stored in the on-chip system's cache or DDR.

[0067] 103. Interface controller 112 obtains a first data slice from a target data slice among multiple data slices. The first data slice includes at least the target data slice. For example, the first data slice includes the target data slice and may optionally further include at least a portion of the data slices adjacent to the edge of the target data slice. In step 103, interface controller 112 may directly read any target data slice from the buffer and send it to the second processor 111 in a subsequent process. Alternatively, it may read the target data slices in a certain order (e.g., zigzag) and send them to the second processor 111 in a subsequent process. This is provided that the target data slice has already been written into the buffer through the processing in steps 101 and 102.

[0068] In embodiments of this application, before initially receiving a data slice or data frame, the interface controller 112 can pre-generate a bitmap. This bitmap can be pre-configured by default, with all bits set to 0 in the initial state (e.g., before the interface controller 112 starts receiving data frames or data slices). Upon receiving a data slice sent by the first processor 110 in the manner described above, or after the interface controller 112 locally divides a data frame into data slices, the bits corresponding to the data slices in the bitmap are set to 1. When the interface controller divides a data frame into data slices, it can directly map each data slice to a bit in the bitmap based on its position within the data frame. When a data slice is sent by the first processor 110, the interface controller 112 can receive the position information of the data slice in the data frame sent by the first processor 110 (e.g., the coordinates of the data slice, or the write address and data length of the data slice) while acquiring the data slice. This allows for a one-to-one correspondence between the data slice and the bits in the bitmap based on the position information of the data slice. In this way, the data slice corresponding to the bit set to 1 in the bitmap is the data slice that is completely transmitted to the interface controller 112. For example... Figure 12As shown, "1" in the bitemap table indicates that the corresponding data slice is ready, and "0" indicates that the data slice has not yet been fully output from the first processor 110 to the interface controller 112.

[0069] Additionally, refer to Figure 9 As shown, when the first processor is a GPU and the second processor is an NPU, the data transferred between the GPU and NPU is an image. Furthermore, because the neural network in the NPU performs calculations on data slices, the width and height of the data slices at the NPU input side are shrunk according to the number of layers and the stride of the neural network before being output. Therefore, the width (height) of the data slices at the NPU output side is usually smaller than the width (height) of the data slices at the input side. Figure 10 As shown, the width W1 of the data slice on the NPU input side is greater than the width W2 of the data slice on the output side; the height H1 of the data slice on the input side is greater than the height H2 of the data slice on the output side. Specifically, assuming the stride in both W and H directions is 1, and the neural network has a total of N layers, each of which is a 3x3 convolution, then W2 = W1 - N × 1, H2 = H1 - N × 1. Thus, if... Figure 5 The data slices shown are used as input to the neural network. After processing by the NPU, there will be gaps between the various data slices output, affecting the quality of the output image. To avoid this problem, it is necessary to ensure that the output data slices completely cover the entire data frame; that is, the sizes (W2 and H2) of the data slices output by the NPU must be consistent with... Figure 5 The data slices shown are all the same size. Therefore, the data slices input to the neural network need to have a larger size than... Figure 5 The data slice shown has a larger width and height. For example, step 103 may specifically involve, after determining that the data slices adjacent to the edge of the target data slice are stored in the buffer (in conjunction with the bitmap mentioned above, i.e., after determining that the bits around the corresponding position of the target data slice in the bitmap table are all 1), reading the target data slice and the data slices adjacent to the edge of the target data slice from the buffer; generating a first data slice based on the target data slice and the data slices adjacent to the edge of the target data slice, wherein the first data slice covers the target data slice, and the first data slice and the data slices adjacent to the edge of the target data slice have overlapping areas. See details. Figure 11 The first data slice includes the target data slice and a portion of multiple data slices adjacent to the edge of the target data slice; this portion is the overlapping region. The width of the overlapping region is determined by the number of layers and stride of the NPU's convolutional network. Figure 11As shown, the first data slice corresponding to the target data slice (i, j) covers the target data slice (i, j); and the first data slice has overlapping areas with the data slices (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1), that is, the first data slice includes a part of the data slices (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), and (i+1, j+1), as shown. Figure 11 As shown. Furthermore, it should be noted that for data slices located at the edges of data frames, since there are no adjacent data slices in certain directions—for example, data slice (i-1, j-1) has no adjacent data slices to its left and top—when calculating the first data slice corresponding to data slice (i-1, j-1), a first data slice that meets the size of the data slice on the NPU input side can be generated by padding. For example, the first data slice corresponding to data slice (i-1, j-1) can be padded with 0s to the left and top of (i-1, j-1) or directly padded with pixel data copied from data slice (i-1, j-1).

[0070] Alternatively, Figure 11 The first data slice may also encompass all of the multiple adjacent data slices. Alternatively, the first data slice may further include more. For example, in Figure 11 In this embodiment, the multiple data slices adjacent to the edge of the target data slice are one data slice adjacent to the target data slice in each direction of that edge. However, alternatively, the number of adjacent data slices in each direction of that edge can be multiple, depending on the ratio between heights H1 and H2 and the ratio between widths W1 and W2. It can be understood that the first data slice can also include multiple data slices adjacent to the edge of the target data slice in each direction, that is, multiple data slices are extended outward from the target data slice to cover a larger area. This embodiment does not limit the extent to which the first data slice is extended from the target data slice.

[0071] Because the data slices output by the first processor 110 may be out of order, rather than in the usual Z-scan order, the following explanation focuses on the out-of-order scenario (it can be understood that sequential output can be categorized as out-of-order output). Embodiments of this application can employ a greedy algorithm to process the out-of-order data slices sequentially (e.g., in Z-scan order). For example, if only one target data slice can be processed, meaning that the target data slice and its adjacent data slices are already stored in the buffer of the interface controller 112, then the calculation of the first data slice corresponding to that target data slice can begin. If multiple target data slices can be processed simultaneously, the calculation of the corresponding first data slices is performed sequentially according to the Z-scan order.

[0072] For details, please refer to Figure 12 Given a bitmap, if the adjacent data slice tiles of a target data slice tile are all ready (bits are 1), the first data slice input to the NPU can be calculated. Figure 12 The dashed box in the middle represents the first data slice to be calculated. Then, step 104 can be executed, transferring the first data slice to the NPU for processing. When several similar slices simultaneously meet the above conditions, a greedy algorithm can be used to calculate them sequentially (e.g., in a zigzag scanning order).

[0073] Specifically, taking a 4K image data frame as an example, if the data slice size is 256×256, then a bitmap size is 16x8 bits. (See reference...) Figure 13As shown, the target data slice (i, j) and its adjacent data slices (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j+1), and (i+1, j+1) have all been stored in the interface controller's buffer. The condition for triggering the calculation of the first data slice is: Cur_stat[i,j] = flag[i-1,j-1]&flag[i-1,j]&flag[i-1,j+1]&flag[i,j-1]&flag[i,j+1]&flag[i+1,j-1]&flag[i+1,j]&flag[i+1,j+1]; where flag[i-1,j-1] indicates that the bit of the data slice at coordinate (i-1,j-1) has been marked as 1, and Cur_stat[i,j] indicates that the data slices at coordinates (i,j), (i-1,j-1), (i-1,j), (i-1,j+1), (i,j-1), (i,j+1), (i+1,j-1), (i+1,j) and (i+1,j+1) have all been stored in the buffer of the interface controller.

[0074] In addition, refer to Figure 14 As shown, the interface controller 112 can also maintain a bitmap of the target data slice for which the first data slice has been calculated. In this bitmap, a bit value of 1 indicates that the first data slice has been calculated for the corresponding target data slice, and a bit value of 0 indicates that the first data slice has not yet been calculated for the corresponding target data slice. During execution... Figure 13 Before calculating the first data slice corresponding to the target data slice (i,j), the calculations for the first data slices corresponding to data slices (i-2,j-2), (i-2,j-1), (i-2,j), and (i-1,j) have already been completed. After completing the calculation of the first data slice corresponding to the target data slice (i,j), we can... Figure 14 The bitmap is updated to Figure 15 The bitmap shown. This is achieved through maintenance and updates. Figure 14 , Figure 15 The provided bitmap can determine whether further processing of the current bitmap is needed. Figure 13 Which target data slice corresponds to the first data slice?

[0075] The granularity of the target data slice calculated above is 1x1. Calculating this once for each target data slice requires acquiring at least eight other data slices adjacent to the edge of the target data slice (excluding the target data slice located at the edge of the data frame); therefore, data in the overlapping area between the first data slice and these other data slices will be repeatedly transmitted. To reduce the repeated transmission of data in overlapping areas, the target data slice provided in the embodiments of this application may include 2n data slices. For example, the granularity of the target data slice may be different sizes such as 1x2, 2x2, ...

[0076] The following example uses a target data slice granularity of 2x2 as another instance, refer to... Figure 16 As shown, when the target data slice including data slices (1,1), (1,2), (2,1), and (2,2) and the surrounding data slices (1,3), (2,3), (3,1), (3,2), and (3,3) have all been stored in the interface controller's buffer, the first data slice corresponding to the target data slice is calculated. This is compared to... Figure 13 The provided 1x1 granularity target data slice requires the simultaneous completion of eight adjacent data slices at the edge of the target data slice before the corresponding first data slice can be calculated. A 2x2 granularity target data slice requires a maximum of 12 adjacent data slices at the edge of the target data slice (since the example 2x2 granularity target data slice is located in the top left corner, therefore...). Figure 13 The diagram shows five adjacent data slices at the edge of the target data slice. In this way, for a 2x2 granularity target data slice, the calculation of the first data slice is equivalent to providing three adjacent data slices at its edge on average, which reduces the repeated transmission of data in overlapping areas.

[0077] In another example, for Figure 17The provided bitmap has a target data slice with a 2x2 granularity, specifically including data slices (i, j), (i+1, j), (i, j+1), and (i+1, j+1). The target data slice and its adjacent data slices (i-1, j-1), (i-1, j), (i-1, j+1), (i-1, j+2), (i, j+1), (i, j+2), (i+1, j-1), (i+1, j+2), (i+2, j-1), (i+2, j), (i+2, j+1), and (i+2, j+2) are all stored in the interface controller's buffer. The condition for triggering the calculation of the first data slice is: Cur_stat[(i,j)&(i+1,j)&(i,j+1)&(i+1,j+1)]=flag[i-1,j-1]&flag[i-1,j]&flag[i-1,j+1]&flag[i-1,j+2]&flag[i,j+1]&flag[i,j+2]&flag[i+1,j-1]&flag[i+1,j+2]&flag[i+2,j-1]&flag[i+2,j]&flag[i+2,j+1]&flag[i+2,j+2]; where flag[i-1,j-1] indicates that the bit of the data slice at coordinate (i-1,j-1) has been marked as 1.

[0078] In addition, refer to Figure 18a As shown, the interface controller 112 can also maintain a bitmap representing the target data slice for which the first data slice can be computed. In this bitmap, (x, y) where x is 1, it indicates that the 2x2 granularity target data slice and its adjacent data slices have been stored in the buffer of the interface controller 112; x is 0, it indicates that the 2x2 granularity target data slice and its adjacent data slices have not yet been fully stored in the buffer of the interface controller 112. y is 0, it indicates that the first data slice has not yet been computed for the corresponding target data slice; y is 1, it indicates that the first data slice has been computed for the corresponding target data slice. As described above... Figure 17 The provided bitmap, without calculating the first data slice for the target data slices (i, j), (i+1, j), (i, j+1), and (i+1, j+1), shows the target data slice in... Figure 18a The corresponding representation is (1, 0). After completing the calculation of the first data slice corresponding to the target data slice, the target data slice can be displayed in... Figure 18b The corresponding value in the middle is updated to (1,1). This is achieved through maintenance and updates. Figure 18a , Figure 18b The provided bitmap can determine the current... Figure 17 Which target data slice corresponds to the first data slice?

[0079] Furthermore, 104, the interface controller 112 transmits the first data slice to the second processor 111, wherein the second processor is used to process the first data slice. Specifically, the interface controller 112 may also send a first processing instruction corresponding to the first data slice to the second processor 111, wherein the second processor 111 processes the first data slice according to the first processing instruction.

[0080] In this scheme, the interface controller 112 can store multiple data slices from the data frame processed by the first processor 110 in a buffer, and can obtain a first data slice based on a target data slice among the multiple data slices. The first data slice at least includes the target data slice. Then, the first data slice is sent to the second processor 111, which processes the corresponding first data slice. Because the data frame is divided into multiple data slices, the size of each data transmission is reduced compared to the data frame. Since the data is transmitted to the second processor 111 through the buffer of the interface controller 112, the shared memory method can be avoided when transmitting data between the first processor 110 and the second processor 111. Furthermore, because the interface controller 112 obtains each first data slice... The interface controller 112 can transmit and notify the second processor 111 to process the data. While the second processor is processing the first data slice, the interface controller 112 can further transmit other data slices to the second processor 111, effectively forming a computational pipeline. Since the data frame is divided into data slices, after the first processor 110 finishes processing each data slice, the data can be transmitted to the second processor 111 via the interface controller 112 for processing. This provides the second processor 111 with more processing time. Compared to the prior art where the algorithm is executed serially across processors, this scheme places lower performance requirements on the second processor 111. For example, if the first processor 110 divides the data frame into multiple data slices, refer to... Figure 19As shown, within a unit cycle, the first processor and the second processor need to complete the processing of one frame of data. During the processing cycle T1 of the first processor, after the first processor 110 finishes processing data slice t1, it can transmit the data slice to the second processor 111 via the interface controller 112. The interface controller 112 transmits each data slice it acquires to the second processor 111 for processing. This allows the second processor 111 to begin processing the data slice transmitted by the interface controller 112 after the first processor 110 finishes processing data slice t1 (i.e., the second processor 111 starts processing the data frame during the processing cycle T2 of the first processor within the processing cycle T1 of the first processor), thus providing more processing time for the second processor 111 and reducing the performance requirements of the second processor 111. Furthermore, while the second processor 111 is processing t1, the first processor 110 can still continue processing other data. The interface controller 112 is further used to transmit other data slices to the second processor 111, effectively forming a computational pipeline. Additionally, if the interface controller 112 divides the data frame into multiple data slices, such as... Figure 8 As shown, as soon as the first processor 110 starts processing the data frame and outputs the data frame data to the interface controller 112, the interface controller 112 can start slicing the data frame. After the interface controller 112 obtains a data slice, it can transmit the data slice to the second processor 111 for processing. In this way, the second processor 111 can also start processing the data slice transmitted by the interface controller 112 after t1, which provides more processing time for the second processor 111, reduces the performance requirements of the second processor 111, and reduces the latency of the entire computing link, thereby improving the overall system performance.

[0081] The following example uses a GPU as the first processor and an NPU as the second processor. Figure 20 As shown, the data transmission method provided by the embodiments of this application will be described as follows. In the initial state, the interface controller sets all bits in the bitmap to 0. This initial state can be a reset after the last data frame transmission ends. For example, the end of a data frame usually carries a data end indicator, and the bitmap can be reset to the initial state according to this data end indicator.

[0082] In step 201, the interface controller determines whether the GPU has data output. If so, it determines whether the GPU outputs a data slice or a data frame. If it determines that the GPU outputs a data frame, it divides the data frame into multiple data slices and executes step 202. If it determines that the GPU outputs a data slice, it executes step 202. In step 202, the interface controller sets the corresponding bit in the bitmap to 1 according to the data slice. In step 203, the interface controller determines whether there is a target data slice according to the bitmap so that the corresponding first data slice can be calculated. If so, it calculates the first data slice and executes step 204. In step 204, the interface controller transmits the first data slice to the NPU and sends a first processing instruction to the NPU to notify the NPU to process the first data slice. In step 205, it determines whether all data slices have been processed. If so, the process ends; otherwise, it executes step 203. Steps 201-205 above describe the basic logical process of a method for data transmission between the GPU and NPU using an interface controller provided in this application. The specific method in each process can be referred to the description in steps 101-106 above.

[0083] The foregoing mainly describes the solutions provided in the embodiments of this application. It is understood that, in order to achieve the above functions, the interface controller includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should understand that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0084] This application embodiment can divide the interface controller into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the division of functional modules according to each function as an example.

[0085] Figure 21This is a schematic diagram of the logical structure of the interface controller 300 provided in an embodiment of this application. The interface controller 300 can implement the data transmission method provided in this embodiment. The interface controller 300 can be a hardware structure, a software module, or a hardware structure plus a software module. Figure 21 As shown, the interface controller 300 includes an information acquisition unit 301, a processing unit 302, a sending unit 303, and a buffer. The acquisition unit 301 can be used to execute step 101 described above. The processing unit 302 can be used to execute steps 102 and 103 described above. The sending unit 303 can be used to execute step 104 described above. All relevant content regarding each step in the above method embodiment can be referenced from the functional descriptions of the corresponding functional units, and will not be repeated here.

[0086] As will be known to those skilled in the art, all or part of the steps in the above methods can be implemented by hardware related to program instructions, and the program can be stored in a computer-readable storage medium, such as ROM, RAM, and optical disc. Embodiments of this application also provide a computer-readable storage medium, which may include a memory. The explanation and beneficial effects of any of the interface controllers 300 provided above can be referred to the corresponding method embodiments provided above, and will not be repeated here.

[0087] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0088] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0089] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A data transmission method, characterized in that, include: Multiple data slices are obtained from a data frame, wherein the data frame is generated by a first processor and the multiple data slices are in the form of a data slice array; Write the multiple data slices into the buffer; Obtaining a first data slice based on a target data slice from the plurality of data slices, wherein the first data slice includes at least the target data slice; wherein obtaining the first data slice based on the target data slice from the plurality of data slices includes: reading the target data slice and data slices adjacent to the edge of the target data slice in the buffer; generating the first data slice based on the target data slice and the data slices adjacent to the edge of the target data slice, wherein the first data slice covers the target data slice, and the first data slice and the data slices adjacent to the edge of the target data slice have an overlapping area; The first data slice is transmitted to a second processor, wherein the second processor is used to process the first data slice; the first processor is a GPU and the second processor is an NPU.

2. The data transmission method according to claim 1, characterized in that, The acquisition of multiple data slices in the data frame includes: The processor receives the plurality of data slices transmitted by the first processor, wherein the first processor is configured to divide the data frame into the plurality of data slices.

3. The data transmission method according to claim 1, characterized in that, Before acquiring multiple data slices in a data frame, the method further includes: receiving the data frame transmitted by the first processor; The step of acquiring multiple data slices in a data frame includes: dividing the data frame into the multiple data slices.

4. The data transmission method according to claim 1, characterized in that, The target data slice includes 2n data slices, where n is a positive integer.

5. An interface controller, characterized in that, The interface controller includes: An acquisition unit is used to acquire multiple data slices in a data frame, wherein the data frame is generated by a first processor, and the multiple data slices are in the form of a data slice array; A processing unit is used to write the plurality of data slices acquired by the acquisition unit into a buffer; The processing unit is configured to obtain a first data slice based on a target data slice among the plurality of data slices in the buffer, wherein the first data slice includes at least the target data slice; specifically, the processing unit is configured to read the target data slice and adjacent data slices on the edge of the target data slice from the buffer; generate the first data slice based on the target data slice and adjacent data slices on the edge of the target data slice, wherein the first data slice covers the target data slice, and the first data slice and adjacent data slices on the edge of the target data slice have an overlapping area; A sending unit is configured to transmit the first data slice acquired by the processing unit to a second processor, wherein the second processor is configured to process the first data slice; the first processor is a GPU, and the second processor is an NPU.

6. The interface controller according to claim 5, characterized in that, The acquisition unit is specifically used to receive the plurality of data slices transmitted by the first processor, wherein the first processor is used to divide the data frame into the plurality of data slices.

7. The interface controller according to claim 5, characterized in that, The acquisition unit is further configured to receive the data frame transmitted by the first processor; specifically, the acquisition unit is configured to divide the data frame into the plurality of data slices.

8. The interface controller according to claim 5, characterized in that, The target data slice includes 2n data slices, where n is a positive integer.

9. A system-on-a-chip, characterized in that, It includes a first processor, a second processor, and an interface controller, wherein the interface controller is connected to the first processor and the second processor, and the interface controller is used to execute the data transmission method as described in any one of claims 1-4.

10. The system-on-a-chip according to claim 9, characterized in that, While the second processor is processing the first data slice, the interface controller is further configured to transfer other data slices to the second processor.

Images