Three-dimensional convolution pipeline with memory organizer unit

Abstract

A processor system includes a memory organizer unit and a matrix computation unit. The memory organizer unit is configured to receive a request for three-dimensional data for a convolutional neural network layer. The requested three-dimensional data is obtained from memory. The obtained three-dimensional data is rearranged in an optimized linear order, and the data rearranged in the optimized linear order is provided to the matrix computation unit. The matrix computation unit is configured to perform at least a portion of a three-dimensional convolution using at least a portion of the provided data rearranged in the optimized linear order.

Description

[0001] background

[0002] Neural networks can be used to solve a whole class of complex artificial intelligence problems. Because these problems are typically computationally and data-intensive, custom hardware solutions are often beneficial for improving the performance of neural networks. These custom microprocessors are usually designed to be optimized for solving a specific set of AI problems. As problem domains expand and neural networks are able to solve increasingly diverse problem types, traditional hardware computing resources may not be used efficiently. Therefore, there is a need for a hardware microprocessor system that can efficiently utilize existing hardware computing resources when handling less frequently used neural network operations (such as 3D convolution operators).

[0003] Overview

[0004] A processor system may be provided, comprising: a memory organizer unit configured to: receive a request for three-dimensional data of a convolutional neural network layer; obtain the requested three-dimensional data from memory; rearrange the obtained three-dimensional data in an optimized linear order; and provide the rearranged data in the optimized linear order to a matrix computation unit; and a matrix computation unit configured to perform at least a portion of a three-dimensional convolution using at least a portion of the provided rearranged data in the optimized linear order.

[0005] In some embodiments, the 3D data of the convolutional neural network layer is video data.

[0006] In some embodiments, the convolutional neural network layer is a layer used to determine the recommended neural network.

[0007] In another embodiment, the convolutional neural network layer is a layer of a neural network used to recognize content.

[0008] The matrix computation unit can be configured to receive a set of weights for performing three-dimensional convolution.

[0009] In some embodiments, the weight set is a three-dimensional matrix.

[0010] In another embodiment, the weight set is a 3x3x3 kernel.

[0011] In another embodiment, the weight set is formatted in a linear order.

[0012] In some embodiments, the matrix computation unit is configured to perform dot product results.

[0013] The matrix computation unit can be a dot product engine.

[0014] In some embodiments, the rearranged data comprises multiple linear two-dimensional slices of three-dimensional data.

[0015] In some embodiments, the matrix computation unit is configured to receive three-dimensional convolution operation instructions.

[0016] The instructions for 3D convolution operations can include a first reference to the data parameters and a second reference to the weight parameters.

[0017] A method may be provided, comprising: receiving a request for three-dimensional data of a convolutional neural network layer; obtaining the requested three-dimensional data from a memory; rearranging the obtained three-dimensional data in an optimized linear order; and providing the data rearranged in the optimized linear order to a matrix computation unit.

[0018] In some embodiments, an optimized linear sequence is used to perform three-dimensional convolution operations.

[0019] Another method may be provided, comprising: receiving a three-dimensional convolution operation instruction, wherein the three-dimensional convolution operation instruction specifies three-dimensional data of a convolutional neural network layer; requesting the three-dimensional data of the convolutional neural network layer from a memory organizer unit; receiving the requested three-dimensional data from the memory organizer unit in an optimized linear order; receiving three-dimensional weight data of a convolutional filter; and performing at least a portion of a three-dimensional convolution using at least a portion of the requested three-dimensional data received in the optimized linear order and at least a portion of the received three-dimensional weight data.

[0020] In some embodiments, the 3D convolution operation instruction specifies weight parameters corresponding to the 3D weighted data.

[0021] In another embodiment, the three-dimensional convolution operation instructions include the memory address of the three-dimensional data of the specified convolutional neural network layer.

[0022] In some embodiments, the 3D data of the convolutional neural network layer is video data.

[0023] In some embodiments, performing a 3D convolution includes at least a portion of performing a dot product operation.

[0024] This invention can be implemented in a variety of ways, including as a process; an apparatus; a system; a component of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form in which the invention may take may be referred to as technology. Generally, within the scope of this invention, the order of steps of the disclosed process may be changed. Unless otherwise stated, components such as processors or memory described as being configured to perform a task may be implemented as general components temporarily configured to perform a task at a given time or manufactured as specific components to perform a task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data (e.g., computer program instructions). Brief description of the attached diagram

[0026] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0027] Figure 1 This is a block diagram illustrating an embodiment of a system that uses neural networks to solve artificial intelligence problems.

[0028] Figure 2 This is a block diagram illustrating an embodiment of a system that uses neural networks to solve artificial intelligence problems.

[0029] Figure 3 This is a flowchart illustrating an example of using neural networks to solve artificial intelligence problems.

[0030] Figure 4 This is a flowchart illustrating an embodiment of the process of performing a three-dimensional convolution operation using a matrix computation unit.

[0031] Figure 5 This is a flowchart illustrating an embodiment of the process of performing a three-dimensional convolution operation using a memory organizer unit.

[0032] Figure 6 This is a flowchart illustrating an embodiment of the process of performing a three-dimensional convolution operation using a memory organizer unit.

[0033] Figure 7A and Figure 7B This is a diagram showing an example two-dimensional matrix slice used to perform a two-dimensional convolution operation.

[0034] Figure 8 This is a flowchart illustrating an embodiment of the process of performing a two-dimensional convolution operation using linearized operands and matrix computation units.

[0035] Detailed description

[0036] The following provides a detailed description of one or more embodiments of the invention, along with accompanying drawings illustrating the principles of the invention. The invention has been described in conjunction with these embodiments, but is not limited to any particular embodiment. The scope of the invention is limited only by the claims, and the invention includes many alternatives, modifications, and equivalents. To provide a full understanding of the invention, numerous specific details are set forth in the following description. These details are provided for illustrative purposes, and the invention may be practiced without some or all of these specific details, as claimed. For clarity, technical materials known in the art related to the invention have not been described in detail so as not to unnecessarily obscure the invention.

[0037] A microprocessor system for efficiently performing three-dimensional convolution is disclosed. In various embodiments, the microprocessor system includes a memory organizer unit that interfaces between memory and a matrix computation unit. The matrix computation unit is capable of performing conventional matrix operations to solve neural networks (e.g., two-dimensional convolutional neural networks). When the matrix computation unit receives a three-dimensional convolution operation, it receives three-dimensional data associated with the convolutional neural network layer from the memory organizer unit. The memory organizer unit provides the three-dimensional data in a data format optimized for the matrix computation unit. In some embodiments, the memory organizer unit first obtains the requested data from memory, where the data may be stored in a layout format that is efficient for reading and / or writing but inefficient for performing convolution operations. Once retrieved from memory, the memory organizer unit rearranges the three-dimensional data into an optimized linear order. For example, the data layout is rearranged into a layout format optimized for performing the three-dimensional convolution operation. The linearized data is then provided to the matrix computation unit. The matrix computation unit receives the requested three-dimensional data, including the weight set associated with the three-dimensional convolution operation and the convolutional neural network layer, in an optimized linear order. Because the data layout is optimized for the matrix computation unit, the computational performance of the matrix computation unit is very efficient when performing 3D convolution operations.

[0038] In some embodiments, the microprocessor system includes a memory organizer unit and a matrix computation unit. For example, the microprocessor system includes a memory organizer unit that retrieves data for the matrix computation unit to perform three-dimensional convolution. The memory organizer unit is configured to receive requests for three-dimensional data for a convolutional neural network layer. For example, the convolutional neural network layer includes three-dimensional input data. In some embodiments, the three-dimensional input data is video data, such as a series of image data captured over time. The three-dimensional data can be data with low arithmetic strength. For example, in response to a convolution instruction to be executed by the matrix computation unit, the memory organizer unit can receive a request for three-dimensional input data from the matrix computation unit. In some embodiments, the request for three-dimensional input data is received from a control unit, rather than directly from the matrix computation unit. The memory organizer unit is configured to obtain the requested three-dimensional data from memory. For example, the three-dimensional data can be stored in memory and read from memory by the memory organizer unit. In various embodiments, the memory organizer unit rearranges the obtained three-dimensional data in an optimized linear order. For example, the three-dimensional data is stored in memory using a format inefficient for the processing of the matrix computation unit. After the data is read from memory, the three-dimensional data is rearranged into an optimized linear order. In some embodiments, the optimized linear order is based on a 3D filter (or kernel) for performing 3D convolution. A memory organizer unit provides the matrix computation unit with data rearranged in the optimized linear order. For example, providing data in a linear order improves the efficiency of the matrix computation unit. In various embodiments, the matrix computation unit receives the 3D filter (or kernel) along with the rearranged data to perform 3D convolution. For example, filters corresponding to filter weights are received by the matrix computation unit and applied to the rearranged data.

[0039] Figure 1 This is a block diagram illustrating an embodiment of a system for solving artificial intelligence problems using neural networks. In the example shown, system 100 includes memory 101 and processing elements 111, 121, 131, and 151. In some embodiments, memory 101 is a shared on-chip memory component accessible by one or more processing elements (e.g., processing elements 111, 121, 131, and 151). For example, processing element 111 can read and write data to the on-chip memory corresponding to computations performed on a subset of a large data matrix. Processing element 121 can read and write data to the on-chip memory corresponding to computations performed on different subsets of the same large data matrix. In this way, different parts of a complex artificial intelligence problem can be solved by distributing the computational load across different processing elements. Processing elements 111, 121, 131, and 151 can each operate in parallel to solve a portion of a larger artificial intelligence problem. In various embodiments, Figure 1System 100 may include fewer or more processing elements. For example, the number of processing elements may be increased or decreased proportionally depending on the anticipated computing requirements. In some embodiments, memory 101 is a final level cache (LLC) and / or may be implemented using static random access memory (SRAM).

[0040] In some embodiments, processing elements are used to solve for layers of a neural network. For example, processing elements (e.g., one of processing elements 111, 121, 131, and / or 151) may be used to perform matrix operations such as convolution operations to apply the neural network to a set of input data retrieved from memory 101. One or more different filters, kernels, convolution matrices, etc., may be applied to the input data. In some embodiments, the convolution operation is a three-dimensional convolution operation, and the input data is three-dimensional input data. Processing elements 111, 121, 131, and 151 may access data from memory 101 via a memory organizer unit (not shown) that rearranges the requested data into an optimized layout for processing by the processing elements. For example, each processing element may include a memory organizer unit (not shown) as part of the processing element for requesting data. As another example, multiple processing elements of system 100 may share a memory organizer unit (not shown) for accessing memory 101. In some embodiments, the memory organizer unit is configured to read data from memory 101, such as data associated with layers of a convolutional neural network. Once data is read from memory 101, the memory organizer unit rearranges the data using a data layout format optimized for performing convolution operations. The rearranged data is then provided to the matrix computation unit (not shown) of the processing element for performing the convolution operation. For example, the data can be rearranged into an optimized linear order to efficiently utilize the computational resources of the matrix computation unit associated with the processing element.

[0041] Figure 2 This is a block diagram illustrating an embodiment of a system for solving artificial intelligence problems using neural networks. In the example shown, system 200 includes a memory 201, a memory organizer unit 205, a matrix calculation unit 211, and data paths 203, 207, and 209. Data paths 203, 207, and 209 connect different components of system 200. For example, data path 203 communicatively connects memory 201 and memory organizer unit 205, data path 207 communicatively connects memory organizer unit 205 and matrix calculation unit 211, and data path 209 communicatively connects memory 201 and matrix calculation unit 211. In some embodiments, memory 201 is... Figure 1 The memory 101. In some embodiments, the matrix calculation unit 211 is such as Figure 1A portion of the processing elements 111, 121, 131, and 151. For example, each processing element (e.g., Figure 1 The processing elements 111, 121, 131, and 151 include memory organizer units (e.g., memory organizer unit 205). In some embodiments, one or more processing elements share one or more memory organizer units. For example, a system with multiple processing elements may share a single memory organizer unit to access data. As another example, a system may include multiple memory organizer units shared by the processing elements.

[0042] In some embodiments, convolutional neural network layer data is stored in memory 101. The data may be stored in a data layout format that is efficient for reading and / or writing data to and / or from processing elements. While efficient for accessing or writing data, the memory data layout format may be inefficient for performing matrix calculations. Once data is retrieved from memory 201, memory organizer unit 205 rearranges the data into a layout for efficiently performing matrix calculations. In various embodiments, matrix calculation unit 211 receives three-dimensional convolution operations. For example, the three-dimensional convolution operation may specify a particular three-dimensional convolution operation and include references to data parameters and weight parameters. The data parameters and weight parameters may correspond to three-dimensional data, such as a three-dimensional matrix. For example, the data parameters may correspond to video data. The weight data may correspond to a three-dimensional convolution filter, such as a 3x3x3 kernel. In some embodiments, the data parameters reference convolutional neural network layer data stored in memory 201, while the weight parameters may reference a three-dimensional convolution filter stored in memory 201 or another suitable location. In various embodiments, matrix calculation unit 211 retrieves weight data associated with the weight parameters via data path 209. For example, weight data can be retrieved in a linear order format. In some embodiments, matrix computation unit 211 requests convolutional neural network layer data associated with data parameters from memory organizer unit 205 via data path 207. In some embodiments, a control unit (not shown) representing matrix computation unit 211 requests convolutional neural network layer data associated with data parameters. For example, the control logic of the control unit (not shown) can issue a three-dimensional convolution instruction to matrix computation unit 211 and an instruction to memory organizer unit 205 to retrieve three-dimensional data parameters. In response to the request for data, matrix organizer unit 205 requests data from memory 201 via data path 203 and rearranges the data in an optimized order (e.g., optimized linear order). The rearranged data is provided by memory organizer unit 205 to matrix computation unit 211 via data path 207.

[0043] In various embodiments, once the matrix computation unit 211 receives the data and weight operands, it performs matrix operations. In some embodiments, the matrix computation unit 211 is a matrix computation engine such as a dot product engine, and the matrix operation is a dot product operation. In various embodiments, the result of the matrix operation is written back to the memory 201 via a data path such as data path 209. In some embodiments, the matrix computation unit 211 utilizes a two-dimensional matrix computation engine. For example, the three-dimensional operands of the data matrix and weight matrix can be sliced ​​into two-dimensional matrices. A three-dimensional convolution operation is performed by performing multiple two-dimensional convolution operations using two-dimensional slices of the three-dimensional matrix parameters. The two-dimensional data matrix is ​​provided to the two-dimensional matrix computation engine of the matrix computation unit 211 by the memory organizer unit 205.

[0044] Figure 3 This is a flowchart illustrating an embodiment of the process of using neural networks to solve artificial intelligence problems. For example, Figure 3 The process is used to determine the result of one layer of a multi-layer neural network. In some embodiments, the neural network is a convolutional neural network (CNN), which is used to analyze three-dimensional data to solve complex artificial intelligence problems such as image recognition and recommendation. For example, CNNs can be used to analyze data such as video data to obtain relevant recommendations. Recommendations may include related videos, products and / or services related to the video, other video creators with relevant content, and / or non-video content related to the video, etc. In some embodiments, analysis is used to identify attributes of the content. For example, the completeness, identifiability, relevance, etc., of the content can be analyzed. In some embodiments, analysis is used to identify attributes of the content. Figure 1 System 100 and / or Figure 2 The system 200 uses neural networks.

[0045] At 301, a set of three-dimensional data is received. For example, input data corresponding to three-dimensional data (e.g., video data) is received. In some embodiments, the data is the output of a previous neural network layer. The three-dimensional data has a corresponding height, width, and number of channels, and can be stored in memory (e.g., shared memory or another suitable storage medium). In some embodiments, the data volume is very large, and the data layout format is optimized for storing and / or retrieving the data. For example, the data layout format can be optimized to efficiently retrieve a three-dimensional subset of a large three-dimensional input data source. Because the data is optimized for memory access, the data layout format may be inefficient for performing convolution operations. In some embodiments, the three-dimensional data set is stored in a storage medium such as… Figure 2 In the memory of memory 201, and via such as Figure 2 The memory organizer unit 205 is configured in an optimized linear order via... Figure 2Data paths 203 and / or 207 are retrieved. In some embodiments, such as... Figure 2 The matrix calculation unit 211 receives three-dimensional data.

[0046] At 303, a three-dimensional convolutional filter is received. In various embodiments, the three-dimensional convolutional filter is used in, for example... Figure 2 The matrix computation unit 211 receives the filter at its matrix computation unit. The size of the three-dimensional convolution filter can be determined based on the matrix computation unit. For example, a matrix computation unit processing two 32-element vectors can utilize a 3x3x3 convolution filter. The 3x3x3 convolution filter can be linearized, and each of the 27 elements can be loaded into the input vector. In some embodiments, the filter can be received in an optimized linear order. The optimized linear order can utilize zero-value elements, for example, as padding. For example, a 32-element vector includes 27 elements from the convolution filter, with the remaining 5 elements padded with zero values. In various embodiments, the spacing of the elements is arranged to optimize the convolution operation, and zero-value elements can be included between rows and / or columns. Depending on the application, the filter can be stored in memory in an optimized format, and / or rearranged into an optimized linear order once the filter is retrieved from memory. In some embodiments, the three-dimensional convolution filter is stored in a matrix such as... Figure 2 In the memory of memory 201, and via Figure 2 Data path 209 in such Figure 2 The matrix computation unit 211 receives the data at its matrix computation unit. In some embodiments, a three-dimensional convolutional filter is retrieved via a memory organizer unit, which rearranges the three-dimensional convolutional filter into a linearly optimized format.

[0047] At 305, a three-dimensional convolution operation is performed. Using the data set received at 301 and the convolution filter received at 303, the three-dimensional convolution operation is performed. In various embodiments, such as... Figure 2 The matrix computation unit 211 performs this operation. This operation can be performed in response to receiving a convolution operation of a specified data set and a convolution filter at the matrix computation unit. In some embodiments, the result of the convolution operation is written to memory, for example... Figure 1 The memory 101 and / or Figure 2 The memory 201. In some embodiments, the convolution result is stored as a partial result in a local accumulator and used for subsequent convolution operations.

[0048] Figure 4This is a flowchart illustrating an embodiment of the process of performing a three-dimensional convolution operation using matrix computation units. For example, using a neural network, an artificial intelligence problem is solved in part by analyzing one or more layers of the neural network using one or more matrix computation units from one or more processing elements. In some embodiments, a complex and large artificial intelligence problem is subdivided into smaller problems assigned to one or more different processing elements. Each processing element can use its associated matrix computation unit to perform its assigned sub-problem. In some embodiments, the processing element is... Figure 1 Processing elements 111, 121, 131, and / or 151. In some embodiments, the matrix calculation unit is... Figure 2 The matrix computation unit 211. In various embodiments, the matrix computation unit solves the convolution problem associated with the neural network in response to receiving a convolution operation instruction.

[0049] At 401, a 3D convolution operation instruction is received. In some embodiments, the convolution operation instruction is a matrix computation unit instruction that performs a 3D convolution operation on a matrix computation unit. This instruction may include references to data operands and weight operands. In various embodiments, the data operands reference a 3D data matrix, and the weight operands reference a 3D weight matrix. The 3D data operands may be convolutional neural network layer data, while the weight operands may be convolutional filters, such as a 3x3x3 kernel. In some embodiments, the operand references may be memory locations, registers, or other suitable data locations. In various embodiments, the operand references may describe the operands, such as the expected size and / or dimension of the operands. In some embodiments, the convolution instruction includes the output destination of the 3D convolution result. For example, the convolution result may be directed to be stored in a vector accumulator, written to memory, and / or stored in another suitable location.

[0050] At 403, an input 3D data operand is received. For example, the 3D data operand is retrieved from memory by requesting input 3D data from the memory organizer unit. In some embodiments, data is received by issuing a retrieval operation to the memory organizer unit and, for example, by referencing the requested data by address location. In some embodiments, the matrix computation unit passively receives data from the memory organizer unit. For example, control logic separate from the matrix computation unit issues a retrieval instruction to the memory organizer unit to provide data to the matrix computation unit. In some embodiments, the memory organizer unit is... Figure 2 The memory organizer unit 205 formats the received data operands using a data layout optimized for the matrix computation unit. For example, the data layout could be an optimized linear order. In some embodiments, in Figure 3 At point 301, execute step 403.

[0051] At 405, a three-dimensional weight operand is received. For example, the three-dimensional weight operand is retrieved from memory by requesting it. In some embodiments, weight data is retrieved from memory without using a memory organizer unit. In some embodiments, weight data is retrieved from memory by issuing a retrieval operation to the memory organizer unit and, for example, referencing the requested weight data by address location. Next, the memory organizer unit retrieves the requested weight data for the matrix calculation unit. In some embodiments, the memory organizer unit is... Figure 2 The memory organizer unit 205 formats the received weight data using a data layout optimized for the matrix computation unit. For example, the weight data layout may be an optimized linear order. The weight data may be retrieved as a two-dimensional slice of three-dimensional weight operands. In some embodiments, in Figure 3 At point 303, execute step 405.

[0052] At 407, matrix computation is performed. For example, the matrix computation unit performs matrix computation using received input 3D data and received 3D weight data. In some embodiments, the matrix computation is a 3D convolution. Weight data is applied to the input data to determine the 3D output result. In various embodiments, data and weight operands are stored in the input buffer of the matrix computation unit, while the output result is stored in the output buffer and / or accumulator of the matrix computation unit. In some embodiments, the output result may be written to memory. In some embodiments, in Figure 3 At point 305, execute step 407.

[0053] Figure 5 This is a flowchart illustrating an embodiment of the process of performing a three-dimensional convolution operation using a memory organizer unit. For example, an artificial intelligence problem is solved, in part, by analyzing one or more layers of a neural network using data requested from the memory organizer unit. Data can be requested from processing elements and / or matrix computation units and is provided by the memory organizer unit in an optimized linear order. In some embodiments, the processing element is... Figure 1 The processing elements 111, 121, 131, and / or 151. In some embodiments, Figure 5 The process is composed of, for example Figure 2 The memory organizer unit 205 is used to perform this. In some embodiments, in Figure 3 301 and / or Figure 4 403 executions Figure 5 The process, and / or execution in response to the matrix computation unit receiving a convolution operation instruction. Figure 5 The process. In various embodiments, it is performed on the three-dimensional weight operands. Figure 5 The process involves linearizing the three-dimensional weight matrix. For example, Figure 5 The process can be Figure 3 303 and / or Figure 4 Execution at point 405.

[0054] At 501, a request for three-dimensional data is received. For example, the memory organizer unit receives a request for three-dimensional data such as convolutional neural network layer data. The data may correspond to three-dimensional data such as video data and may be requested by the processing element in response to receiving a convolution operation instruction. In some embodiments, the matrix calculation unit of the processing element receives the convolution operation instruction and issues a request for the corresponding data to the memory organizer unit. In various embodiments, the request for three-dimensional data includes an address location, such as a memory location associated with the data. This location may include the depth, width, and number of channels to be retrieved from the memory location.

[0055] At 503, the requested 3D data is retrieved from memory. In response to the request received at 501, the memory organizer unit retrieves the requested data from memory. In some embodiments, the memory is such as... Figure 1 The memory 101 and / or Figure 2 The memory 201 is a memory. The memory organizer unit can retrieve data from the memory as one or more memory read requests. For example, the requested data may be a subset of a large three-dimensional data set and may be stored in the memory as non-contiguous data. One or more non-sequential reads may be issued to retrieve the requested data. In various embodiments, the data may be stored in the memory in a layout format that is efficient for memory access but inefficient for matrix computation.

[0056] At step 505, the retrieved 3D data is linearized. For example, data requested from memory is received and rearranged into a linear format. In various embodiments, the data is linearized to an optimized linear order. This optimized linear order allows the data to be processed efficiently by the matrix computation unit, for example, for performing convolution operations. Data can be linearized when one or more data requests to memory are satisfied and the data is received by the memory organizer unit. In various embodiments, the requested 3D data is linearized into blocks of a size determined by the operation unit of the matrix computation unit. For example, for a matrix computation unit operating on a 32-element input vector, the memory organizer unit can linearize the requested data into a 32-element vector. In some embodiments, the data is linearized by slicing the requested 3D data into multiple 2D slices. Each 2D slice is then linearized to an optimized linear order.

[0057] At 507, linearized three-dimensional data is provided to the matrix computation unit. For example, the requested data is provided to the matrix computation unit in an optimized linear order for performing matrix computations such as convolution operations. In some embodiments, the data is provided as a continuous, linearly ordered data stream. The data can be organized based on two-dimensional slices of the requested three-dimensional data.

[0058] Figure 6 This is a flowchart illustrating an embodiment of the process of performing a three-dimensional convolution operation using a memory organizer unit. In the example shown, three-dimensional convolution is performed by slicing three-dimensional data into multiple two-dimensional slices. The sliced ​​two-dimensional data is retrieved from memory, linearized, and provided to a matrix computation unit for performing multiple two-dimensional convolution operations that constitute the three-dimensional convolution. In some embodiments, the three-dimensional convolution operation is performed by a processing element (e.g., Figure 1 The processing elements 111, 121, 131 and / or 151, and their respective matrix calculation units, perform the operation. In some embodiments, Figure 6 The process is composed of, for example Figure 2 The memory organizer unit 205 is used to perform this. In some embodiments, in Figure 3 301 and / or Figure 4 403 executions Figure 6 The process, and / or execution in response to the matrix computation unit receiving a convolution operation instruction. Figure 6 The process. In some embodiments, in Figure 5 Execute at 501, 503, 505 and / or 507 Figure 6 The process. In various embodiments, it is performed on the three-dimensional weight operands. Figure 6 The process involves linearizing the three-dimensional weight matrix using two-dimensional slices of the weight matrix. For example, Figure 6 The process can be Figure 3 303 Figure 4 405 and / or Figure 5 Execute at 501, 503, 505 and / or 507.

[0059] In section 601, three-dimensional data is sliced ​​into two-dimensional data. For example, a set of three-dimensional data is conceptually sliced ​​into two-dimensional slices. As another example, a three-dimensional matrix is ​​sliced ​​into multiple two-dimensional matrices. Once sliced, each two-dimensional slice can be retrieved from memory and formatted in an optimized linear order for performing matrix operations.

[0060] At 603, a selected two-dimensional data slice is retrieved from memory. For example, a two-dimensional data slice is selected from segmented three-dimensional data. In some embodiments, one or more memory reads may be issued to the memory for the selected two-dimensional data slice to retrieve data associated with that slice. The data associated with the slice is retrieved in a layout format used to store the data. In some embodiments, the layout format may be non-contiguous and not optimized for three-dimensional convolution operations.

[0061] At 605, the retrieved 2D data slice is linearized. For example, the retrieved 2D data slice is formatted with an optimized linear order for matrix computation units. The slice is linearized into a vector, and padding can be inserted. For example, zero-value elements can be inserted between rows (or columns) to align the 2D data with the linearized weight matrix. In various embodiments, the utilized linear order is optimized for performing multiple 2D convolutions to complete a 3D convolution.

[0062] At 607, linearized two-dimensional data slices are provided to the matrix computation unit. For example, using a data path between the memory organizer unit and the matrix computation unit, linearized two-dimensional data slices are provided to the matrix computation unit for performing convolution operations. In some embodiments, linearized two-dimensional data slices are provided to an input buffer of the matrix computation unit for receiving input operands. In various embodiments, the matrix computation unit receives the provided linearized two-dimensional data slices for performing two-dimensional convolution operations. For example, the matrix computation unit receives a pair of linearized two-dimensional slices (e.g., a slice corresponding to a three-dimensional data matrix operand and a slice corresponding to a three-dimensional weight matrix operand) to perform matrix operations.

[0063] At 609, it is determined whether additional 2D data is available for processing. If additional 2D data slices are available, the processing loop returns to 603. In this way, the processing loops through all slices of the 3D data until every slice has been processed and no additional slices remain unprocessed. If no additional 2D data slices are available, the processing ends. In some embodiments, the completion of the processing indicates that the entire requested 3D data has been provided to the matrix computation unit in an optimized linear order.

[0064] Figure 7A and Figure 7B This is a diagram illustrating an example slice of a two-dimensional matrix used to perform a two-dimensional convolution operation. In various embodiments, the two-dimensional convolution operation is a depthwise convolution operation. Figure 7AIn the diagram, weight matrix slices 701, 703, and 705 represent example two-dimensional (e.g., 3x3) slices of a three-dimensional (e.g., 3x3x3) matrix of weight elements. Other dimensions of the three-dimensional weight (or filter) matrix may also be suitable, such as 5x5x5, 7x7x7, 9x9x9, etc. In various embodiments, the three-dimensional 3x3x3 weight matrix may correspond to... Figure 3 The 3D convolutional filter received at position 303 and / or Figure 4 The 3D weight operand is received at position 405. Regarding the 3D 3x3x3 weight matrix, weight matrix slice 701 is a 3x3 slice at depth 1, weight matrix slice 703 is a 3x3 slice at depth 2, and weight matrix slice 705 is a 3x3 slice at depth 3. In the example shown, the subscript associated with each weight element uses (row, column, depth) notation. Weight matrix slice 701 references the element at depth 1. For the first row of weight matrix slice 701, W... 1,1,1 It is the weighted element located in column 1 and depth 1, W 1,2,1 These are the weighted elements located in column 2 and depth 1, and W. 1,3,1 It is the weight element located in column 3 and depth 1.

[0065] exist Figure 7B In this context, data matrix slice 711 represents an exemplary two-dimensional slice of a three-dimensional matrix of data elements. The dimension of data matrix slice 711 is greater than that of weight matrix slices 701, 703, and 705, and is based on the size capability of matrix computation units. Figure 7B In the example, only rows 1-4 and 32-34 of the first depth of data matrix slice 711 are shown. The width of data matrix slice 711 can extend beyond 11 columns. Similarly, the height of data matrix slice 711 can extend beyond 34 rows (not shown). Figure 7B In the example shown, data matrix slice 711 is depicted with 34 rows to ensure compatibility with 3x3 weight matrix slices 701, 703, and 705, as well as matrix computation units with 32 vector units. In the example shown, data matrix slice 711 is simply a slice within a three-dimensional data matrix. In various embodiments, the three-dimensional data matrix may correspond to... Figure 3 The 3D data set received at 301 locations and / or Figure 4 The 3D data operand received at position 403.

[0066] In some embodiments, weight matrix slices 701, 703, and 705 and data matrix slice 711 are examples of two-dimensional matrices sliced ​​along the depth axis from a three-dimensional matrix. Other slicing techniques are also suitable. In some embodiments, the slices are cut along the depth axis. Figure 6At position 601, the operation is performed, and at least a portion of the slice is linearized and provided to the matrix computation unit for performing two-dimensional convolution operations. In some embodiments, using... Figure 2 The matrix computation unit 211 is used to perform convolution operations, such as depthwise convolution.

[0067] Figure 8 This is a flowchart illustrating an embodiment of the process of performing a two-dimensional convolution operation using linearized operands and matrix computation units. For example, using... Figure 6 In this process, the input matrix is ​​linearized from a slice of a two-dimensional matrix and fed as an input vector to the matrix computation unit. In some embodiments, each pair of linearized input vectors (e.g., a weight input vector and a data input vector) is processed by the vector unit of the matrix computation unit. The vector unit of the matrix computation unit uses the linearized operands to compute at least a partial result of the two-dimensional convolution operation. In some embodiments, this partial result is a dot product result used to compute a three-dimensional convolution result. In various embodiments, Figure 8 The process in Figure 6 Executed at points 603, 605, and 607. In some embodiments, using Figure 2 The memory organizer unit 205 and the matrix calculation unit 211 are used to perform the operation. Figure 8 The process.

[0068] At 801, data input vectors are identified. For example, one or more data input vectors are identified from a slice of data matrix. In some embodiments, a data input vector is identified and prepared for each vector unit of the matrix computation unit. The data input vector may include overlapping elements and corresponds to selected elements for performing matrix operations (e.g., two-dimensional convolution operations).

[0069] At 803, the data input vector is prepared and loaded into the vector unit. For example, elements corresponding to a subset of the data matrix slice are identified. In some embodiments, the data matrix slice can be a two-dimensional slice of a larger three-dimensional data matrix. In various embodiments, the identified elements are subsets of the data matrix slice and are selected based on the corresponding weight matrix slice, particularly the dimension of the weight matrix slice and the number of rows to be linearized. For example, in some embodiments, the entire weight matrix slice can be expanded and linearized to fit the vector unit operands. However, as the dimension of the weight matrix slice increases relative to the size of the operands supported by the vector unit, the weight matrix slice may become too large to be expanded into a single input vector. Instead, the weight matrix slice and the corresponding data matrix slice are expanded and linearized over multiple iterations.

[0070] In various embodiments, the weight matrix slice is linearized by fully expanding the slice in one iteration or partially expanding it over multiple iterations. Whether the matrix slice is fully or partially expanded can depend on the dimension of the two-dimensional matrix parameters (specifically the weight matrix slice) in relation to the size of the input vector supported by the matrix computation unit. For example, in a scenario with a 3x3 weight matrix slice and supporting an input vector of 32 elements, all nine elements of the weight matrix slice (corresponding to three rows) can be expanded into a single 32-element weight input vector. As part of preparing the corresponding data input vector, elements are selected from the columns corresponding to the three-row data matrix slice. The elements selected from the three rows are linearized into a one-dimensional vector. The number of columns used can be based on the size of the data input vector. In an example scenario with a 32-element data input vector, three rows are identified, and 10 elements (corresponding to 10 columns) are selected for each row. A total of 30 elements are selected from the three rows and ten columns. The remaining two positions in the data input vector, which are insufficient to fit the entire column from the identified three rows, are padded, for example, with two zero-value elements. In this way, a 3x10 slice of the data matrix can be expanded and linearized into a data input vector, padded with two zero-value elements. Once ready, the data input vector is loaded into the appropriate vector cell of the matrix computation unit. In various embodiments, each vector cell of the matrix computation unit is loaded with the prepared data input vector. For example, each vector cell of the matrix computation unit can be loaded by selecting elements starting from different rows.

[0071] In some embodiments, the initial number of columns selected to prepare the data input vector is determined by dividing the size of the data input vector by the number of rows to be expanded. For example, a 32-element data input vector can be loaded with elements from sixteen, ten, or eight columns, respectively, when expanding two, three, or four rows. Any unused elements in the data input vector can be filled with zero-value elements as padding. Once the number of columns is determined, the same number of columns can be used in each iteration. For example, a 7x7 matrix slice can be loaded into a 32-element data input vector in four iterations. For the first three iterations, two rows are expanded in each iteration (i.e., rows 1-2, 3-4, and 5-6), while the fourth iteration expands the last row (i.e., row 7). Each iteration can select 16 elements from different columns of the corresponding selected row. The last iteration has additional padding because only one row is expanded. In various embodiments, zero-value elements are used as padding elements. Padding elements with a value of zero allow vector units to perform certain vector operations, such as vector multiplication and vector addition, without affecting the resulting vector sum or dot product.

[0072] In various embodiments, once properly prepared using an appropriate format, the data input vector is loaded into the corresponding vector cell of the matrix computation unit. In some embodiments, the data input vector is prepared and loaded into each vector cell of the matrix computation unit. For example, the data input vector prepared for each vector cell may correspond to a submatrix of a slice of the data input matrix, where each submatrix begins from a different row of the slice. For example, in the case where the weight matrix slice is a fully expanded 3x3 matrix, the first vector cell may correspond to a submatrix with elements from rows 1-3, the second vector cell may correspond to a submatrix with elements from rows 2-4, the third vector cell may correspond to a submatrix with elements from rows 3-5, and so on. In one embodiment, when using 32 vector cells, the 32nd vector cell may correspond to a submatrix with elements from rows 32-34.

[0073] At 805, weight input rows are identified. For example, one or more rows from a slice of the weight input matrix are identified for performing vector unit operations. In some embodiments, the weight input matrix slice is fully expanded, and each row of the weight input matrix slice is identified for expansion and linearization. In various embodiments, the weight input matrix slice may be too large to be fully expanded and is only partially expanded. Only the rows selected for expansion are identified for further processing at 807. For example, a 3x3 weight matrix slice (with 9 elements) can be fully expanded into a 32-element weight input vector, thus utilizing each row of the weight matrix slice. As another example, a 7x7 weight matrix slice (with 49 elements) cannot be fully expanded into a 32-element weight input vector. Only the expanded rows are identified for the current iteration. In various embodiments, successive iterations are used to identify the remaining rows.

[0074] At 807, a weight input mask is prepared and broadcast to the vector units. For example, elements corresponding to the weight input rows identified at 805 are prepared into the weight input mask. The weight input mask is linearized into a one-dimensional input vector for the vector units of the matrix computation unit. In various embodiments, the linearized weight elements are formatted with padding to align the beginning of each weight input row with the associated data input row prepared at 803. For example, using a 32-element vector and a 3x3 weight matrix slice (where all rows are identified at 805), all nine elements of the weight matrix slice are selected. The nine elements of the weight matrix slice are formatted into the weight input mask using an additional 23 padding elements. In some embodiments, 30 elements are used for three rows of the selected 3x3 matrix. Each row includes three elements from a row, followed by seven padding elements. The remaining elements of the data input vector not used for the rows of the selected 3x3 matrix (two elements in this example) are padded with additional padding elements. In various embodiments, zero-value elements are used as padding elements. Padding elements with a value of zero allow vector units to perform multiplication and addition vector operations without affecting the resulting vector sum.

[0075] In various embodiments, the formatted weight input mask is an input vector broadcast to each vector unit of the matrix computation unit. For example, the weight input vector is used to store a prepared weight input mask in a linearized one-dimensional vector format. The weight input mask is then broadcast as an input vector operand to each appropriate vector unit. For example, a single weight input mask may be prepared and used by all vector units of the matrix computation unit. In some embodiments, row padding elements are used to allow the weight input mask to be shifted to realign the elements of the weight matrix slice with different columns of the data matrix slice referenced in the data input vector.

[0076] At 809, vector unit operations are performed. Using the input vector loaded from the data input vector and the broadcast weight input mask, the corresponding vector unit of the matrix computation unit performs vector unit operations. In some embodiments, vector unit operations include vector multiplication and addition. For example, each element from the data input vector is multiplied by its corresponding element from the weight input mask. The result is a vector of multiplication results. In some embodiments, vector multiplication is performed by the vector multiplication unit of the vector unit. Using the vector multiplication result, a vector sum result is computed by adding each element in the vector of the multiplication result. In some embodiments, the vector adder unit of the vector unit is used to determine the sum. For example, the vector adder unit of the adder tree can compute the sum of vector elements. In some embodiments, the matrix computation unit includes an accumulator for adding the vector adder result to the previous vector adder result. For example, an accumulator can be used to accumulate intermediate or partial results from each iteration to determine a running cumulative sum. Accumulation allows the vector sum from one iteration (or pass) to be added to the result of subsequent iterations (or passes). In various embodiments, zero-padding elements prepared for the weight input mask (and / or data input vector) result in zero-multiplication and have no effect on vector addition results. In some embodiments, each vector unit of the matrix computation unit performs a dot product operation using its corresponding loaded data input vector and broadcast weight input mask. The dot product result may be accumulated with the result from the previous iteration (or pass) and / or stored to be added to the result of future iterations.

[0077] At 811, the vector result is output. For example, the vector result determined by performing vector unit operations at 809 is output from the matrix calculation unit. In some embodiments, the vector result is output to an output vector register, which is used to write the output to memory, for example... Figure 2 The memory 201. In some embodiments, the result is output and used as a parameter for subsequent matrix operations.

[0078] Although the foregoing embodiments have been described in detail for clarity of understanding, the invention is not limited to the details provided. Many alternative ways of implementing the invention exist. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A processor system, comprising: The memory organizer unit is configured as follows: Receive requests for 3D data of the data matrix of a convolutional neural network layer; Obtain the requested 3D data from memory; The obtained three-dimensional data is rearranged in an optimized linear order, including by aligning slices of the obtained data matrix that are different from the weight matrix with a linearized version of the weight matrix, including by aligning slices of the obtained data matrix that are different from the weight matrix with a linearized version of the weight matrix, including by aligning slices of the obtained three-dimensional data that are different from the weight matrix with one or more zero-value elements, to optimize the data organization layout at the hardware component that interfaces between the memory and the matrix computation unit. as well as The data, rearranged in the optimized linear order, is provided to the matrix calculation unit; as well as The matrix computation unit is configured to perform at least a portion of a three-dimensional convolution using at least a portion of the provided data rearranged in the optimized linear order.

2. The system according to claim 1, wherein, The three-dimensional data of the convolutional neural network layer is video data.

3. The system according to claim 1 or claim 2, wherein, The convolutional neural network layer is a layer of a neural network used to determine recommendations; or, the convolutional neural network layer is a layer of a neural network used to identify content.

4. The system according to claim 1 or claim 2, wherein, The matrix computation unit is configured to receive a set of weights for performing the three-dimensional convolution; and the set of weights is a three-dimensional matrix; and the set of weights is a 3x3x3 kernel.

5. The system according to claim 4, wherein, The weight set is formatted in a linear order.

6. The system according to claim 1 or claim 2, wherein, The matrix computation unit is configured to perform dot product results; and / or, wherein the matrix computation unit is a dot product engine.

7. The system according to claim 1 or claim 2, wherein, The rearranged data comprises multiple linear two-dimensional slices of the three-dimensional data.

8. The system according to claim 1 or claim 2, wherein, The matrix calculation unit is configured to receive a three-dimensional convolution operation instruction; and the three-dimensional convolution operation instruction includes a first reference to data parameters and a second reference to weight parameters.

9. A method for calculating three-dimensional data, comprising: Receive requests for 3D data of the data matrix of a convolutional neural network layer; Obtain the requested 3D data from memory; The obtained three-dimensional data is rearranged in an optimized linear order, including by aligning slices of the obtained data matrix that are different from the weight matrix with a linearized version of the weight matrix, including by aligning slices of the obtained data matrix that are different from the weight matrix with a linearized version of the weight matrix, including by aligning slices of the obtained three-dimensional data that are different from the weight matrix with one or more zero-value elements, to optimize the data organization layout at the hardware component that interfaces between the memory and the matrix computation unit. as well as The data, rearranged in an optimized linear order, is provided to the matrix calculation unit.

10. The method according to claim 9, wherein, The optimized linear sequence is used to perform 3D convolution operations.

11. A method for calculating three-dimensional data, comprising: Receive a three-dimensional convolution operation instruction, wherein the three-dimensional convolution operation instruction specifies the three-dimensional data of the data matrix of the convolutional neural network layer; Request the three-dimensional data of the convolutional neural network layer from the memory organizer unit; A linearized version of the weight matrix of the receiving convolutional filter; The requested 3D data is received from the memory organizer unit in an optimized linear order, wherein the optimized linear order aligns slices of the 3D data, which are different from the weight matrix, with the linearized version of the weight matrix, and optimizes the data organization layout at the hardware component interfacing between the memory and the matrix computation unit by including one or more zero-value elements inserted between the elements of one or more rows or columns of the slices of the 3D data. as well as At least a portion of the 3D convolution is performed using at least a portion of the requested 3D data received in an optimized linear order and at least a portion of the received 3D weight data.

12. The method according to claim 11, wherein, The three-dimensional convolution operation instruction specifies the weight parameters corresponding to the three-dimensional weight data.

13. The method according to claim 11 or claim 12, wherein, The three-dimensional convolution operation instruction includes the memory address location of the three-dimensional data of the specified convolutional neural network layer.

14. The method according to claim 11 or claim 12, wherein, The three-dimensional data of the convolutional neural network layer is video data.

15. The method according to claim 11 or claim 12, wherein, Performing at least a portion of the three-dimensional convolution includes performing a dot product operation.

Images