A data processing method, apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a data processing method, apparatus, electronic device, and storage medium. The method comprises: in response to a data processing instruction for a target input feature map, obtaining a target input data block from an input data block arrangement corresponding to the target input feature map in a memory; the input data block arrangement is based on the data throughput of a general matrix processing engine in each clock cycle, and the target input feature map is divided into multiple input data blocks and stored in the memory according to a preset arrangement; the target input data blocks are cached; and based on the cached target input data blocks, matrix multiplication operations are performed using the general matrix processing engine to obtain data processing results. The embodiments of the present disclosure improve the data processing efficiency based on Img2Col, thereby improving the data processing efficiency of electronic devices based on deep learning networks.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device, electronic device, and storage medium. Background Art

[0002] With the continuous development of artificial intelligence (AI) technology, various AI models have been widely applied in various fields, which has also put forward new requirements for hardware computing architecture. Convolution, full connection, and matrix multiplication are the most basic and computationally intensive operators in various deep learning networks. Therefore, the data processing efficiency of electronic devices that process data based on deep learning networks is closely related to the computational efficiency of these operators.

[0003] Currently, when electronic devices process data based on deep learning networks, they perform general matrix multiplication (GEMM) operations based on Img2Col (Image to Column). However, the data processing efficiency of electronic devices based on deep learning networks in related technologies is low and cannot meet the business needs of related businesses. Summary of the Invention

[0004] The present disclosure provides a data processing method, device, and system to at least address the problem of low data processing efficiency of electronic devices based on deep learning networks in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

[0006] In response to a data processing instruction for a target input feature map, a target input data block is obtained from an input data block arrangement corresponding to the target input feature map in a memory; the input data block arrangement is obtained by dividing the target input feature map into a plurality of input data blocks based on a data throughput of a general matrix processing engine in each clock cycle and storing the input data blocks in the memory in a preset arrangement;

[0007] caching the target input data block;

[0008] Based on the cached target input data block, the general matrix processing engine is used to perform matrix multiplication processing to obtain a data processing result.

[0009] In an exemplary embodiment, performing matrix multiplication processing on the target input data block based on the cache using the general matrix processing engine to obtain a data processing result includes:

[0010] Performing matrix conversion on the cached target input data block according to the convolution parameters of the current convolution layer;

[0011] When it is determined that the matrix conversion of the target input data block is completed, the feature block matrix obtained by the matrix conversion is output to the general matrix processing engine, so that the general matrix processing engine performs matrix multiplication operation processing to obtain a data processing result.

[0012] In an exemplary embodiment, in response to a data processing instruction for a target input feature map, obtaining a target input data block from an input data block arrangement corresponding to the target input feature map in a memory includes:

[0013] In response to a data processing instruction for the target input feature map, entering a waiting state from an idle state;

[0014] Sending a read data request to the memory in the waiting state, and entering the ready state from the waiting state; the read data request is used to request to obtain the input data block in the input data block arrangement;

[0015] When a target input data block returned by the memory based on the read data request is received, a matrix conversion working state is entered; the target input data block includes a group of input data blocks arranged in a row direction.

[0016] In an exemplary embodiment, performing matrix conversion on the cached target input data block according to the convolution parameters of the current convolution layer includes:

[0017] Determining address offset information according to a feature map size parameter of the target input feature map, the convolution parameter, and a block size of the input data block; the feature map size parameter includes a feature map width, a feature map height, and a number of feature map channels;

[0018] Obtaining a current input data block to be processed from the cached target input data block according to the address offset information;

[0019] Performing matrix expansion on the current input data block to be processed according to the convolution parameters to obtain an expanded block matrix, and transposing the expanded block matrix to obtain the feature block matrix;

[0020] In a case where the current input data block to be processed is not the last input data block in the target input data block, the steps of obtaining the current input data block to be processed from the cached target input data block according to the address offset information and the matrix expansion are performed until the current input data block to be processed is the last input data block in the target input data block, and it is determined that the matrix conversion of the target input data block is completed, and the matrix conversion working state enters the completion state.

[0021] In an exemplary embodiment, the method further comprises:

[0022] If there are still input data blocks that have not been matrix-converted when the input data blocks are arranged in the column direction, then the waiting state is entered again from the completion state;

[0023] The step of sending a read data request to the memory in the waiting state is executed until the input data blocks are arranged in the column direction and no input data blocks that have not been matrix-converted exist, and the idle state is entered from the completion state.

[0024] In an exemplary embodiment, performing matrix expansion on the current input data block to be processed according to the convolution parameters includes:

[0025] Determining overlapping data corresponding to adjacent sliding windows in the input data block to be processed according to a moving step size of the sliding window in the convolution parameter;

[0026] Temporarily storing the overlapping data according to the channel dimension as data to be multiplexed;

[0027] Acquire data to be converted from the data corresponding to the input data block to be processed in the sliding window, wherein the data to be converted does not include the overlapping data;

[0028] Matrix expansion is performed based on the data to be multiplexed and the data to be converted.

[0029] In an exemplary embodiment, before obtaining the target input data block from the input data block arrangement corresponding to the target input feature map in the memory in response to the data processing instruction for the target input feature map, the method further includes:

[0030] Convert the target input feature map into input data in the memory; the height of the input data is the length multiplied by the width of the target input feature map, and the width of the input data is the number of channels of the target input feature map;

[0031] Obtaining the data throughput of the general matrix processing engine in each clock cycle;

[0032] Based on the data throughput of the general matrix processing engine in each clock cycle, the input data is divided into multiple input data blocks, the data amount of each input data block is the data throughput, the internal data of each input data block is arranged in a Z-shape, and the input data blocks are continuous in the channel direction.

[0033] In an exemplary embodiment, dividing the input data into a plurality of input data blocks based on the data throughput of the general matrix processing engine in each clock cycle includes:

[0034] When the size of the input data is not an integer multiple of the size indicated by the data throughput, padding the input data to obtain padded input data; the size of the padded input data is an integer multiple of the size indicated by the data throughput;

[0035] The padded input data is divided into the plurality of input data blocks based on a data throughput of the general matrix processing engine in each clock cycle.

[0036] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing apparatus, including:

[0037] An input data block acquisition unit is configured to execute a data processing instruction in response to a target input feature map, and acquire a target input data block from an input data block arrangement corresponding to the target input feature map in a memory; the input data block arrangement is obtained by dividing the target input feature map into a plurality of input data blocks based on a data throughput of a general matrix processing engine in each clock cycle and storing the input data blocks in the memory in a preset arrangement;

[0038] An input data block caching unit, configured to cache the target input data block;

[0039] The data processing module is configured to execute the target input data block based on the cache, use the general matrix processing engine to perform matrix multiplication processing, and obtain a data processing result.

[0040] In an exemplary embodiment, the data processing module includes:

[0041] a matrix conversion unit, configured to perform matrix conversion on the cached target input data block according to the convolution parameters of the current convolution layer;

[0042] The feature block matrix output unit is configured to output the feature block matrix obtained by the matrix transformation to the general matrix processing engine when it is determined that the matrix transformation of the target input data block is completed, so that the general matrix processing engine performs matrix multiplication calculation.

[0043] In an exemplary embodiment, the input data block acquiring unit includes:

[0044] A first state transition unit is configured to execute a data processing instruction in response to the target input feature map and enter a waiting state from an idle state;

[0045] A second state conversion unit is configured to execute sending a read data request to the memory in the waiting state, and enter a ready state from the waiting state; the read data request is used to request to obtain an input data block in the input data block arrangement;

[0046] The third state conversion unit is configured to enter a matrix conversion working state when receiving a target input data block returned by the memory based on the read data request; the target input data block includes a group of input data blocks arranged in a row direction.

[0047] In an exemplary embodiment, the matrix conversion unit includes:

[0048] An address offset determining unit is configured to determine address offset information according to a feature map size parameter of the target input feature map, the convolution parameter, and a block size of the input data block; the feature map size parameter includes a feature map width, a feature map height, and a number of feature map channels;

[0049] a cache acquisition unit, configured to acquire a current input data block to be processed from the target input data block in the cache according to the address offset information;

[0050] a matrix expansion unit configured to perform matrix expansion on the current input data block to be processed according to the convolution parameters to obtain an expanded block matrix, and transpose the expanded block matrix to obtain the feature block matrix;

[0051] The fourth state transition unit is configured to execute, when the current input data block to be processed is not the last input data block in the target input data block, the steps of obtaining the current input data block to be processed from the cached target input data block according to the address offset information and the matrix expansion, until the current input data block to be processed is the last input data block in the target input data block, determine that the matrix conversion of the target input data block is completed, and enter the completion state from the matrix conversion working state.

[0052] In an exemplary embodiment, the matrix conversion unit further includes:

[0053] a fifth state transition unit configured to execute, if the input data blocks arranged in the column direction still have input data blocks that have not been matrix-transformed, to enter the waiting state again from the completion state;

[0054] The sixth state transition unit is configured to execute the step of sending a read data request to the memory in the waiting state until the input data blocks are arranged in the column direction and no input data blocks that have not been matrix-converted exist, and enter the idle state from the completion state.

[0055] In an exemplary embodiment, the matrix expansion unit includes:

[0056] an overlapping data determining unit, configured to determine overlapping data corresponding to adjacent sliding windows in the input data block to be processed according to a moving step length of the sliding window in the convolution parameter;

[0057] an overlapping data temporary storage unit, configured to temporarily store the overlapping data according to a channel dimension as data to be multiplexed;

[0058] a data-to-be-converted acquiring unit configured to acquire data to be converted from the data corresponding to the input data block to be processed from a sliding window, wherein the data to be converted does not include the overlapping data;

[0059] The matrix expansion subunit is configured to perform matrix expansion based on the data to be multiplexed and the data to be converted.

[0060] In an exemplary embodiment, the apparatus further comprises:

[0061] An input conversion unit is configured to convert the target input feature map into input data in the memory; the height of the input data is the length multiplied by the width of the target input feature map, and the width of the input data is the number of channels of the target input feature map;

[0062] A throughput acquisition unit is configured to acquire the data throughput of the general matrix processing engine in each clock cycle;

[0063] The data block division unit is configured to divide the input data into multiple input data blocks based on the data throughput of the general matrix processing engine in each clock cycle, the data volume of each input data block is the data throughput, the internal data of each input data block is arranged in a Z-shape, and the input data blocks are continuous in the channel direction.

[0064] In an exemplary embodiment, the data block division unit includes:

[0065] a data padding unit configured to perform data padding on the input data when the size of the input data is not an integer multiple of the size indicated by the data throughput, so as to obtain padded input data; wherein the size of the padded input data is an integer multiple of the size indicated by the data throughput;

[0066] The data block division subunit is configured to divide the padded input data into the multiple input data blocks based on the data throughput of the general matrix processing engine in each clock cycle.

[0067] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0068] processor;

[0069] a memory for storing instructions executable by the processor;

[0070] The processor is configured to execute the instructions to implement the data processing method of the first aspect mentioned above.

[0071] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data processing method of the first aspect described above.

[0072] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which implements the data processing method of the first aspect when executed by a processor.

[0073] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0074] By responding to a data processing instruction for a target input feature map, a target input data block is obtained from an input data arrangement corresponding to the target input feature map in a memory. The input data arrangement is based on the data throughput of a general matrix processing engine in each clock cycle, and the target input feature map is divided into multiple input data and stored in a preset arrangement in the memory. The target input data block is then cached, and based on the cached target input data block, a general matrix processing engine is used to perform matrix multiplication operations to obtain a data processing result, thereby realizing regular memory access, improving data access efficiency, greatly improving the data processing efficiency based on Img2Col, and thereby improving the data processing efficiency of electronic devices based on deep learning networks.

[0075] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0077] Figure 1 is a schematic diagram of an application environment of a data processing method according to an exemplary embodiment;

[0078] Figure 2 is a flowchart illustrating a data processing method according to an exemplary embodiment;

[0079] Figure 3 is a flowchart illustrating another data processing method according to an exemplary embodiment;

[0080] Figure 4 This is an example of arranging input data blocks corresponding to a target input feature map in a memory according to an exemplary embodiment;

[0081] Figure 5 is a flowchart illustrating another data processing method according to an exemplary embodiment;

[0082] Figure 6 is a flowchart illustrating another data processing method according to an exemplary embodiment;

[0083] Figure 7 is a block diagram of a data processing device according to an exemplary embodiment;

[0084] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0085] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0086] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0087] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0088] See also Figure 1 , which is a schematic diagram of an application environment of a data processing method according to an exemplary embodiment. The application environment may include a terminal 110 and a server 120. The terminal 110 and the server 120 may be connected via a wired network or a wireless network.

[0089] The terminal 110 may be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited thereto. Client software such as an application (App) that provides data processing functions may be installed in the terminal 110. The application may be a standalone application or a subroutine within the application. The data processing function may be implemented based on a pre-trained AI model, such as a deep learning network model such as GPT or Diffusion. The user of the terminal 110 may log in to the application using pre-registered user information, which may include an account number and password.

[0090] The server 120 can be a server that provides background services for the application in the terminal 110, or it can be other servers connected and communicating with the background server of the application. It can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0091] The server 120 may store an AI model for implementing data processing, and the server 120 may train and update the AI model according to a predetermined period. When data processing is required, the terminal 110 may send the corresponding data to the server 120, and the server 120 may call the AI model to process the data and return the processing results to the terminal 110. It is understandable that the terminal 110 may also download the AI model from the server 120 and store it locally. When data processing is required, the terminal 110 may directly call the locally stored AI model to perform data processing.

[0092] Figure 2 is a flow chart showing a data processing method according to an exemplary embodiment. Figure 2 As shown, the data processing method of the embodiment of the present disclosure includes the following steps:

[0093] In step S21, in response to a data processing instruction for a target input feature map, a target input data block is obtained from an input data block arrangement corresponding to the target input feature map in a memory.

[0094] The input data block arrangement is based on the data throughput of the general matrix processing engine in each clock cycle, and the target input feature map is divided into multiple input data blocks and stored in the memory according to a preset arrangement method.

[0095] The target input feature map refers to the feature map that will be used as the input of the current convolutional layer for convolution calculation.

[0096] Based on this, in some exemplary embodiments, before the above step S21, as Figure 3 As shown in , the method may further include:

[0097] In step S31, the target input feature map is converted into input data in the memory.

[0098] The height of the input data is the length multiplied by the width of the target input feature map, and the width of the input data is the number of channels of the target input feature map.

[0099] In step S33 , the data throughput of the general matrix processing engine in each clock cycle is obtained.

[0100] The data throughput of the general matrix processing engine in each clock cycle is consistent with the bandwidth of the memory.

[0101] In step S35, based on the data throughput of the general matrix processing engine in each clock cycle, the input data is divided into multiple input data blocks, the data amount of each input data block is the data throughput, the internal data of each input data block is arranged in a Z shape, and the input data blocks are continuous in the channel direction.

[0102] The multiple input data blocks have the same size.

[0103] In order to meet the requirements of large-scale AI chips for high bandwidth and high throughput data volume, the embodiments of the present disclosure store the target input feature map in the form of blocks, and the data volume of an input data block is calibrated as the bandwidth of the on-chip memory. Generally, the bandwidth of the memory is equal to the data throughput of the general matrix processing engine in each clock cycle.

[0104] Taking the memory bandwidth as 16x16 bytes (i.e. 256 bytes) as an example, the size of an input data block is 16x16 bytes. Figure 4The figure shows an example of the arrangement of the input data blocks corresponding to the target input feature map in the memory. The size of the target input feature map is H*W*C, where H represents height, W represents width, and C represents channel. The target input feature map is converted into input data in the memory. The height of the input data is W*H and the width is C. At this time, the input data is continuous along the C direction in the memory, and the rows are arranged in a Z-shaped pattern, connected end to end. The above input data is divided into input data blocks of 256 bytes, as shown in Figure 4 As shown on the far right, each input data block is arranged in a Z-shape, and the input data blocks are continuous in the direction of channel C, thereby obtaining the input data block arrangement corresponding to the target input feature map in the memory.

[0105] In the above-mentioned embodiment, the target input feature map is arranged in the input data block corresponding to the memory based on the data throughput of the general matrix processing engine in each clock cycle, wherein the internal data of each input data block in the input data block arrangement is arranged in a Z-shape, and the input data blocks are continuous in the channel direction, which can give full play to the technical advantages of the high-bandwidth data bus in practical applications, achieve regular memory access, improve data access efficiency, and give full play to the computing power of the general matrix computing engine.

[0106] In some exemplary embodiments, in order to better implement regular memory access, improve the accuracy of subsequent acquisition of input data blocks, and maximize the computational efficiency of the general matrix processing engine, when implementing the data throughput of the general matrix processing engine in each clock cycle to divide the input data into multiple input data blocks, it is possible to first determine whether the size of the input data is an integer multiple of the size indicated by the data throughput (i.e., the bandwidth size). If the size of the input data is not an integer multiple of the bandwidth size, the input data can be padded to obtain padded input data, and the size of the padded input data is an integer multiple of the bandwidth size; then, based on the data throughput in each clock cycle, the padded input data is divided into multiple input data blocks to obtain the input data arrangement corresponding to the target input feature map in the memory.

[0107] Taking the aforementioned memory bandwidth of 16x16 bytes as an example, if the width C of the input data is not an integer multiple of 16, data padding processing may be performed on the input data in the width direction. Similarly, if the height W*H of the input data is not an integer multiple of 16, data padding processing may be performed on the input data in the height direction. Exemplarily, the data padding processing may use a preset value for padding, for example, the preset value may be 0.

[0108] In step S23, the target input data block is cached.

[0109] In step S25, matrix multiplication processing is performed based on the cached target input data block using the general matrix processing engine to obtain a data processing result.

[0110] In an exemplary embodiment, Figure 5 As shown, when implementing the cache-based target input data block and using the general matrix processing engine to perform matrix multiplication processing to obtain a data processing result, step S25 may include:

[0111] In step S51, matrix conversion is performed on the cached target input data block according to the convolution parameters of the current convolution layer.

[0112] The convolution parameters of the current convolution layer may include the moving step size of the sliding window and the window size of the sliding window.

[0113] Specifically, the matrix conversion algorithm of Img2Col may be used to perform matrix conversion on the cached target input data block.

[0114] In step S53, when it is determined that the matrix conversion of the target input data block is completed, the feature block matrix obtained by the matrix conversion is output to the general matrix processing engine, so that the general matrix processing engine performs a matrix multiplication operation.

[0115] Specifically, when it is determined that the matrix conversion of the target input data block is completed, a data valid signal can be generated. The data valid signal includes the characteristic block matrix obtained by the matrix conversion of the target input data block and the size of the target input data block. Therefore, when the general matrix processing engine receives the data valid signal, it can extract the characteristic block matrix and the data size from the data valid signal, and then call the matrix processing unit in a non-idle state to perform calculations. In this way, external storage can be bypassed, and the characteristic block matrix obtained by the matrix conversion can be directly used by the general matrix processing engine, reducing the number of large-scale data reading and writing times, and further improving the data processing efficiency of the electronic device.

[0116] In some exemplary embodiments, Figure 6 As shown, the aforementioned step S21, in response to the data processing instruction for the target input feature map, when obtaining the target input data block from the input data block arrangement corresponding to the target input feature map in the memory, may include:

[0117] In step S61, in response to the data processing instruction of the target input feature map, the idle state enters the waiting state.

[0118] In step S63, a read data request is sent to the memory in the waiting state, and the waiting state enters the ready state; the read data request is used to request to obtain the input data block in the input data block arrangement.

[0119] In step S65, upon receiving the target input data block returned by the memory based on the read data request, the matrix conversion working state is entered.

[0120] The target input data block includes a group of input data blocks arranged in a row direction.

[0121] Continue to see Figure 6 Accordingly, when performing matrix conversion on the cached target input data block according to the convolution parameters of the current convolution layer, the aforementioned step S51 may include:

[0122] In step S67, address offset information is determined according to the feature map size parameter of the target input feature map, the convolution parameter and the block size of the input data block.

[0123] The feature map size parameters include the width, height, and number of channels. Convolution parameters include the width and height of the convolution kernel (i.e., the window size of the sliding window) and the kernel's step size in the row and column directions (i.e., the moving step size of the sliding window). The address offset information addr_offset can be calculated using the following formula: addr_offset = ((cumulative step size in row direction + kernel row index + (cumulative step size in column direction + kernel column index) x input feature height) / feature block size) x feature block size + ((cumulative step size in row direction + kernel row index + (cumulative step size in column direction + kernel column index) x input feature height) / feature block size) x (feature block size ^ 2) x ceil (number of feature map channels / feature block size) + channel direction index x (feature block size ^ 2). The ceil(x) function returns the smallest integer greater than or equal to x.

[0124] In step S69, the current input data block to be processed is obtained from the target input data block in the cache according to the address offset information.

[0125] In step S611, the current input data block to be processed is expanded according to the convolution parameters to obtain an expanded block matrix, and the expanded block matrix is transposed to obtain the feature block matrix.

[0126] In some exemplary embodiments, in order to improve the data reuse rate and thus improve the data processing efficiency, when the current input data block to be processed is matrix expanded according to the convolution parameters, the corresponding overlapping number of adjacent sliding windows in the input data block to be processed can be determined according to the moving step size of the sliding window in the convolution parameters; then the overlapping data is temporarily stored in the channel dimension as the data to be reused, and then the data to be converted is obtained from the data corresponding to the sliding window in the input data block to be processed, and the data to be converted does not include the overlapping data; then, the matrix is expanded based on the data to be reused and the data to be converted, and then the expanded block matrix is transposed to obtain the feature block matrix corresponding to the current input data block to be processed.

[0127] The above implementation increases the data reuse rate and is used to temporarily store intermediate data by giving priority to storing data on the same channel. The converted result is in units of data blocks.

[0128] In step S613, when the current input data block to be processed is not the last input data block in the target input data block, the process returns to execute the aforementioned steps S69 and 611 until the current input data block to be processed is the last input data block in the target input data block, and it is determined that the matrix conversion of the target input data block is completed, and the process enters the completion state from the matrix conversion working state.

[0129] In some exemplary embodiments, see Figure 6 After step S613, the method may further include:

[0130] In step S615 , if there are input data blocks that have not been matrix-converted in the arrangement of the input data blocks in the column direction, the process returns to the waiting state from the completion state.

[0131] In step S617, the step of sending a read data request to the memory in the waiting state is executed until the input data blocks are arranged in the column direction and no input data blocks that have not been matrix-converted exist, and the idle state is entered from the completion state.

[0132] It can be understood that when the input data blocks are arranged in the column direction and there are no input data blocks that have not been matrix converted, it indicates that the matrix conversion of the target input feature map has been completed, and the idle state can be entered from the completed state. At this time, an end signal can be generated and sent to the general matrix processing engine, so that the general matrix processing engine outputs the final convolution result when the calculation is completed.

[0133] The disclosed embodiment converts the target input feature map originally used for convolution calculations into an input that can directly perform general matrix multiplication calculations, simplifying subsequent convolution operations and realizing parallel img2col conversion. The converted results are still input to the general matrix processing engine in blocks, allowing the general matrix processing engine to perform highly parallel calculations. In addition, the converted feature block matrix can be directly used by the general matrix engine, bypassing external storage or L0 cache, reducing the number of large-scale data reads and writes, greatly improving access efficiency and processing efficiency, indirectly leveraging the computing power of the general matrix processing engine, and helping to improve the efficiency of convolution calculations, thereby greatly improving the data processing efficiency of electronic devices based on deep learning networks.

[0134] Figure 7 FIG. 1 is a block diagram of a data processing device according to an exemplary embodiment. Figure 7 , the data processing device 700 includes:

[0135] The input data block acquisition unit 710 is configured to execute a data processing instruction in response to a target input feature map, and acquire a target input data block from an input data block arrangement corresponding to the target input feature map in a memory; the input data block arrangement is obtained by dividing the target input feature map into a plurality of input data blocks based on the data throughput of the general matrix processing engine in each clock cycle and storing the blocks in the memory in a preset arrangement;

[0136] An input data block caching unit 720 is configured to cache the target input data block;

[0137] The data processing module 730 is configured to execute the target input data block based on the cache, and use the general matrix processing engine to perform matrix multiplication processing to obtain a data processing result.

[0138] In an exemplary embodiment, the data processing module 730 includes:

[0139] a matrix conversion unit, configured to perform matrix conversion on the cached target input data block according to the convolution parameters of the current convolution layer;

[0140] The feature block matrix output unit is configured to output the feature block matrix obtained by the matrix transformation to the general matrix processing engine when it is determined that the matrix transformation of the target input data block is completed, so that the general matrix processing engine performs a matrix multiplication operation.

[0141] In an exemplary embodiment, the input data block acquiring unit 710 includes:

[0142] A first state transition unit is configured to execute a data processing instruction in response to the target input feature map and enter a waiting state from an idle state;

[0143] A second state conversion unit is configured to execute sending a read data request to the memory in the waiting state, and enter a ready state from the waiting state; the read data request is used to request to obtain an input data block in the input data block arrangement;

[0144] The third state conversion unit is configured to enter a matrix conversion working state when receiving a target input data block returned by the memory based on the read data request; the target input data block includes a group of input data blocks arranged in a row direction.

[0145] In an exemplary embodiment, the matrix conversion unit includes:

[0146] An address offset determining unit is configured to determine address offset information according to a feature map size parameter of the target input feature map, the convolution parameter, and a block size of the input data block; the feature map size parameter includes a feature map width, a feature map height, and a number of feature map channels;

[0147] a cache acquisition unit, configured to acquire a current input data block to be processed from the target input data block in the cache according to the address offset information;

[0148] a matrix expansion unit configured to perform matrix expansion on the current input data block to be processed according to the convolution parameters to obtain an expanded block matrix, and transpose the expanded block matrix to obtain the feature block matrix;

[0149] The fourth state transition unit is configured to execute, when the current input data block to be processed is not the last input data block in the target input data block, the steps of obtaining the current input data block to be processed from the cached target input data block according to the address offset information and the matrix expansion, until the current input data block to be processed is the last input data block in the target input data block, determine that the matrix conversion of the target input data block is completed, and enter the completion state from the matrix conversion working state.

[0150] In an exemplary embodiment, the matrix conversion unit further includes:

[0151] a fifth state transition unit configured to execute, if the input data blocks arranged in the column direction still have input data blocks that have not been matrix-transformed, to enter the waiting state again from the completion state;

[0152] The sixth state conversion unit is configured to execute the step of sending a read data request to the memory in the waiting state until the input data blocks are arranged in the column direction and there are no input data blocks that have not been matrix converted, and enter the idle state from the matrix conversion working state.

[0153] In an exemplary embodiment, the matrix expansion unit includes:

[0154] an overlapping data determining unit, configured to determine overlapping data corresponding to adjacent sliding windows in the input data block to be processed according to a moving step length of the sliding window in the convolution parameter;

[0155] an overlapping data temporary storage unit, configured to temporarily store the overlapping data according to a channel dimension as data to be multiplexed;

[0156] a data-to-be-converted acquiring unit configured to acquire data to be converted from the data corresponding to the input data block to be processed from a sliding window, wherein the data to be converted does not include the overlapping data;

[0157] The matrix expansion subunit is configured to perform matrix expansion based on the data to be multiplexed and the data to be converted.

[0158] In an exemplary embodiment, the apparatus 700 further includes:

[0159] An input conversion unit is configured to convert the target input feature map into input data in the memory; the height of the input data is the length multiplied by the width of the target input feature map, and the width of the input data is the number of channels of the target input feature map;

[0160] A throughput acquisition unit is configured to acquire the data throughput of the general matrix processing engine in each clock cycle;

[0161] The data block division unit is configured to divide the input data into multiple input data blocks based on the data throughput of the general matrix processing engine in each clock cycle, the data volume of each input data block is the data throughput, the internal data of each input data block is arranged in a Z-shape, and the input data blocks are continuous in the channel direction.

[0162] In an exemplary embodiment, the data block division unit includes:

[0163] a data padding unit configured to perform data padding on the input data when the size of the input data is not an integer multiple of the size indicated by the data throughput, so as to obtain padded input data; wherein the size of the padded input data is an integer multiple of the size indicated by the data throughput;

[0164] The data block division subunit is configured to divide the padded input data into the multiple input data blocks based on the data throughput of the general matrix processing engine in each clock cycle.

[0165] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0166] In some exemplary embodiments, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the data processing method provided in any of the above embodiments.

[0167] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Figure 8 is a block diagram of an electronic device for data processing according to an exemplary embodiment. Figure 8 As shown, the server 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 810 (the processor 810 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 830 for storing data, and one or more storage media 820 (such as one or more mass storage devices) for storing application programs 823 or data 822. Among them, the memory 830 and the storage medium 820 can be temporary storage or permanent storage. The program stored in the storage medium 820 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the central processing unit 810 can be configured to communicate with the storage medium 820 to execute a series of instruction operations in the storage medium 820 on the server 800. The server 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input and output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0168] The input / output interface 840 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communication provider of the server 800. In one embodiment, the input / output interface 840 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the input / output interface 840 can be a radio frequency (RF) module for wirelessly communicating with the Internet.

[0169] It can be understood by those skilled in the art that Figure 8 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 8 More or fewer components than shown, or with Figure 8 Different configurations shown.

[0170] In some exemplary embodiments, a computer-readable storage medium including instructions is further provided, such as a memory 830 including instructions. The instructions may be executed by the processor 810 of the apparatus 800 to perform the above-described data processing method. Alternatively, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0171] In some exemplary embodiments, a computer program product is further provided, including a computer program, which implements the data processing method provided in any of the above embodiments when executed by a processor.

[0172] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0173] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: In response to a data processing instruction for a target input feature map, obtaining a target input data block from an input data block arrangement corresponding to the target input feature map in a memory; The input data block arrangement is based on the data throughput of the general matrix processing engine in each clock cycle, and the target input feature map is divided into multiple input data blocks and stored in the memory according to a preset arrangement method; caching the target input data block; Determining address offset information according to a feature map size parameter of the target input feature map, a convolution parameter of a current convolutional layer, and a block size of an input data block; Obtaining a current input data block to be processed from the cached target input data block according to the address offset information; Performing matrix expansion on the current input data block to be processed according to the convolution parameters to obtain an expanded block matrix, and transposing the expanded block matrix to obtain a feature block matrix; If the current input data block to be processed is not the last input data block in the target input data block, performing the steps of obtaining the current input data block to be processed from the cached target input data block according to the address offset information and performing the matrix expansion until the current input data block to be processed is the last input data block in the target input data block, and determining that the matrix transformation of the target input data block is completed; The characteristic block matrix obtained by the matrix conversion is output to a general matrix processing engine, so that the general matrix processing engine performs matrix multiplication processing to obtain a data processing result.

2. The data processing method according to claim 1, wherein: The step of obtaining, in response to a data processing instruction for a target input feature map, a target input data block from an input data block arrangement corresponding to the target input feature map in a memory includes: In response to a data processing instruction for the target input feature map, entering a waiting state from an idle state; Sending a read data request to the memory in the waiting state, and entering the ready state from the waiting state; the read data request is used to request to obtain the input data block in the input data block arrangement; When a target input data block returned by the memory based on the read data request is received, a matrix conversion working state is entered; the target input data block includes a group of input data blocks arranged in a row direction.

3. The data processing method according to claim 2, characterized in that: The method further comprises: After determining that the matrix conversion of the target input data block is completed, the matrix conversion working state enters the completion state.

4. The data processing method according to claim 3, characterized in that: The method further comprises: If there are still input data blocks that have not been matrix-converted when the input data blocks are arranged in the column direction, then the waiting state is entered again from the completion state; The step of sending a read data request to the memory in the waiting state is executed until the input data blocks are arranged in the column direction and no input data blocks that have not been matrix-converted exist, and the idle state is entered from the completion state.

5. The data processing method according to claim 1, wherein: The performing matrix expansion on the current input data block to be processed according to the convolution parameters includes: Determining overlapping data corresponding to adjacent sliding windows in the input data block to be processed according to a moving step size of the sliding window in the convolution parameter; Temporarily storing the overlapping data according to the channel dimension as data to be multiplexed; Acquire data to be converted from the data corresponding to the input data block to be processed in the sliding window, wherein the data to be converted does not include the overlapping data; Matrix expansion is performed based on the data to be multiplexed and the data to be converted.

6. The data processing method according to any one of claims 1 to 5, characterized in that: Before obtaining the target input data block from the input data block arrangement corresponding to the target input feature map in the memory in response to the data processing instruction for the target input feature map, the method further includes: Convert the target input feature map into input data in the memory; the height of the input data is the length multiplied by the width of the target input feature map, and the width of the input data is the number of channels of the target input feature map; Obtaining the data throughput of the general matrix processing engine in each clock cycle; Based on the data throughput of the general matrix processing engine in each clock cycle, the input data is divided into multiple input data blocks, the data amount of each input data block is the data throughput, the internal data of each input data block is arranged in a Z-shape, and the input data blocks are continuous in the channel direction.

7. The data processing method according to claim 6, characterized in that: Based on the data throughput of the general matrix processing engine in each clock cycle, the input data is divided into a plurality of input data blocks, including: When the size of the input data is not an integer multiple of the size indicated by the data throughput, padding the input data to obtain padded input data; the size of the padded input data is an integer multiple of the size indicated by the data throughput; The padded input data is divided into the plurality of input data blocks based on a data throughput of the general matrix processing engine in each clock cycle.

8. A data processing device, characterized in that: include: an input data block acquisition unit, configured to execute a data processing instruction in response to a target input feature map, and acquire a target input data block from an input data block arrangement corresponding to the target input feature map in a memory; The input data block arrangement is based on the data throughput of the general matrix processing engine in each clock cycle, and the target input feature map is divided into multiple input data blocks and stored in the memory according to a preset arrangement method; An input data block caching unit, configured to cache the target input data block; The data processing module is configured to determine address offset information based on a feature map size parameter of the target input feature map, a convolution parameter of a current convolution layer, and a block size of an input data block; obtain a current input data block to be processed from the cached target input data block based on the address offset information; perform matrix expansion on the current input data block to be processed based on the convolution parameter to obtain an expanded block matrix; and transpose the expanded block matrix to obtain a feature block matrix. In the case that the current input data block to be processed is not the last input data block in the target input data block, the steps of obtaining the current input data block to be processed from the cached target input data block and the matrix expansion are performed according to the address offset information until the current input data block to be processed is the last input data block in the target input data block, and it is determined that the matrix transformation of the target input data block is completed; the feature block matrix obtained by the matrix transformation is output to the general matrix processing engine, so that the general matrix processing engine performs matrix multiplication processing to obtain a data processing result.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is caused to execute the data processing method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images