Data processing method and apparatus, computer device, and storage medium

Abstract

A data processing method and apparatus, a computer device, and a storage medium, relating to the technical field of computers. The method comprises: acquiring target data and the number of columns of a systolic array, the target data comprising input data and at least one convolution kernel, and a calculation unit of each column in the systolic array being used for convolving the input data and one convolution kernel (201); when the number of convolution kernels is not greater than the number of columns of the systolic array, determining an expansion coefficient on the basis of a ratio of the number of columns of the systolic array to the number of convolution kernels (202); on the basis of the expansion coefficient, expanding the convolution kernels, so that the quantity, size and convolution stride of the expanded convolution kernels are increased (203); and on the calculation units of the systolic array, convolving the input data and the expanded convolution kernels to obtain output data (204). The systolic array can accelerate matrix multiplication and convolution operation in a model, and the present application can improve the utilization rate of the systolic array.

Description

Data processing method, device, computer equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed on August 25, 2023, with application number 202311076186.4 and invention name “Data processing method, device, computer equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Art

[0003] A systolic array is an array-structured computing device in a processor that can be used to implement multiply-accumulate operations. Since convolution operations involve multiply-accumulate operations, systolic arrays can be used to perform convolution in convolutional neural networks.

[0004] Among them, the computing units in each column of the systolic array can be used to perform a convolution operation based on a convolution kernel.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a data processing method, apparatus, computer device, and storage medium that can improve the utilization rate of a systolic array and facilitate full utilization of the computing power of the systolic array. The technical solution is as follows.

[0007] In one aspect, a data processing method is provided, which is applied to a computer device, and the method comprises:

[0008] Obtaining target data and a column number of a systolic array, wherein the target data includes input data and at least one convolution kernel, the systolic array being a computing device in a processor, the computing device including a plurality of computing units, and the computing units in each column of the systolic array being configured to convolve the input data with a convolution kernel;

[0009] determining an expansion factor based on a ratio between the number of columns of the systolic array and the number of convolution kernels when the number of convolution kernels is not greater than the number of columns of the systolic array;

[0010] Expanding the convolution kernel based on the expansion coefficient so as to increase the number, size and convolution step of the expanded convolution kernel;

[0011] The input data and the expanded convolution kernel are convolved on the computing unit of the systolic array to obtain output data.

[0012] In another aspect, a data processing device is provided, comprising:

[0013] a data acquisition module, configured to acquire target data and a column number of a systolic array, wherein the target data includes input data and at least one convolution kernel, the systolic array being a computing device in a processor, the computing device including a plurality of computing units, and the computing unit in each column of the systolic array being configured to convolve the input data with a convolution kernel;

[0014] a coefficient determination module, configured to determine an expansion coefficient based on a ratio between the number of columns of the systolic array and the number of convolution kernels when the number of convolution kernels is not greater than the number of columns of the systolic array;

[0015] An expansion module, configured to expand the convolution kernel based on the expansion coefficient, so as to increase the number, size and convolution step length of the expanded convolution kernel;

[0016] A data processing module is used to convolve the input data and the expanded convolution kernel on the computing unit of the systolic array to obtain output data.

[0017] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the data processing method described in the above aspects.

[0018] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the data processing method described in the above aspects.

[0019] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is loaded and executed by a processor to implement the operations performed by the data processing method as described in the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] FIG1 is a system architecture diagram of a data processing method provided in an embodiment of the present application;

[0022] FIG2 is a flow chart of a data processing method provided in an embodiment of the present application;

[0023] FIG3 is a flow chart of another data processing method provided in an embodiment of the present application;

[0024] FIG4 is a schematic diagram of a bias parameter expansion method provided in an embodiment of the present application;

[0025] FIG5 is a schematic diagram of a convolution kernel expansion method provided in an embodiment of the present application;

[0026] FIG6 is a schematic diagram of another data processing method provided in an embodiment of the present application;

[0027] FIG7 is a flowchart of another data processing method provided in an embodiment of the present application;

[0028] FIG8 is a flowchart of another data processing method provided in an embodiment of the present application;

[0029] FIG9 is a flowchart of another data processing method provided in an embodiment of the present application;

[0030] FIG10 is a flowchart of another data processing method provided in an embodiment of the present application;

[0031] FIG11 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0032] FIG12 is a schematic structural diagram of a terminal provided in an embodiment of the present application;

[0033] FIG13 is a schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0035] It is understood that the terms "first," "second," and the like used herein may be used to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are used solely to distinguish one concept from another. For example, a first dimension may be referred to as a second dimension, and similarly, a second dimension may be referred to as a first dimension without departing from the scope of this application.

[0036] Here, at least one refers to one or more than one. For example, the at least one convolution kernel can be one convolution kernel, two convolution kernels, three convolution kernels, or any other integer greater than or equal to one. Multiple refers to two or more than two. For example, the multiple convolution kernels can be two convolution kernels, three convolution kernels, or any other integer greater than or equal to two. Each refers to each of at least one. For example, each convolution kernel refers to each convolution kernel in the multiple convolution kernels. If the multiple convolution kernels are three convolution kernels, then each convolution kernel refers to each of the three convolution kernels.

[0037] It is understandable that in the implementation methods of this application, when data related to user information (such as target data) is involved, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0038] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0039] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0040] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D (three-dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, and smart transportation. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0041] The systolic array (Systolic Array Architecture) is a pipelined, high-speed computing structure with a large throughput, which is composed of multiple processing units and multiple latches arranged according to certain interconnection rules. During operation, data moves synchronously in their respective directions between the processing units of the array structure. The systolic array can be integrated into an AI accelerator. The systolic array can parallelly dequeue matrix operations such as convolution operations and matrix multiplication in the AI ​​large model, thereby improving the operation speed and reducing power consumption, thereby accelerating the training process and reasoning process of the AI ​​large model, and thus helping to improve the training speed and computing efficiency of the AI ​​large model. In addition, the data processing method provided in the present application can effectively utilize the computing resources of the systolic array, and can further improve the training speed and computing efficiency of the AI ​​large model.

[0042] The following describes the data processing method provided in the embodiment of the present application based on artificial intelligence technology and computer vision technology.

[0043] The data processing method provided in the embodiment of the present application can be used in a computer device. Optionally, the computer device is a server or a terminal. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart terminal, etc., but is not limited to this.

[0044] In one possible implementation, the computer program involved in the embodiments of the present application can be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network can constitute a blockchain system.

[0045] FIG1 is a system architecture diagram of a data processing method provided in an embodiment of the present application, wherein the execution subject of the embodiment of the present application is a computer device, for example, the computer device is a server or a terminal. As shown in FIG1 , a matrix operation acceleration engine, a data handling engine, a secondary memory and a tertiary memory are integrated on the processor of the computer device, for example, the processor can be an artificial intelligence processor (AI Core). The matrix operation acceleration engine is used to implement matrix operations, and the data handling engine is used to carry out data handling between the primary memory, the secondary memory, the tertiary memory and the ACC (Accumulator) memory, and the secondary memory and the tertiary memory are used to store data.

[0046] The matrix acceleration engine includes a primary memory, a weight matrix memory, a bias parameter memory, a systolic array, an ACC memory, and a programmable controller. The primary memory is used to store data transferred from the secondary or tertiary memory by the data transfer engine, the weight matrix memory is used to store the weight matrix, the bias parameter memory is used to store the bias parameters, the systolic array is used to perform matrix operations, and in this embodiment of the application, the systolic array is used to perform convolution operations. The ACC memory is used to store the output data of the systolic array.

[0047] The embodiment of the present application can be applied to the scenario of performing convolution operation on the data in the convolutional neural network, where the input data and convolution kernel in the convolutional neural network are stored in the secondary memory or the tertiary memory, and the data transfer engine transfers the input data and convolution kernel from the secondary memory or the tertiary memory to the primary memory. In addition, in order to improve the utilization rate of the systolic array, the convolution kernel is also expanded (the weight matrix and bias parameters are expanded). The subsequent systolic array obtains the input data and the expanded convolution kernel from the primary memory, wherein the weight matrix in the expanded convolution kernel can be first transferred from the primary memory to the weight matrix memory, and then the systolic array obtains the expanded weight matrix from the weight matrix memory, and the bias parameters in the expanded convolution kernel can be first transferred from the primary memory to the bias parameter memory, and then the systolic array obtains the expanded bias parameters from the bias parameter memory. After obtaining the input data and the expanded convolution kernel, the systolic array convolves the input data and the expanded convolution kernel to obtain output data, and stores the output data in the ACC memory. The data transfer engine can transfer the output data in the ACC memory to the secondary memory or tertiary memory as input data for the next convolution operation.

[0048] The data processing method provided in the embodiments of the present application can be applied to any scenario where convolution operations need to be performed on data. For example, it is applied to the scenario of processing data in a convolutional neural network. Among them, convolutional neural networks are the earliest, most developed, and most widely used type of deep learning algorithms in the field of deep learning. They are widely used in various digital processing fields such as image processing, video processing, and speech processing. A notable feature of deep learning is the large amount of calculation and the large number of parameters. Traditional processors are difficult to process in real time. For this reason, AI (Artificial Intelligence) processors have been developed in related fields. A core feature of the AI ​​processor is the design of the systolic array. The computing units in the systolic array are used to implement convolution operations.

[0049] However, systolic arrays are more suitable for computationally intensive convolutional layers and matrix multiplications. However, neural networks, especially those used for superclassification and image generation, inevitably have convolutional layers with a small number of kernels. Directly utilizing the systolic array in an AI processor results in low computational utilization, becoming a bottleneck for algorithm acceleration. For example, if the systolic array is 2×64×64 in size, with 64 columns, the computational utilization of the systolic array will inevitably be less than 100% if the number of kernels in a convolutional neural network convolutional layer is less than 64. For example, if the number of kernels in a convolutional layer is 3, the maximum computational utilization of the systolic array is only 3 / 64.

[0050] Therefore, the data processing method provided in the embodiment of the present application can be adopted. If the number of convolution kernels is small, the convolution kernels are expanded before using the systolic array for convolution to increase the number of convolution kernels. The computational complexity of the convolution kernels in the height or width dimension is transferred to the dimension of the number of convolution kernels, and the systolic array is used to perform convolution operations based on the expanded convolution kernels, thereby effectively improving the computing power utilization of the systolic array and accelerating the data processing speed.

[0051] The following examples illustrate the application scenarios of the embodiments of the present application:

[0052] 1. Scenario for extracting image features from image data: The input data is image data, and the convolutional neural network is the feature extraction network.

[0053] In this scenario, after acquiring the image data, it is necessary to acquire the image features of the image data through the feature extraction network. Therefore, the image data and the convolution kernel are stored in the secondary memory or the tertiary memory, and the data transfer engine transfers the image data and the convolution kernel from the secondary memory or the tertiary memory to the primary memory. In addition, the convolution kernel is expanded (the weight matrix and bias parameters are expanded). Subsequently, the systolic array acquires the image data and the expanded convolution kernel from the primary memory, wherein the weight matrix in the expanded convolution kernel can be first transferred from the primary memory to the weight matrix memory, and then the systolic array acquires the expanded weight matrix from the weight matrix memory, and the bias parameters in the expanded convolution kernel can be first transferred from the primary memory to the bias parameter memory, and then the systolic array acquires the expanded bias parameters from the bias parameter memory. After acquiring the image data and the expanded convolution kernel, the systolic array convolves the image data and the expanded convolution kernel to obtain output data, and stores the output data in the ACC memory. The data transfer engine can transfer the output data from the ACC memory to the secondary or tertiary memory as input data for the next convolution operation. The output data obtained after multiple convolution operations is the image features of the image data, which can be used to describe the image data.

[0054] 2. Scenario of generating matching image data based on text data: The input data is text data, which is used to describe the style of the image to be generated, and the convolutional neural network is the image generation network.

[0055] In this scenario, after obtaining text data, it is necessary to generate image data that matches the text data through an image generation network. Therefore, the text data and the convolution kernel are stored in a secondary memory or a tertiary memory, and the data transfer engine transfers the text data and the convolution kernel from the secondary memory or the tertiary memory to the primary memory. In addition, the convolution kernel is expanded (the weight matrix and bias parameters are expanded). The subsequent systolic array obtains the text data and the expanded convolution kernel from the primary memory, wherein the weight matrix in the expanded convolution kernel can be first transferred from the primary memory to the weight matrix memory, and then the systolic array obtains the expanded weight matrix from the weight matrix memory, and the bias parameters in the expanded convolution kernel can be first transferred from the primary memory to the bias parameter memory, and then the systolic array obtains the expanded bias parameters from the bias parameter memory. After obtaining the text data and the expanded convolution kernel, the systolic array convolves the text data and the expanded convolution kernel to obtain output data, and stores the output data in the ACC memory. The data transfer engine can transfer the output data from the ACC memory to the secondary or tertiary memory as input data for the next convolution operation. The output data obtained after multiple convolution operations is the image data matched with the text data.

[0056] The above two application scenarios are only examples. The embodiments of the present application can also be applied to other scenarios that require convolution operations. The embodiments of the present application do not limit the application scenarios.

[0057] FIG2 is a flow chart of a data processing method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Referring to FIG2 , the method includes the following steps.

[0058] 201. A computer device obtains target data and a column number of a systolic array, wherein the target data includes input data and at least one convolution kernel. The systolic array is a computing device in a processor, and the computing device includes a plurality of computing units. The computing unit in each column of the systolic array is used to convolve the input data with a convolution kernel.

[0059] The target data includes input data and at least one convolution kernel, wherein the target data is data that needs to be processed by the computer device, and the at least one convolution kernel is used to convolve the input data. The input data is the data that needs to be convolved. For example, the input data is the data output by a convolution layer of a convolutional neural network, which is used to be input to the next convolution layer for convolution operation. Then the convolution kernel is the convolution kernel of the next convolution layer. For example, the input data includes one or more different types such as image data and text data. The number of convolution kernels in the target data can be one or more. Optionally, when the number of convolution kernels is multiple, the sizes of the multiple convolution kernels are equal.

[0060] A systolic array is a computing device within a computer processor. It is hardware integrated into the processor and consists of multiple computing units. These units form an array structure, hence the name systolic array. A systolic array includes multiple columns of computing units, each of which is used to convolve input data with a convolution kernel.

[0061] Optionally, the input data exists in matrix form, represented by [Hi, Wi, Ci], where Hi represents the height of the input data, Wi represents the width of the input data, and Ci represents the number of channels of the input data. The convolution kernel exists in matrix form, represented by [Kh, Kw, Ci, Co], where Kh represents the height of the convolution kernel, Kw represents the width of the convolution kernel, Ci represents the number of input channels of the convolution kernel, which is equal to the number of channels of the input data, and Co represents the number of convolution kernels, which is also the number of output channels of the convolution kernel, which is equal to the number of channels of the output data.

[0062] 202. When the number of convolution kernels is not greater than the number of columns of the systolic array, the computer device determines an expansion factor based on a ratio between the number of columns of the systolic array and the number of convolution kernels.

[0063] The computer device compares the number of convolution kernels in the target data with the number of columns in the systolic array. If the number of convolution kernels is not greater than the number of columns in the systolic array, this indicates that the utilization of the systolic array is low due to the small number of convolution kernels. To improve the utilization of the systolic array, the convolution kernels may be expanded. Therefore, the computer device determines an expansion factor based on the ratio between the number of columns in the systolic array and the number of convolution kernels. The expansion factor refers to a multiple by which the convolution kernels are expanded. Optionally, the expansion factor is an integer greater than 1.

[0064] In one possible implementation, the computer device may multiply the convolution kernels. The computer device then determines the difference between the number of convolution kernels and half the number of columns of the systolic array. If the number of convolution kernels is not greater than the number of columns of the systolic array but greater than half the number of columns, then after the multiplication of the convolution kernels, the number of convolution kernels will be greater than the number of columns of the systolic array, resulting in the systolic array needing to perform convolutions on the convolution kernels in batches. To ensure that the systolic array does not need to perform convolutions on a convolution kernel in batches, it is necessary to ensure that the number of convolution kernels after the expansion is no greater than the number of columns of the systolic array. Therefore, when the number of convolution kernels is no greater than half the number of columns of the systolic array, the computer device determines an expansion factor based on the ratio between the number of columns of the systolic array and the number of convolution kernels, to ensure that the determined expansion factor is no greater than the ratio between the number of columns and the number of convolution kernels. When the number of convolution kernels is not greater than the number of columns in the systolic array, but greater than half of the number of columns in the systolic array, the convolution kernels are no longer expanded.

[0065] 203. The computer device expands the convolution kernel based on the expansion coefficient so as to increase the number, size and convolution step of the expanded convolution kernel.

[0066] After determining the expansion coefficient, the computer device expands the convolution kernel based on the expansion coefficient, so that the number, size, and convolution step size of the expanded convolution kernel are increased. Optionally, a ratio between the number of convolution kernels after the expansion and the number of convolution kernels before the expansion is equal to the expansion coefficient, and a ratio between the convolution step size of the convolution kernel after the expansion and the convolution step size of the convolution kernel before the expansion is equal to the expansion coefficient.

[0067] The size of the convolution kernel refers to its size in the height or width dimension. The convolution step size refers to the length of each movement of the convolution kernel during convolution. The size of the convolution step size affects the accuracy of the convolution operation and the size of the output data. The larger the convolution step size, the faster the convolution kernel slides and the smaller the output data size.

[0068] Increasing the kernel size and convolution stride reduces the amount of computation required in the kernel's height or width dimensions. Increasing the number of kernels increases the amount of computation required in the output channel dimension. This is equivalent to expanding the kernel to convert the computation required in the height or width dimensions to the computation required in the output channel dimension. Furthermore, increasing the number of kernels fully utilizes the computational power of the systolic array.

[0069] 204. The computer device convolves the input data and the expanded convolution kernel on the computing unit of the systolic array to obtain output data.

[0070] After obtaining the expanded convolution kernel, the computer device convolves the input data with the expanded convolution kernel on the computational units of the systolic array to obtain output data. Multiple columns of computational units in the systolic array perform convolutions on the input data and the convolution kernel in parallel. As the number of expanded convolution kernels increases, the number of columns of computational units in the systolic array used also increases, thereby improving the utilization of the systolic array.

[0071] In the method provided in an embodiment of the present application, if the number of convolution kernels is not greater than the number of columns of the systolic array, some computing units in the systolic array will not be used. In this case, an expansion factor is determined according to the ratio between the number of columns of the systolic array and the number of convolution kernels. Then, the original convolution kernel is expanded according to the expansion factor to increase the number of convolution kernels. At the same time, the size of the convolution kernel and the convolution step size are also increased to offset the impact of the increase in the number of convolution kernels. This is equivalent to converting the computational effort of a small number of convolution kernels at a smaller convolution step size into the computational effort of a larger number of convolution kernels at a larger convolution step size. Since the number of convolution kernels is increased, more computing units in the systolic array can be used, thereby improving the utilization rate of the systolic array and fully utilizing the computing power of the systolic array.

[0072] The embodiment shown in FIG2 is merely a brief description of the data processing method. The detailed process of determining the expansion coefficient and expanding the convolution kernel can be found in the embodiment shown in FIG3 below. FIG3 is a flow chart of another data processing method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Referring to FIG3, the method includes the following steps.

[0073] 301. A computer device obtains target data and a column number of a systolic array, wherein the target data includes input data and at least one convolution kernel. The systolic array is a computing device in a processor, and the computing device includes a plurality of computing units. The computing unit in each column of the systolic array is used to convolve the input data with a convolution kernel.

[0074] The step 301 is similar to the above step 201 and will not be described again here.

[0075] 302. When the number of convolution kernels is not greater than the number of columns of the systolic array, the computer device determines an expansion factor based on a ratio between the number of columns of the systolic array and the number of convolution kernels.

[0076] The computer device determines whether the number of convolution kernels is greater than the number of columns of the systolic array. If the number of convolution kernels is not greater than the number of columns of the systolic array, the convolution kernels need to be expanded to improve utilization of the systolic array. The computer device determines an expansion factor based on a ratio between the number of columns of the systolic array and the number of convolution kernels. The expansion factor refers to a multiple of the convolution kernel expansion.

[0077] In one possible implementation, the computer device determines the expansion factor based on a ratio between the number of columns of the systolic array and the number of convolution kernels when the number of convolution kernels is no greater than half the number of columns of the systolic array. If the number of convolution kernels is no greater than the number of columns of the systolic array but greater than half the number of columns of the systolic array, the convolution kernels are no longer expanded.

[0078] In another possible implementation, the computer device determines the expansion factor based on a ratio between the number of columns in the systolic array and the number of convolution kernels. This includes: the computer device determines the ratio between the number of columns in the systolic array and the number of convolution kernels, rounds down the ratio to obtain a reference value, and the computer device determines the expansion factor based on the reference value.

[0079] In the embodiments of the present application, the convolution kernel is multiplied based on the expansion factor. Therefore, to ensure that the number of convolution kernels after expansion is no greater than the number of columns in the systolic array, the expansion factor must be an integer no greater than the ratio of the number of columns in the systolic array to the number of convolution kernels. Therefore, after determining the ratio between the number of columns in the systolic array and the number of convolution kernels, the computer device first rounds down the ratio to obtain a reference value, and then determines the expansion factor based on the reference value. The expansion factor should be an integer no greater than the reference value. Determining the expansion factor based on the reference value can be divided into two cases. The detailed process is shown in the embodiments shown in Figures 7 and 8 below, and is not described here.

[0080] 303. The computer device determines a second convolution step length based on the expansion coefficient and the first convolution step length, where the first convolution step length is the convolution step length of the convolution kernel before expansion, and the second convolution step length is the convolution step length of the convolution kernel after expansion.

[0081] The computer device determines the convolution step size of the convolution kernel after expansion based on the convolution step size and the expansion coefficient of the convolution kernel before expansion.

[0082] In one possible implementation, the computer device multiplies the expansion coefficient and the first convolution step size to obtain a second convolution step size.

[0083] 304. The computer device determines a second size based on the expansion coefficient and the first size, where the first size is the size of the convolution kernel before expansion, and the second size is the size of the convolution kernel after expansion.

[0084] The computer device determines the size of the convolution kernel after expansion based on the size of the convolution kernel before expansion and the expansion factor. Optionally, the size of the convolution kernel includes height, width, and number of channels, and the second size, compared to the first size, changes only in height or width, while the number of channels remains unchanged.

[0085] In one possible implementation, the computer device determines the second size based on the expansion factor, the first size, the first convolution step size, and the second convolution step size, such that a ratio of the first product to the second product is equal to the expansion factor. The first product represents the product of the height and width of output data obtained by convolving the input data with a convolution kernel of the first size and the first convolution step size, and the second product represents the product of the height and width of the output data obtained by convolving the input data with a convolution kernel of the second size and the second convolution step size.

[0086] The output data includes three dimensions: height, width, and number of channels. Since the ratio of the number of convolution kernels after expansion to the number of convolution kernels before expansion is equal to the expansion factor, which is also the expansion multiple, and the number of convolution kernels is equal to the number of channels in the output data, then after the convolution kernel is expanded, the ratio between the number of channels in the output data and the original number of channels is also equal to the expansion factor. To ensure that the output data obtained after expansion is equivalent, the multiple by which the size of the output data is reduced needs to be equal to the expansion factor. Therefore, it is necessary to ensure that the ratio between the product of the height and width of the output data obtained before expansion and the product of the height and width of the output data obtained after expansion is equal to the expansion factor.

[0087] For example, if the output data before expansion is [Ho1, Wo1, Co1] and the output data after expansion is [Ho2, Wo2, Co2], and the expansion factor is 4, then it is necessary to ensure that Co2 / Co1 = 4 and (Ho2×Wo2) / (Ho1×Wo1) = 1 / 4. Ho represents the height of the output data, Wo represents the width of the output data, and Co represents the number of channels of the output data.

[0088] Optionally, the second convolution step size is equal to the product of the first convolution step size and the expansion factor, and the first size and the second size both represent a width of the convolution kernel. The computer device determines the second size based on the expansion factor, the first size, the first convolution step size, and the second convolution step size, such that a ratio of the first product to the second product is equal to the expansion factor, including: determining a product of k-1 and the first convolution step size, and adding the product to the first size to obtain the second size.

[0089] That is, the first size is referred to as the first width, the second size is referred to as the second width, and the computer device uses the following formula to determine the second width: New_Kw=Kw+(Co_Expansion-1)×Stride_W;

[0090] Wherein, New_Kw represents the second width, Kw represents the first width, Co_Expansion represents the expansion coefficient (k=Co_Expansion), and Stride_W represents the first convolution step size.

[0091] 305. The computer device copies at least one convolution kernel k-1 times, where k is equal to the expansion factor.

[0092] The computer device needs to expand the number of convolution kernels so that the ratio of the number of convolution kernels after expansion to the number of convolution kernels before expansion is equal to the expansion factor. The expansion factor is called k. The computer device then copies the at least one convolution kernel k-1 times. The number of convolution kernels after expansion is k times the number of convolution kernels before expansion. For example, if there are 16 convolution kernels and the expansion factor is 4, the computer device needs to copy the 16 convolution kernels 3 times, which means that there are 48 new convolution kernels. The number of convolution kernels after expansion is 64.

[0093] In one possible implementation, the convolution kernel also includes a bias parameter, which represents the weight of the convolution kernel among all the convolution kernels and is used to weight the output data obtained by convolution based on the convolution kernel, thereby obtaining weighted output data. The bias parameter of the convolution kernel is a parameter used to control the final output data reference value. The bias parameter can ensure that the final output data has a certain signal strength and will not always be zero or a small value. For each convolution kernel, the computer device copies the convolution kernel and the bias parameter of the convolution kernel k-1 times. That is, each convolution kernel has its own bias parameter. When the computer device copies each convolution kernel, it also copies the bias parameter corresponding to the convolution kernel. In this way, after expansion, it can also be guaranteed that each convolution kernel has a unique corresponding bias parameter.

[0094] Figure 4 is a schematic diagram of a bias parameter expansion method provided in an embodiment of the present application. As shown in Figure 4, there are 16 convolution kernels, and each convolution kernel corresponds to its own bias parameter. The bias parameters of the 16 convolution kernels are Co0, Co1, Co2, Co3...Co14 and Co15, respectively. The expansion factor is 4. While copying each convolution kernel 3 times, each bias parameter is also copied 3 times, and finally 64 convolution kernels and 64 bias parameters are obtained.

[0095] 306. The computer device pads each convolution kernel so that the size of the padded convolution kernel is equal to the second size.

[0096] After expanding the number of convolution kernels, the computer device pads each convolution kernel so that the size of the padded convolution kernel is equal to the second size. For example, if the first size of the convolution kernel is 3×3 and the second size is 3×6, the convolution kernel needs to be padded to obtain a convolution kernel of size 3×6.

[0097] In one possible implementation, the convolution kernel is represented by a weight matrix. The computer device fills the weight matrix of each convolution kernel with a target value according to a first size and a second size, so that the size of the filled weight matrix is ​​equal to the second size and the target value is equal to 0.

[0098] Optionally, the computer device fills the target value in the weight matrix of each convolution kernel according to the first size and the second size, including the following steps 3061-3063.

[0099] 3061. The computer device divides the copied multiple convolution kernels into k groups of convolution kernel sets in order, each group of convolution kernel sets includes multiple consecutive convolution kernels, and the number of convolution kernels in each group of convolution kernel sets is equal.

[0100] Taking the number of convolution kernels before expansion as the first number as an example, the number of convolution kernels after expansion is the second number, the second number divided by the first number is equal to k, k is the expansion coefficient, and the second number of convolution kernels is divided into k groups of convolution kernel sets in sequence, each group of convolution kernel sets includes a continuous first number of convolution kernels.

[0101] 3062. The computer device determines the difference between the second size and the first size as the third size.

[0102] For example, if the first size represents the width of the convolution kernel before expansion, and the second size represents the width of the convolution kernel after expansion, then the third size represents the required padding width. For example, if the first size represents the height of the convolution kernel before expansion, and the second size represents the height of the convolution kernel after expansion, then the third size represents the required padding height. Alternatively, if the first size includes the width and height of the convolution kernel before expansion, and the second size includes the width and height of the convolution kernel after expansion, then the third size includes the required padding width and height.

[0103] 3063. The computer device fills the target values ​​in the weight matrix of each convolution kernel in each group of convolution kernel sets, wherein the total size of the positions filled with the target values ​​in the weight matrix of each convolution kernel is equal to the third size, and the positions filled with the target values ​​in the weight matrices of the convolution kernels in the same group of convolution kernel sets are the same, and the positions filled with the target values ​​in the weight matrices of the convolution kernels in different groups of convolution kernel sets are not exactly the same.

[0104] Among them, the position of the target value filled in the weight matrix refers to: based on the weight matrix, the position of the filled target value relative to the weight matrix, such as the first column position on the left side of the weight matrix, the first column position on the right side, the second column position on the right side, etc.

[0105] Convolution kernels within the same set of convolution kernels are padded in the same way, while kernels within different sets of convolution kernels are padded in different ways. For example, the weight matrix for a particular convolution kernel is represented as [Kh, Kw, Ci], where Kh represents the kernel's height, Kw represents the kernel's width, and Ci represents the number of input channels. If the weight matrix before expansion is [Kh, 3, Ci] and the weight matrix after expansion is [Kh, 6, Ci], then a zero-value matrix of size [Kh, 3, Ci] is required for padding, where each value in this zero-value matrix is ​​equal to 0.

[0106] Optionally, if the size of the convolution kernel is expanded in the width dimension, padding is required in the width dimension. For example, the first dimension, the second dimension, and the third dimension are all widths, the third dimension is equal to m, and m is a positive integer. The computer device then fills i-1 columns of target values ​​on the left side of the weight matrix of each convolution kernel in the i-th group of convolution kernel sets, and fills m-i+1 columns of target values ​​on the right side of the weight matrix of each convolution kernel, where i is a positive integer not greater than k. When the value of i changes, different numbers of columns of target values ​​can be filled on the left or right side of the weight matrix of the convolution kernels in different groups of convolution kernel sets, so that the positions of the filled target values ​​in the weight matrices of the convolution kernels in different groups of convolution kernel sets are not exactly the same.

[0107] Figure 5 is a schematic diagram of a convolution kernel expansion method provided by an embodiment of the present application. The expansion coefficient is equal to 4, the number of convolution kernels Co before expansion is 16, the number of convolution kernels Co after expansion is 64, the width Kw of the convolution kernel before expansion is 3, the width Kw of the convolution kernel after expansion is 6, the height Kh of the convolution kernel and the number of input channels Ci remain unchanged, then it is necessary to fill the right side of the weight matrix of each convolution kernel in the first group of convolution kernel sets with a zero value matrix with a width of 3; fill the left side of the weight matrix of each convolution kernel in the second group of convolution kernel sets with a zero value matrix with a width of 1, and fill the right side with a zero value matrix with a width of 2; fill the left side of the weight matrix of each convolution kernel in the third group of convolution kernel sets with a zero value matrix with a width of 2, and fill the right side with a zero value matrix with a width of 1; fill the left side of the weight matrix of each convolution kernel in the fourth group of convolution kernel sets with a zero value matrix with a width of 3.

[0108] Optionally, if the convolution kernel is expanded in size in the height dimension, padding is required in the height dimension. The first, second, and third dimensions are all heights, and the third dimension is equal to n, where n is a positive integer. The computer device pads the upper side of the weight matrix of each convolution kernel in the i-th set of convolution kernels with i-1 rows of target values, and pads the lower side of the weight matrix of each convolution kernel with n-i+1 rows of target values, where i is a positive integer not greater than k.

[0109] For example, if the expansion coefficient is 4, the number of convolution kernels Co before expansion is 16, the number of convolution kernels Co after expansion is 64, the height Kw of the convolution kernel before expansion is 3, and the height Kw of the convolution kernel after expansion is 6, then it is necessary to fill the lower side of the weight matrix of each convolution kernel of the first group of convolution kernel sets with a zero value matrix with a height of 3; fill the upper side of the weight matrix of each convolution kernel of the second group of convolution kernel sets with a zero value matrix with a height of 1, and fill the lower side with a zero value matrix with a height of 2; fill the upper side of the weight matrix of each convolution kernel of the third group of convolution kernel sets with a zero value matrix with a height of 2, and fill the lower side with a zero value matrix with a height of 1; fill the upper side of the weight matrix of each convolution kernel of the fourth group of convolution kernel sets with a zero value matrix with a height of 3.

[0110] It should be noted that, in addition to the two optional schemes described above, the computer device can also adopt other optional schemes to fill the target value in the weight matrix of each convolution kernel in each group of convolution kernel sets. It is only necessary to ensure that the positions of the target values ​​filled in the weight matrices of the same group are the same, and the positions of the target values ​​filled in the weight matrices of different groups are not exactly the same. For example, for the four groups of convolution kernel sets shown in Figure 5, the weight matrix of each convolution kernel in the second group of convolution kernel sets is filled with a zero value matrix with a width of 2 on the left, and a zero value matrix with a width of 1 on the right; the weight matrix of each convolution kernel in the third group of convolution kernel sets is filled with a zero value matrix with a width of 1 on the left, and a zero value matrix with a width of 2 on the right, while the filling method of the weight matrix of each convolution kernel in the first and fourth groups of convolution kernel sets remains unchanged.

[0111] 307. The computer device convolves the input data and the expanded convolution kernel on the computing unit of the systolic array to obtain output data.

[0112] After obtaining the expanded convolution kernel, the computer device convolves the input data with the expanded convolution kernel on the computational units of the systolic array to obtain output data. Multiple columns of computational units in the systolic array perform convolutions on the input data and the convolution kernel in parallel. As the number of expanded convolution kernels increases, the number of columns of computational units in the systolic array used also increases, thereby improving the utilization of the systolic array.

[0113] It should be noted that if the convolution kernel is expanded only in the width dimension, the output data obtained after expansion can be directly equivalent to the output data obtained before expansion, without the need for data rearrangement. The output data can be directly used as the processing result or as the next input data. If the convolution kernel is expanded in the height dimension or the height and width dimensions, after obtaining the output data, it is necessary to rearrange the output data according to the expansion method so that the output data obtained after rearrangement is equivalent to the output data obtained before expansion.

[0114] Figure 6 is a schematic diagram of another data processing method provided by an embodiment of the present application. Figure 6 takes expansion in the width dimension as an example. The input data is [Hi, Wi, Ci], Hi represents the height of the input data, Wi represents the width of the input data, Ci represents the number of channels of the input data, and the convolution kernel is [Kh, Kw, Ci, Co], Kh represents the height of the convolution kernel, Kh=3, Kw represents the width of the convolution kernel, Kw=3, Ci represents the number of input channels of the convolution kernel, Co represents the number of output channels of the convolution kernel, which is also the number of convolution kernels, Co=16, and the convolution step size Stride_W of the convolution kernel is 1. If the convolution kernel is not expanded, the output data is [Ho, Wo, Co].

[0115] To improve the utilization of the systolic array, the convolution kernel is expanded in the width dimension with an expansion factor of Co_Expansion = 4. The expanded convolution kernel is [Kh, New_Kw, Ci, New_Co], New_Kw = Kw + (Co_Expansion - 1) × Stride_W = 6, New_Co = Co_Expansion × Co = 64, and the expanded output data is [Ho, Wo / 4, New_Co]. The number of expanded convolution kernels is 64, as shown in Figure 6. These 64 convolution kernels are arranged from the first in the first row to the last in the first row, then from the first in the second row to the last in the second row, and so on, until the last in the last row. The bolded portion of the convolution kernel is obtained by copying the original convolution kernel, and the non-bold portion is filled with 0.

[0116] Before the convolution kernel is expanded, the computation time required for the convolution operation on the systolic array can be expressed as: Compute_Time = Ho × Wo × Ceil (Co / N) × Kh × Kw × Ceil (Ci / K);

[0117] Among them, Compute_Time represents the computing time before expansion, Ho represents the height of the output data, Wo represents the width of the output data, Co represents the number of channels of the output data, N represents the number of columns of the systolic array, Ceil(Co / N) represents rounding up Co / N, Kh represents the height of the convolution kernel, Kw represents the width of the convolution kernel, Ci represents the number of input channels of the convolution kernel, which is also the number of channels of the input data, K represents the number of rows of the systolic array, and Ceil(Ci / K) represents rounding up Ci / K.

[0118] After expanding the convolution kernel, the computation time required to perform the convolution operation on the systolic array can be expressed as:

[0119] New_Compute_Time=Ho×Wo / Co_Expansion×Ceil(Co×Co_Expansion / N)×Kh×New_Kw×Ceil(Ci / K);

[0120] Among them, New_Compute_Time represents the calculation time after expansion, Ho represents the height of the output data, Wo represents the width of the output data, Co_Expansion represents the expansion coefficient, Co represents the number of channels of the output data, N represents the number of columns of the systolic array, Ceil(Co×Co_Expansion / N) represents rounding up Co×Co_Expansion / N, Kh represents the height of the convolution kernel, New_Kw represents the width of the convolution kernel after expansion, Ci represents the number of input channels of the convolution kernel, which is also the number of channels of the input data, K represents the number of rows of the systolic array, and Ceil(Ci / K) represents rounding up Ci / K.

[0121] For scenarios where Co is small, Ceil(Co / N) and Ceil(Co×Co_Expansion / N) in the above two formulas are both equal to 1. Therefore, by expanding the convolution kernel, the acceleration factor of the operation time can be expressed as: Ratio=Co_Expansion×Kw / New_Kw;

[0122] Where Ratio represents the computational time acceleration, Co_Expansion represents the expansion coefficient, Kw represents the kernel width, and New_Kw represents the kernel width after expansion. The computational time acceleration varies with Co_Expansion and Kw. For example, for a convolution kernel with Kh = 3, Kw = 3, Stride_w = 1, and Co = 16, the acceleration is 2 for a systolic array of N = 64.

[0123] In the method provided in an embodiment of the present application, if the number of convolution kernels is not greater than the number of columns of the systolic array, some computing units in the systolic array will not be used. In this case, an expansion factor is determined according to the ratio between the number of columns of the systolic array and the number of convolution kernels. Then, the original convolution kernel is expanded according to the expansion factor to increase the number of convolution kernels. At the same time, the size of the convolution kernel and the convolution step size are also increased to offset the impact of the increase in the number of convolution kernels. This is equivalent to converting the computational effort of a small number of convolution kernels at a smaller convolution step size into the computational effort of a larger number of convolution kernels at a larger convolution step size. Since the number of convolution kernels is increased, more computing units in the systolic array can be used, thereby improving the utilization rate of the systolic array and fully utilizing the computing power of the systolic array.

[0124] In another embodiment, target data to be processed is stored in a first memory on a processor, and the systolic array is configured to retrieve the target data from a second memory on the processor. The processor further includes a data transfer engine configured to transfer the target data from the first memory to the second memory, so that the systolic array can retrieve the target data from the second memory. For example, the first memory is the secondary memory or the tertiary memory shown in FIG1 , and the second memory is the primary memory shown in FIG1 .

[0125] Then, when the convolution kernel is expanded using the data processing method provided in the embodiment of the present application, the amount of data carried by the convolution kernel increases, thereby increasing the amount of data carried by the data handling engine. Therefore, when the convolution kernel needs to be expanded, it is also necessary to consider whether the amount of data carried by the data handling engine is too large. Based on this, the following embodiments shown in Figures 7 and 8 are provided to illustrate the data processing process, respectively, to determine whether the data handling engine supports on-path expansion.

[0126] FIG7 is a flowchart of another data processing method provided by an embodiment of the present application. The embodiment of the present application is executed by a computer device. FIG7 takes the data handling engine supporting on-line expansion as an example. Referring to FIG7 , the method includes the following steps.

[0127] 701. A computer device obtains target data and a column number of a systolic array, where the target data includes input data and at least one convolution kernel. The systolic array is a computing device in a processor, and the computing device includes a plurality of computing units. The computing unit in each column of the systolic array is used to convolve the input data with a convolution kernel.

[0128] The step 701 is similar to the above step 201 and will not be described again here.

[0129] 702. When the number of convolution kernels is no more than half the number of columns of the systolic array, the computer device determines a ratio between the number of columns of the systolic array and the number of convolution kernels, and rounds down the ratio to obtain a reference value.

[0130] The computer device determines whether the number of convolution kernels is greater than half the number of columns of the systolic array. If the number of convolution kernels is not greater than half the number of columns of the systolic array, the convolution kernels need to be expanded to improve utilization of the systolic array. The computer device determines a ratio between the number of columns of the systolic array and the number of convolution kernels, rounds down the ratio to obtain a reference value, and subsequently determines an expansion factor based on the reference data.

[0131] 703. When the data transfer engine supports on-path expansion, the computer device determines the reference value as the expansion coefficient.

[0132] The computer device determines whether the data handling engine in the processor supports on-path expansion. On-path expansion refers to the expansion of the convolution kernel in any memory. If the data handling engine supports on-path expansion, it means that the data handling engine has the function of expanding the convolution kernel in any memory. In this case, the data handling engine can first transfer the target data from the first memory to the second memory, and then expand the convolution kernel in the second memory. Since the convolution kernel has not been expanded during the transfer, there is no need to consider the impact of the expanded convolution kernel on the data handling process. The expansion coefficient must be an integer not greater than the reference value, that is, the maximum value of the expansion coefficient is the reference value. The larger the expansion coefficient, the higher the utilization rate of the systolic array. Therefore, the computer device directly determines the reference value as the expansion coefficient.

[0133] 704. The computer device transfers the input data and the convolution kernel from the first memory to the second memory through the data transfer engine, and expands the convolution kernel in the second memory based on the expansion coefficient to increase the number, size and convolution step of the expanded convolution kernel.

[0134] Since the data handling engine has the ability of on-path expansion, the computer device first transfers the input data and the convolution kernel from the first memory to the second memory through the data handling engine, and then expands the convolution kernel in the second memory based on the expansion coefficient through the data handling engine, so that the number, size and convolution step length of the expanded convolution kernel are increased, thereby ensuring that the subsequent systolic array can obtain the expanded convolution kernel in the second memory.

[0135] The process of expanding the convolution kernel based on the expansion coefficient in the second memory is the same as the process of steps 303 to 306 above, and will not be described in detail here.

[0136] 705. The computer device obtains the input data and the expanded convolution kernel in the second memory through the systolic array, and convolves the input data and the expanded convolution kernel on the computing unit of the systolic array to obtain output data.

[0137] The computer device obtains the input data and the expanded convolution kernel stored in the second memory through the systolic array, inputs the input data and the expanded convolution kernel into the computing unit of the systolic array, convolves the input data and the expanded convolution kernel to obtain output data.

[0138] The method provided in an embodiment of the present application sets the expansion coefficient to the maximum achievable value when the data handling engine supports in-path expansion. The input data and the convolution kernel are first transferred from the first memory to the second memory through the data handling engine, and then the convolution kernel is expanded in the second memory based on the expansion coefficient through the data handling engine. On the one hand, the utilization rate of the systolic array can be maximized, and on the other hand, the influence of the expanded convolution kernel on the data handling process can be avoided, which is beneficial to improving the overall efficiency of the data processing process and realizing accelerated data processing.

[0139] FIG8 is a flowchart of another data processing method provided by an embodiment of the present application. The embodiment of the present application is executed by a computer device. FIG7 takes the case where the data transport engine does not support on-path expansion as an example. Referring to FIG8 , the method includes the following steps.

[0140] 801. A computer device obtains target data and a column number of a systolic array, where the target data includes input data and at least one convolution kernel. The systolic array is a computing device in a processor, and the computing device includes a plurality of computing units. The computing unit in each column of the systolic array is used to convolve the input data with a convolution kernel.

[0141] The step 701 is similar to the above step 201 and will not be described again here.

[0142] 802. When the number of convolution kernels is no more than half the number of columns of the systolic array, the computer device determines a ratio between the number of columns of the systolic array and the number of convolution kernels, and rounds down the ratio to obtain a reference value.

[0143] The computer device determines whether the number of convolution kernels is greater than half the number of columns of the systolic array. If the number of convolution kernels is not greater than half the number of columns of the systolic array, the convolution kernels need to be expanded to improve utilization of the systolic array. The computer device determines a ratio between the number of columns of the systolic array and the number of convolution kernels, rounds down the ratio to obtain a reference value, and subsequently determines an expansion factor based on the reference data.

[0144] 803. When the data transfer engine does not support on-path expansion, the computer device determines an integer greater than 1 and less than a reference value as an expansion coefficient.

[0145] The computer device determines whether the data handling engine in the processor supports on-path expansion. On-path expansion refers to expanding the convolution kernel in any memory. If the data handling engine does not support on-path expansion, it indicates that the data handling engine does not have the function to expand the convolution kernel in any memory. In this case, it is impossible to first move the data and then expand it. Therefore, the impact of expanding the convolution kernel on the data handling process needs to be considered. The expansion factor must be an integer not greater than a reference value, that is, the maximum value of the expansion factor is this reference value. The larger the expansion factor, the higher the utilization of the systolic array. However, the larger the expansion factor, the greater the amount of data required to be moved and the longer the data transfer time. Therefore, the computer device determines the expansion factor as an integer greater than 1 and less than the reference value to minimize the amount of data required to be moved, taking into account the computation time saved by expanding the convolution kernel and the increased processing time. By expanding based on a reasonable expansion factor, the sum of the processing time and computation time after expansion is less than the sum of the processing time and computation time before expansion.

[0146] 804. The computer device expands the convolution kernel in the first memory based on the expansion coefficient to increase the number, size and convolution step of the expanded convolution kernel, and transfers the input data and the expanded convolution kernel from the first memory to the second memory through the data transfer engine.

[0147] Before moving data, the computer device first expands the convolution kernel based on the expansion coefficient in the first memory to increase the number, size and convolution step of the expanded convolution kernel. After completing the expansion of the convolution kernel, the input data and the expanded convolution kernel are moved from the first memory to the second memory through the data transfer engine, thereby ensuring that the subsequent systolic array can obtain the expanded convolution kernel in the second memory.

[0148] The process of expanding the convolution kernel based on the expansion coefficient in the first memory is the same as the process of steps 303 to 306 above, and will not be described in detail here.

[0149] 805. The computer device obtains the input data and the expanded convolution kernel in the second memory through the systolic array, and convolves the input data and the expanded convolution kernel on the computing unit of the systolic array to obtain output data.

[0150] The computer device obtains the input data and the expanded convolution kernel stored in the second memory through the systolic array, inputs the input data and the expanded convolution kernel into the computing unit of the systolic array, convolves the input data and the expanded convolution kernel to obtain output data.

[0151] The method provided in the embodiment of the present application sets the expansion coefficient to an integer less than the maximum achievable value if the data handling engine does not support on-path expansion, taking into account the computing time saved by the expanded convolution kernel and the increased handling time of the expanded convolution kernel. By performing expansion based on a reasonable expansion coefficient, the sum of the handling time and computing time spent after the expansion is less than the sum of the handling time and computing time spent before the expansion. On the one hand, it can improve the utilization rate of the systolic array, and on the other hand, it can minimize the impact of the expanded convolution kernel on the data handling process, which is conducive to improving the overall efficiency of the data processing process and realizing accelerated data processing.

[0152] FIG9 is a flow chart of another data processing method provided by an embodiment of the present application. As shown in FIG9 , a convolutional neural network provides input data and matrix data, and the matrix data refers to the convolution kernel and the bias parameters of the convolution kernel. The computer device determines whether the data transfer engine supports on-path expansion. If on-path expansion is supported, the expansion coefficient is calculated according to the maximum computing power utilization of the systolic array, that is, the maximum expansion coefficient, such as the value of the expansion coefficient is the reference value. If on-path expansion is not supported, it is necessary to comprehensively consider the computing power utilization and the data transfer volume to calculate the expansion coefficient to ensure that the saved computing time is greater than the increased transfer time, such as the value of the expansion coefficient is an integer greater than 1 and less than the reference value.

[0153] Figure 10 is a flow chart of another data processing method provided by an embodiment of the present application. As shown in Figure 10, the convolutional neural network provides input data and matrix data. The computer device compares the number of convolution kernels Co with the number of columns N of the systolic array. If Co is greater than N / 2, the matrix data is not expanded. If Co is not greater than N / 2, the matrix data needs to be expanded. The computer device calculates the maximum expansion coefficient (i.e., the reference value) and determines whether the data transfer engine supports on-path expansion. If on-path expansion is supported, the matrix data is expanded on-path according to the maximum expansion coefficient (expanded after transfer). If on-path expansion is not supported, the optimal expansion coefficient is calculated by comprehensively considering the operation time and transfer time. The optimal expansion coefficient is less than the maximum expansion coefficient, and the matrix data is expanded offline according to the optimal expansion coefficient (expanded before transfer). The computer device stores the expanded matrix data in the second memory, and performs convolution processing on the input data and the expanded matrix data through the systolic array to obtain output data.

[0154] The data processing method provided in the embodiments of the present application can improve the performance of systolic arrays processing a small number of convolution kernels exponentially, without requiring hardware architecture modifications. This method is widely applicable to different hardware architectures and convolution parameters. For example, when the height and width of the convolution kernel are equal to 3 and the convolution step size is 1, the expansion coefficient values ​​and corresponding computational speedups for different numbers of convolution kernels are shown in Table 1 below.

[0155] Table 1

[0156] As can be seen from Table 1, the embodiment of the present application can exponentially accelerate the operation time by expanding the convolution kernel without increasing the hardware complexity, thereby fully utilizing the computing power of the systolic array.

[0157] FIG11 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Referring to FIG11 , the device includes:

[0158] A data acquisition module 1101 is configured to acquire target data and a column number of a systolic array, where the target data includes input data and at least one convolution kernel. The systolic array is a computing device in a processor, and the computing device includes multiple computing units. Each computing unit in a column of the systolic array is configured to convolve the input data with a convolution kernel.

[0159] a coefficient determination module 1102 for determining an expansion coefficient based on a ratio between the number of columns of the systolic array and the number of convolution kernels when the number of convolution kernels is not greater than the number of columns of the systolic array;

[0160] An expansion module 1103 is configured to expand the convolution kernel based on the expansion coefficient so as to increase the number, size and convolution step length of the expanded convolution kernel;

[0161] The data processing module 1104 is configured to convolve the input data with the expanded convolution kernel on the computing units of the systolic array to obtain output data.

[0162] In the data processing device provided in an embodiment of the present application, if the number of convolution kernels is not greater than the number of columns of the systolic array, some computing units in the systolic array will not be used. Therefore, an expansion coefficient is determined according to the ratio between the number of columns of the systolic array and the number of convolution kernels. Then, the original convolution kernel is expanded according to the expansion coefficient to increase the number of convolution kernels. The size of the convolution kernel and the convolution step size are also increased to offset the effect of the increased number of convolution kernels. This is equivalent to converting the computational effort of a small number of convolution kernels at a smaller convolution step size into the computational effort of a larger number of convolution kernels at a larger convolution step size. Since the number of convolution kernels is increased, more computing units in the systolic array can be used, thereby improving the utilization rate of the systolic array and fully utilizing the computing power of the systolic array.

[0163] Optionally, the expansion module 1103 is configured to:

[0164] Based on the expansion coefficient and the first convolution step length, determining the second convolution step length, the first convolution step length is the convolution step length of the convolution kernel before expansion, and the second convolution step length is the convolution step length of the convolution kernel after expansion;

[0165] Determine a second size based on the expansion coefficient and the first size, where the first size is the size of the convolution kernel before expansion and the second size is the size of the convolution kernel after expansion;

[0166] Copy at least one convolution kernel k-1 times, where k is equal to the expansion factor;

[0167] Each convolution kernel is padded so that the size of the padded convolution kernel is equal to the second size.

[0168] Optionally, the expansion module 1103 is configured to:

[0169] Determining a second size based on the expansion coefficient, the first size, the first convolution step size, and the second convolution step size so that a ratio of the first product to the second product is equal to the expansion coefficient;

[0170] Among them, the first product represents the product of the height and width of the output data obtained by convolving the input data with the convolution kernel of the first size according to the first convolution step size, and the second product represents the product of the height and width of the output data obtained by convolving the input data with the convolution kernel of the second size according to the second convolution step size.

[0171] Optionally, the second convolution step size is equal to the product of the first convolution step size and the expansion coefficient, and the first size and the second size both represent the width of the convolution kernel; the expansion module 1103 is used to:

[0172] Determine the product of k-1 and the first convolution stride, and add the product to the first size to get the second size.

[0173] Optionally, the expansion module 1103 is configured to:

[0174] Multiply the expansion coefficient and the first convolution step size to get the second convolution step size.

[0175] Optionally, the convolution kernel is represented by a weight matrix, and the expansion module 1103 is used to:

[0176] According to the first size and the second size, the target value is filled in the weight matrix of each convolution kernel, so that the size of the filled weight matrix is ​​equal to the second size and the target value is equal to 0.

[0177] Optionally, the expansion module 1103 is configured to:

[0178] Divide the copied multiple convolution kernels into k groups of convolution kernel sets in sequence, each group of convolution kernel sets includes multiple consecutive convolution kernels, and the number of convolution kernels in each group of convolution kernel sets is equal;

[0179] determining a difference between the second size and the first size as a third size;

[0180] The target value is filled in the weight matrix of each convolution kernel in each group of convolution kernel sets, wherein the total size of the position where the target value is filled in the weight matrix of each convolution kernel is equal to the third size, and the positions where the target values ​​are filled in the weight matrices of the convolution kernels in the same group of convolution kernel sets are the same, and the positions where the target values ​​are filled in the weight matrices of the convolution kernels in different groups of convolution kernel sets are not exactly the same.

[0181] Optionally, the first size, the second size, and the third size are all widths, the third size is equal to m, and m is a positive integer; the expansion module 1103 is configured to:

[0182] The left side of the weight matrix of each convolution kernel in the i-th group of convolution kernels is filled with i-1 columns of target values, and the right side of the weight matrix of each convolution kernel is filled with m-i+1 columns of target values, where i is a positive integer not greater than k.

[0183] Optionally, the first size, the second size, and the third size are all heights, the third size is equal to n, and n is a positive integer; the expansion module 1103 is configured to:

[0184] The upper side of the weight matrix of each convolution kernel in the i-th group of convolution kernels is filled with i-1 rows of target values, and the lower side of the weight matrix of each convolution kernel is filled with n-i+1 rows of target values, where i is a positive integer not greater than k.

[0185] Optionally, the convolution kernel further includes a bias parameter, an expansion module 1103, for:

[0186] For each convolution kernel, copy the convolution kernel and its bias parameters k-1 times respectively.

[0187] Optionally, the coefficient determination module 1102 is configured to:

[0188] Determine the ratio between the number of columns in the systolic array and the number of convolution kernels;

[0189] Round down the comparison value to obtain the reference value;

[0190] Based on the reference value, an expansion factor is determined.

[0191] Optionally, the target data is stored in a first memory on the processor, the systolic array is used to obtain the target data from a second memory on the processor, and the processor further includes a data handling engine;

[0192] The coefficient determination module 1102 is used to:

[0193] In the case where the data handling engine supports on-path expansion, the reference value is determined as the expansion coefficient. On-path expansion refers to expanding the convolution kernel in any memory;

[0194] The expansion module 1103 is used to:

[0195] The input data and the convolution kernel are transferred from the first memory to the second memory through the data transfer engine, and the convolution kernel is expanded in the second memory based on the expansion coefficient.

[0196] Optionally, the target data is stored in a first memory on the processor, the systolic array is used to obtain the target data from a second memory on the processor, and the processor further includes a data handling engine;

[0197] The coefficient determination module 1102 is used to:

[0198] In the case where the data handling engine does not support on-path expansion, an integer greater than 1 and less than a reference value is determined as the expansion coefficient. On-path expansion refers to expanding the convolution kernel in any memory;

[0199] The expansion module 1103 is used to:

[0200] The convolution kernel is expanded in the first memory based on the expansion coefficient, and the input data and the expanded convolution kernel are transferred from the first memory to the second memory through the data transfer engine.

[0201] It should be noted that the data processing device provided in the above embodiment is merely illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0202] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the data processing method of the above embodiment.

[0203] Optionally, the computer device is provided as a terminal. FIG12 shows a schematic structural diagram of a terminal 1200 provided in an exemplary embodiment of the present application.

[0204] The terminal 1200 includes a processor 1201 and a memory 1202 .

[0205] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0206] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one computer program, which is used to be used by the processor 1201 to implement the data processing method provided in the method embodiment of the present application.

[0207] In some embodiments, terminal 1200 may optionally include a peripheral device interface 1203 and at least one peripheral device. Processor 1201, memory 1202, and peripheral device interface 1203 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1203 via a bus, signal lines, or circuit boards. Optionally, the peripheral device includes at least one of a radio frequency circuit 1204, a display screen 1205, or a camera assembly 1206.

[0208] The peripheral device interface 1203 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0209] RF circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. RF circuit 1204 communicates with communication networks and other communication devices using electromagnetic signals. RF circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals.

[0210] Display screen 1205 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, or any combination thereof. When display screen 1205 is a touch screen display, display screen 1205 is also capable of collecting touch signals on or above the surface of display screen 1205. The touch signals may be input as control signals to processor 1201 for processing.

[0211] The camera assembly 1206 is used to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. The front camera is set on the front panel of the terminal 1200, and the rear camera is set on the back of the terminal 1200.

[0212] Those skilled in the art will understand that the structure shown in FIG12 does not constitute a limitation on the terminal 1200 , and the terminal 1200 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0213] Optionally, the computer device is provided as a server. Figure 13 is a structural diagram of a server provided in an embodiment of the present application. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein the memory 1302 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1301 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

[0214] An embodiment of the present application further provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the data processing method of the above embodiment.

[0215] An embodiment of the present application further provides a computer program product, including a computer program, which is loaded and executed by a processor to implement the operations performed by the data processing method of the above embodiment.

[0216] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0217] The above description is merely an optional embodiment of the embodiments of the present application and is not intended to limit the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application shall be included in the scope of protection of the present application.

Claims

1. A data processing method, applied to a computer device, comprising: Obtaining target data and the number of columns of a systolic array, wherein the target data includes input data and at least one convolution kernel, the systolic array is a computing device in a processor, the computing device includes a plurality of computing units, and the computing unit of each column in the systolic array is used to convolve the input data and a convolution kernel; In a case where the number of convolution kernels is not greater than the number of columns of the systolic array, determining an expansion coefficient based on a ratio between the number of columns of the systolic array and the number of convolution kernels; Based on the expansion coefficient, the convolution kernel is expanded so that the number, size and convolution step length of the expanded convolution kernel are increased; On the computing unit of the systolic array, the input data and the expanded convolution kernel are convolved to obtain output data.

2. The method according to claim 1, wherein: The step of expanding the convolution kernel based on the expansion coefficient so as to increase the number, size and convolution step length of the expanded convolution kernel includes: Based on the expansion coefficient and the first convolution step length, determining a second convolution step length, wherein the first convolution step length is the convolution step length of the convolution kernel before expansion, and the second convolution step length is the convolution step length of the convolution kernel after expansion; Determine a second size based on the expansion coefficient and the first size, wherein the first size is the size of the convolution kernel before expansion, and the second size is the size of the convolution kernel after expansion; Copying the at least one convolution kernel by k-1 copies, where k is equal to the expansion coefficient; Each convolution kernel is padded so that the size of the padded convolution kernel is equal to the second size.

3. The method according to claim 2, wherein: The determining the second size based on the expansion factor and the first size includes: Determining the second size based on the expansion coefficient, the first size, the first convolution step size, and the second convolution step size so that a ratio of the first product to the second product is equal to the expansion coefficient; Among them, the first product represents the product of the height and width of the output data obtained by convolving the input data with the convolution kernel of the first size according to the first convolution step size, and the second product represents the product of the height and width of the output data obtained by convolving the input data with the convolution kernel of the second size according to the second convolution step size.

4. The method according to claim 3, wherein: The second convolution step length is equal to the product of the first convolution step length and the expansion coefficient, and the first size and the second size both represent the width of the convolution kernel; and the determining the second size based on the expansion coefficient, the first size, the first convolution step length, and the second convolution step length so that the ratio of the first product to the second product is equal to the expansion coefficient comprises: Determine the product of the k-1 and the first convolution step size, and add the product to the first size to obtain the second size.

5. The method according to claim 2, wherein: The convolution kernel is represented by a weight matrix, and padding each convolution kernel so that the size of the padded convolution kernel is equal to the second size, including: According to the first size and the second size, a target value is filled in the weight matrix of each convolution kernel so that the size of the filled weight matrix is ​​equal to the second size, and the target value is equal to 0.

6. The method according to claim 5, wherein: Filling the target value in the weight matrix of each convolution kernel according to the first size and the second size so that the size of the filled weight matrix is ​​equal to the second size, includes: The copied multiple convolution kernels are divided into k groups of convolution kernel sets in order, each group of convolution kernel sets includes multiple consecutive convolution kernels, and the number of convolution kernels in each set of convolution kernels is equal; determining a difference between the second size and the first size as a third size; The target value is filled in the weight matrix of each convolution kernel in each group of convolution kernel sets, wherein the total size of the position where the target value is filled in the weight matrix of each convolution kernel is equal to the third size, and the positions where the target values ​​are filled in the weight matrices of the convolution kernels in the same group of convolution kernel sets are the same, while the positions where the target values ​​are filled in the weight matrices of the convolution kernels in different groups of convolution kernel sets are not exactly the same.

7. The method according to claim 6, wherein: The first size, the second size and the third size are all widths, the third size is equal to m, and m is a positive integer; Filling the target value in the weight matrix of each convolution kernel in each convolution kernel set includes: The i-1 columns of target values ​​are filled on the left side of the weight matrix of each convolution kernel in the i-th group of convolution kernels, and the m-i+1 columns of target values ​​are filled on the right side of the weight matrix of each convolution kernel, where i is a positive integer not greater than k.

8. The method according to claim 6, wherein: The first size, the second size and the third size are all heights, the third size is equal to n, and n is a positive integer; Filling the target value in the weight matrix of each convolution kernel in each convolution kernel set includes: The upper side of the weight matrix of each convolution kernel in the i-th group of convolution kernels is filled with the target values ​​in i-1 rows, and the lower side of the weight matrix of each convolution kernel is filled with n-i+1 rows of the target values, where i is a positive integer not greater than k.

9. The method according to claim 2, wherein: The convolution kernel further includes a bias parameter, and the step of replicating the at least one convolution kernel by k-1 copies includes: For each of the convolution kernels, the convolution kernel and the bias parameters of the convolution kernel are respectively copied k-1 times.

10. The method according to claim 2, wherein: The determining the second convolution step length based on the expansion coefficient and the first convolution step length comprises: The expansion coefficient and the first convolution step length are multiplied to obtain the second convolution step length.

11. The method according to any one of claims 1 to 10, wherein: The determining of the expansion coefficient based on the ratio between the number of columns of the systolic array and the number of convolution kernels comprises: Determining a ratio between the number of columns of the systolic array and the number of convolution kernels; Rounding down the ratio to obtain a reference value; Based on the reference value, the expansion factor is determined.

12. The method according to claim 11, wherein: The target data is stored in a first memory on the processor, the systolic array is used to obtain the target data from a second memory on the processor, and the processor also includes a data handling engine; The step of determining the expansion coefficient based on the reference value includes: In the case where the data handling engine supports on-path expansion, determining the reference value as the expansion coefficient, wherein the on-path expansion refers to expanding the convolution kernel in any memory; The step of expanding the convolution kernel based on the expansion coefficient includes: The input data and the convolution kernel are transferred from the first memory to the second memory through the data transfer engine, and the convolution kernel is expanded in the second memory based on the expansion coefficient.

13. The method according to claim 11, wherein: The target data is stored in a first memory on the processor, the systolic array is used to obtain the target data from a second memory on the processor, and the processor also includes a data handling engine; The step of determining the expansion coefficient based on the reference value includes: In the case where the data handling engine does not support on-path expansion, an integer greater than 1 and less than the reference value is determined as the expansion coefficient, wherein the on-path expansion refers to expanding the convolution kernel in any memory; The step of expanding the convolution kernel based on the expansion coefficient includes: The convolution kernel is expanded in the first memory based on the expansion coefficient, and the input data and the expanded convolution kernel are transferred from the first memory to the second memory through the data transfer engine.

14. A data processing device, comprising: a data acquisition module, configured to acquire target data and a column number of a systolic array, wherein the target data includes input data and at least one convolution kernel, the systolic array is a computing device in a processor, the computing device includes a plurality of computing units, and the computing unit of each column in the systolic array is configured to convolve the input data and a convolution kernel; a coefficient determination module, configured to determine an expansion coefficient based on a ratio between the number of columns of the systolic array and the number of convolution kernels when the number of convolution kernels is not greater than the number of columns of the systolic array; An expansion module, used to expand the convolution kernel based on the expansion coefficient, so that the number, size and convolution step length of the expanded convolution kernel are increased; A data processing module is used to convolve the input data and the expanded convolution kernel on the computing unit of the systolic array to obtain output data.

15. A computer device, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the data processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the data processing method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, wherein the computer program is loaded and executed by a processor to implement the operations performed by the data processing method according to any one of claims 1 to 13.

Images