Data access optimization method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a data access optimization method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the shape of data and the memory arrangement type of the data; determining a processing unit arrangement mode set based on the memory arrangement type of the data and the number of processing units on hardware; and based on the shape of the data, determining a target arrangement mode from the processing unit arrangement mode set, and applying the target arrangement mode to access the data. According to the method and device, the electronic equipment and the storage medium provided by the invention, adaptive adjustment of the arrangement mode of the processing units can be realized, and the utilization rate of hardware resources is maximized, so that efficient access of data is realized, and the overall performance of operator operation is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data access optimization method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of big data and artificial intelligence, the demand for computing resources for large models has increased dramatically. GPU (Graphics Processing Unit) and GPGPU (General-Purpose computing on Graphics Processing Units) have become the key to supporting the operation of these large models due to their powerful parallel computing capabilities.

[0003] In GPU and GPGPU, different PUs (Processing Units) use their own independent hardware resources to access data at corresponding locations. For example, in MMA (Matrix Multiply and Accumulate) operations, multiple matrices need to be read. The memory layout types of matrices include row-major and column-major. However, for different memory layout types, the layout of data accessed by different PUs in GPU or GPGPU is fixed, and it cannot be adaptively adjusted according to the memory layout type of the matrix, thus limiting the full use of hardware performance and affecting data access efficiency. Summary of the invention

[0004] The present invention provides a data access optimization method, device, electronic device and storage medium, which are used to solve the defect that the arrangement of processing units in the related art cannot be adaptively adjusted, resulting in low hardware performance and data access efficiency.

[0005] The present invention provides a data access optimization method, comprising: Obtaining the shape of data and the memory arrangement type of the data; Determine a set of processing unit arrangement modes based on the memory arrangement type of the data and the number of processing units on the hardware; Based on the shape of the data, a target arrangement is determined from the set of processing unit arrangement modes, and the target arrangement is applied to access the data.

[0006] According to a data access optimization method provided by the present invention, determining a set of processing unit arrangement modes based on the memory arrangement type of the data and the number of processing units on the hardware includes: Based on the number of processing units on the hardware, determining a set of initial arrangements of the processing units on the hardware; Based on the memory arrangement type of the data, the initial arrangement mode set is screened to obtain the processing unit arrangement mode set.

[0007] According to a data access optimization method provided by the present invention, determining an initial arrangement set of processing units on the hardware based on the number of processing units on the hardware includes: Splitting the number of processing units on the hardware on a specified dimension to obtain a split result, wherein the specified dimension is two-dimensional or three-dimensional; Based on the splitting result, an initial arrangement set of processing units on the hardware is obtained.

[0008] According to a data access optimization method provided by the present invention, when the specified dimension is two-dimensional, the specified dimension includes a row dimension and a column dimension; The filtering of the initial arrangement mode set based on the memory arrangement type of the data to obtain the processing unit arrangement mode set includes: When the memory arrangement type of the data is row-major order, an arrangement method in which the value of the column dimension is greater than or equal to the value of the row dimension is selected from the initial arrangement method set to obtain the processing unit arrangement method set; When the memory arrangement type of the data is column-major order, arrangement methods whose row dimension values ​​are greater than or equal to column dimension values ​​are filtered out from the initial arrangement method set to obtain the processing unit arrangement method set.

[0009] According to a data access optimization method provided by the present invention, the target arrangement mode is determined from the processing unit arrangement mode set based on the shape of the data, including: Determining, based on the data block dimension of the processing unit, the data dimension corresponding to each arrangement mode in the processing unit arrangement mode set; Compare the data dimensions corresponding to the various arrangements with the shape of the data, and determine a candidate arrangement from the various arrangements; When there is one candidate arrangement, the candidate arrangement is determined as the target arrangement.

[0010] According to a data access optimization method provided by the present invention, the step of determining a candidate arrangement mode from the various arrangement modes further includes: In the case where there are at least two candidate arrangements, the target arrangement is determined from the candidate arrangements based on the difference between the data dimension corresponding to each candidate arrangement and the shape of the data.

[0011] The present invention also provides a data access optimization device, comprising: An acquisition unit, used to acquire the shape of data and the memory arrangement type of the data; A determination unit, configured to determine a set of processing unit arrangement modes based on a memory arrangement type of the data and a number of processing units on the hardware; The access unit is used to determine a target arrangement from the processing unit arrangement set based on the shape of the data, and access the data using the target arrangement.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned data access optimization methods is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data access optimization method described in any one of the above methods is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data access optimization methods.

[0015] The data access optimization method, device, electronic device and storage medium provided by the present invention can determine a set of processing unit arrangement modes by obtaining the shape and memory arrangement type of the data, and according to the memory arrangement type of the data and the number of processing units on the hardware, which provides a wealth of candidate solutions for subsequent optimization selection, so that the most suitable target arrangement mode can be found from different memory arrangement types, thereby realizing adaptive adjustment of the processing unit arrangement mode, thereby maximizing the utilization rate of hardware resources. By selecting the best target arrangement mode from the set of processing unit arrangement modes based on the shape of the data, and applying the target arrangement mode for data access, the efficiency of data access can be significantly improved, because the target arrangement mode can reduce the delay and conflict of memory access, improve the cache hit rate, thereby improving the execution speed of operator operations, and realizing the improvement of the overall performance of the computing platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1It is a schematic diagram of the arrangement of the column-major sequential time processing unit provided by the present invention; Figure 2 It is a schematic diagram of the arrangement of the processing units in the row-major sequential order provided by the present invention; Figure 3 It is a flow chart of the data access optimization method provided by the present invention; Figure 4 It is a flow chart of a data access optimization method based on adaptive arrangement of processing units provided by the present invention; Figure 5 It is a schematic diagram of the process of selecting the optimal arrangement mode provided by the present invention; Figure 6 It is a structural schematic diagram of the data access optimization device provided by the present invention; Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In the operation process of large models, the operator operation (especially MMA) is the core operation unit, and its performance directly determines the overall computing efficiency. However, the performance bottleneck of GPU, GPGPU, etc. is often not in its powerful computing power itself, but in the data access efficiency. Because in GPU and GPGPU, data reading and writing operations need to interact frequently with the main memory, the access delay and bandwidth limitation in this process become the main obstacles to performance improvement. Therefore, by optimizing the data access efficiency, the delay and bandwidth consumption of data transmission can be reduced, thereby effectively improving the execution speed of operator operations and significantly enhancing the performance of the entire computing platform (such as GPU, GPGPU, etc.).

[0020] In the operator development phase, developers need to design the processing unit arrangement and hardware resource allocation strategy according to the characteristics of the hardware and the expected application scenarios. However, this design is often based on certain assumptions and expectations, and once determined, it will not be modified in subsequent stages, which leads to the solidification of the processing unit arrangement and hardware resource allocation method. It should be understood that the processing unit arrangement refers to the arrangement of the processing unit in the calculation process. The arrangement includes determining the memory area that each processing unit is responsible for accessing. Hardware resource allocation refers to the allocation of effective hardware resources to different processing units. Here, the processing unit can be a programmable multiprocessor, such as an SPC (Streaming Processing Cluster) in a GPU, a GPU has multiple SPCs, each SPC has multiple CUs (Compute Units), and each CU has multiple EUs (Execution Units); the processing unit can also be a CU, and the processing unit can also be an EU, which is not specifically limited in the embodiment of the present invention.

[0021] It should be noted that the above-mentioned processing unit arrangement does not refer to the physical arrangement and distribution of the processing units on the hardware, but refers to how to optimally allocate data blocks to each processing unit and determine the memory area that each of them is responsible for accessing and processing when processing tasks (such as performing matrix multiplication calculations) to ensure efficient calculations and reasonable data flow.

[0022] Since the design of processing unit arrangement and hardware resource allocation strategy requires a deep understanding of hardware characteristics and application scenario requirements, this process is often highly dependent on the experience of developers and is difficult to dynamically adjust according to actual application scenarios. With the diversification of application scenarios, this fixed strategy cannot fully utilize hardware resources, resulting in performance waste.

[0023] In addition, the memory arrangement types of data include row-major order and column-major order. The memory arrangement types here mainly refer to the storage order and layout of data in memory, especially for multi-dimensional data structures (such as matrices). Row-major order means that data is stored in memory in the order of rows, and column-major order means that data is stored in memory in the order of columns. Faced with different data memory arrangement types, due to the fixed arrangement of processing units, it cannot be adaptively adjusted, and the fixed arrangement of processing units cannot be applied to all data memory arrangement types, resulting in the inability to give full play to the hardware performance. It should be understood that the fixed arrangement of processing units is usually designed based on a specific memory arrangement type, so it cannot be applied to all memory arrangement types. For example, for the arrangement type of row-major order, a corresponding arrangement of processing units is designed to optimize the access efficiency of row-major order data, but for column-major order data, this arrangement may cause a decrease in cache hit rate due to frequent cross-memory access, thereby reducing data access efficiency.

[0024] In this regard, the present invention provides a data access optimization method, which can realize adaptive adjustment of the processing unit arrangement mode according to the shape of the data and the memory arrangement type, so as to maximize the utilization rate of hardware resources, thereby realizing efficient access to data, improving the overall performance of the operator, and thus overcoming the above-mentioned defects. It should be noted that the processing unit arrangement mode in the present invention does not refer to the tangible arrangement of the processing units at the hardware level, but an optimization decision at the software level, which is determined based on factors such as the shape of the data and the memory arrangement type and the performance characteristics of the processing units, with the purpose of maximizing the efficiency of data access and making full use of the parallel processing capabilities of the hardware.

[0025] Figure 1 Schematic diagram of the arrangement of the column-major sequential processing unit provided by the present invention, such as Figure 1 As shown, for column-major order, the memory area is arranged in the column direction first. For example, assuming that the memory area is divided into 4, that is, Figure 1 As shown in the figure, the four memory regions are arranged along the column direction. In this case, the processing unit array (i.e. Figure 1PU0, PU1, PU2 and PU3 shown in the figure are arranged vertically to access memory, so that different PUs access different memory areas, which can avoid memory access conflicts, and these PUs can access corresponding memory areas in parallel, thereby effectively improving bandwidth and access efficiency. It should be understood that if two PUs access the same memory area, then the two PUs will execute serially when performing memory access. For example, if PU0 and PU1 both access Region0, then PU1 can access Region0 only after PU0 has accessed Region0. This arrangement has low access efficiency.

[0026] Figure 2 is a schematic diagram of the arrangement of the processing units in the row-major order provided by the present invention, such as Figure 2 As shown, for row-major order, the memory area is arranged in the row direction first, for example Figure 2 As shown in FIG, there are four memory regions Region0, Region1, Region2, and Region3 arranged in the row direction. In this case, in order to improve data access efficiency, the processing unit array (i.e. Figure 2 PU0, PU1, PU2 and PU3 shown in the figure are arranged horizontally for memory access, so that different PUs can access different memory areas, thereby avoiding memory access conflicts, and these PUs can access corresponding memory areas in parallel, thereby effectively improving bandwidth and access efficiency. Figure 1 and Figure 2 It can be seen that for different memory layout types, the processing unit also needs to make corresponding layout adjustments when accessing data to optimize data access performance.

[0027] Figure 3 is a flow chart of the data access optimization method provided by the present invention, such as Figure 3 As shown, the method includes: Step 310, obtaining the shape of the data and the memory arrangement type of the data; Specifically, the shape of data refers to the dimensional information of the data, especially for multidimensional arrays or matrices. For example, for matrices, the shape of data is usually described as the number of rows and columns. The shape of a matrix can be m×n, indicating that the matrix has m rows and n columns. In the operation process of large models, the shape of the data and the memory layout type can be obtained through the programming language and library used. For example, when using the NumPy library in the programming language Python, the shape of the data can be obtained through the .shape attribute. Assume that a matrix named A is defined, and A is represented as follows:

[0028] The shape of the matrix A can be obtained by A.shape in the programming language. The shape of the matrix A is (2, 2), which means that the matrix A has 2 rows and 2 columns. It should be understood that in the operation process of the large model, operator operations, especially matrix multiplication calculations, occupy a core position. For the matrix multiplication A×B=C, the above data is the matrix A, B, and C. Therefore, when obtaining the shape of the data, it is necessary to obtain the shapes of the matrices A, B, and C at the same time.

[0029] It is understandable that the memory layout type of data is related to the way the data is stored, the programming language and library used. For example, in the NumPy library, data is stored in row-major order by default. The memory layout type of the data can be determined by the programming language and library used. In addition, in matrix multiplication operations, especially when using deep learning frameworks such as TensorFlow and PyTorch, the memory layout type of the matrix is ​​usually managed by the data structure and layout rules within the framework. Therefore, by applying the framework to query metadata such as data shape, data type, and whether it is continuous, the memory layout of the data can be understood, thereby determining the memory layout type of the data.

[0030] Step 320, determining a processing unit arrangement set based on the memory arrangement type of the data and the number of processing units on the hardware; Specifically, after obtaining the memory arrangement type of the data, the set of processing unit arrangement methods can be determined according to the memory arrangement type of the data and the number of processing units on the hardware. The hardware here refers to the computing platform, which can be a GPU or GPGPU, etc. The processing unit on the hardware refers to the hardware component responsible for executing instructions and processing data on the hardware. For example, the processing unit can be a PU in a GPU or GPGPU. The number of processing units on the hardware refers to the total number of processing units available on the hardware, which determines the specific way in which the processing units are arranged along different dimensions.

[0031] In one embodiment, for either row-major or column-major memory arrangement types, a processing unit arrangement mode set can be generated based on the number of processing units on the hardware and possible arrangement dimensions (such as two-dimensional or three-dimensional), and the set includes a variety of different processing unit arrangement modes. For example, assuming that there are 16 processing units on the hardware, and these processing units are arranged in a two-dimensional space, the arrangement modes may include 1×16, 2×8, 4×4, 8×2, and 16×1, where 1×16 means that these 16 processing units are arranged in two-dimensional space according to 1 row and 16 columns. Subsequently, according to different memory arrangement types, all processing unit arrangement modes in the processing unit arrangement mode set can be traversed to select an arrangement mode that best matches the shape of the data.

[0032] In another embodiment, considering that the memory arrangement type of data (such as row-major or column-major) is different, the arrangement method of the memory area is different. For example, when the memory area is row-major, it will be arranged in the row direction first, and the processing units will be arranged horizontally at this time, while when the memory area is column-major, it will be arranged in the column direction first, and the processing units will be arranged vertically at this time. Therefore, when determining the processing unit arrangement method set, an initial arrangement method set can be generated according to the number of processing units on the hardware, and then the initial arrangement method set can be further filtered according to the memory arrangement type of the data. For row-major data, it is more inclined to select an arrangement method with a larger number of columns, and for column-major data, it is more inclined to select an arrangement method with a larger number of rows. Through screening, the number of different arrangement methods in the processing unit arrangement method set can be reduced, thereby shortening the subsequent traversal time, so as to quickly match the optimal target arrangement method.

[0033] It is understood that the processing unit arrangement set refers to a set of possible processing unit arrangements, which are determined based on the memory arrangement type of data and the number of processing units on the hardware. Each arrangement in the set corresponds to a specific processing unit layout, and by selecting a suitable arrangement from the set, the data access process can be optimized and computing performance can be improved.

[0034] Step 330: Based on the shape of the data, determine a target arrangement from the set of processing unit arrangement methods, and apply the target arrangement to access the data.

[0035] Specifically, after determining the set of processing unit arrangement methods, the optimal target arrangement method can be selected from them according to the shape of the data. Specifically, first, it is necessary to clarify the shape of the data, that is, the size of the data in each dimension. Next, it is necessary to determine the data block dimension (or data block size) that each processing unit (i.e., PU) can process, which usually depends on factors such as the internal architecture and cache size of the PU. For example, during the MMA operation, a PU can efficiently process a data block of a specific size (such as a 16×16 matrix). Subsequently, each arrangement method in the processing unit arrangement method set is traversed to select the arrangement method that can cover the shape of the data. Here, it is possible to determine whether the currently traversed arrangement method can cover the shape of the data based on the data block dimension that each PU can process, thereby determining the target arrangement method.

[0036] It is understood that the target arrangement refers to an optimal arrangement selected from a set of processing unit arrangements, which can achieve efficient data access and processing according to the shape of the data and the characteristics of the processing unit. The selection of the target arrangement is intended to maximize the data processing efficiency and parallel processing capabilities while reducing data access delays and conflicts.

[0037] Furthermore, after the target arrangement is determined, the target arrangement can be applied to data access during the operator operation. Specifically, first, the layout of each processing unit and the memory area accessed by each processing unit can be determined according to the target arrangement; then, each processing unit will use its corresponding memory resources and other hardware resources to access the data at the corresponding location, and process the data according to the predetermined algorithm and logic.

[0038] The method provided by the embodiment of the present invention can determine a set of processing unit arrangement modes by obtaining the shape and memory arrangement type of the data, and according to the memory arrangement type of the data and the number of processing units on the hardware, which provides a wealth of candidate solutions for subsequent optimization selection, so that the most suitable target arrangement mode can be found from different memory arrangement types, thereby realizing adaptive adjustment of the processing unit arrangement mode, thereby maximizing the utilization rate of hardware resources. By selecting the best target arrangement mode from the set of processing unit arrangement modes based on the shape of the data, and applying the target arrangement mode for data access, the efficiency of data access can be significantly improved, because the target arrangement mode can reduce the delay and conflict of memory access, improve the cache hit rate, and make data processing more in line with the parallel processing mode of the hardware, thereby improving the execution speed of operator operations and achieving an improvement in the overall performance of the computing platform.

[0039] Based on the above embodiment, step 320 specifically includes: Step 321, based on the number of processing units on the hardware, determining an initial arrangement set of processing units on the hardware; Specifically, determining the initial arrangement set of processing units on the hardware mainly depends on the number of processing units and the arrangement dimension. Specifically, first, the total number of processing units available on the hardware is clarified. This number may be a fixed value or a variable range (such as the number of processing units dynamically allocated according to task requirements). According to the characteristics and application scenarios of the processing units, the arrangement dimension is determined. In most cases, the arrangement dimension is two-dimensional (such as a grid arrangement), but it can also be three-dimensional or higher (depending on the specific application and hardware support). According to the number of processing units and the arrangement dimension, an initial arrangement set can be generated, which contains all possible (or reasonable) arrangements, each of which corresponds to a specific processing unit layout. For example, if the total number of processing units is 16 and a two-dimensional arrangement is considered, the possible arrangements include 1×16, 2×8, 4×4, 8×2, 16×1, etc. It should be understood that the initial arrangement set refers to a set containing all possible processing unit arrangements, which provides a basis for subsequent screening and optimization.

[0040] Step 322: Based on the memory arrangement type of the data, the initial arrangement method set is screened to obtain the processing unit arrangement method set.

[0041] It should be noted that the memory arrangement type (such as row-major or column-major) determines how data is stored in memory, which directly affects the efficiency and performance of data access. Different processing unit arrangements have different adaptability to different memory arrangement types. Therefore, in order to maximize the efficiency and parallelism of data access, the initial arrangement set can be screened according to the memory arrangement type to select a processing unit arrangement that is relatively more suitable for the current data memory arrangement.

[0042] Specifically, for each arrangement in the initial arrangement set, its adaptability to the current memory arrangement type can be evaluated, which can be achieved by simulating the data access process, analyzing the cache hit rate, predicting data conflicts, etc. Based on the results of the adaptability evaluation, some relatively better arrangements are selected from the initial arrangement set to obtain a processing unit arrangement set corresponding to the current memory arrangement type.

[0043] The method provided by an embodiment of the present invention can reduce the number of arrangements in the processing unit arrangement set by filtering the initial arrangement set by the memory arrangement type of the data, thereby shortening the time for subsequent traversal of the processing unit arrangement set, so as to quickly match the optimal target arrangement.

[0044] Based on any of the above embodiments, step 321 specifically includes: Step 3211, splitting the number of processing units on the hardware in a specified dimension to obtain a split result, wherein the specified dimension is two-dimensional or three-dimensional; Step 3212: Based on the splitting result, obtain a set of initial arrangement methods of the processing units on the hardware.

[0045] It should be noted that the specified dimension refers to the number of dimensions considered when determining the arrangement of processing units on the hardware. In most computing scenarios, the arrangement of processing units can be two-dimensional (such as a grid arrangement) or three-dimensional (such as a cube arrangement). The specified dimension determines how the processing units are laid out in space, which affects the efficiency of data access and processing.

[0046] Specifically, when determining the initial arrangement set according to the number of processing units on the hardware, it can be achieved by splitting the number of processing units on the hardware on a specified dimension. In essence, it is to distribute the total number of processing units to different dimensions to form a specific arrangement. Specifically, first clarify the total number of processing units available on the hardware, and then select the appropriate dimension (two-dimensional or three-dimensional) according to the application scenario and hardware characteristics. For example, in a two-dimensional arrangement, the total number of processing units is split into the product of the number of rows and the number of columns. Assuming that there are 16 processing units on the hardware, possible two-dimensional splits include 1×16, 2×8, 4×4, 8×2, 16×1, etc. The result of the split is a set of specific dimensional values ​​(such as the number of rows and columns), which together determine the arrangement of the processing units.

[0047] It can be understood that the split result refers to the values ​​of each dimension obtained after the total number of processing units on the hardware is split in the specified dimension. These values ​​directly determine the arrangement of the processing units in the physical or logical space. For example, in a two-dimensional split, the split result may be a set of row and column numbers (such as 4×4), indicating that the processing units are arranged in a grid of 4 rows and 4 columns.

[0048] After obtaining the split results, the initial arrangement of the processing units on the hardware can be obtained based on the split results. This step actually takes all possible split results (under the specified dimension) as candidate sets of arrangement methods. Specifically, according to the total number of processing units and the specified dimension, all possible split methods can be listed. For example, in a two-dimensional arrangement, if the total number of processing units is 16, the possible split results include 1×16, 2×8, 4×4, 8×2, 16×1, etc.; then, all the listed split results are regarded as different arrangements and combined into a set, which is the initial arrangement of the processing units on the hardware.

[0049] Based on any of the above embodiments, when the specified dimension is two-dimensional, the specified dimension includes a row dimension and a column dimension.

[0050] It should be noted that in two-dimensional space, the row dimension and column dimension are used to describe the distribution or arrangement of data or objects in the row and column directions. In the arrangement of processing units, these two dimensions are equally important, and they determine the specific position and shape of the processing units in the two-dimensional grid. Here, the row dimension refers to the number of processing units arranged along the row direction (from top to bottom) in the two-dimensional arrangement, which determines the number of rows in which the processing units are arranged. The column dimension refers to the number of processing units arranged along the column direction (from left to right) in the two-dimensional arrangement, which determines the number of columns in which the processing units are arranged.

[0051] Accordingly, step 322 specifically includes: When the memory arrangement type of the data is row-major order, an arrangement method in which the value of the column dimension is greater than or equal to the value of the row dimension is selected from the initial arrangement method set to obtain the processing unit arrangement method set; When the memory arrangement type of the data is column-major order, arrangement methods whose row dimension values ​​are greater than or equal to column dimension values ​​are filtered out from the initial arrangement method set to obtain the processing unit arrangement method set.

[0052] Specifically, when the specified dimension is two-dimensional, the arrangement of processing units can be filtered according to the memory arrangement type of the data (row-major or column-major), which can be achieved through the following steps: First, based on the number of processing units on the hardware and the requirements of the two-dimensional arrangement, an initial arrangement set is generated. This set contains all possible two-dimensional arrangements, and each arrangement is determined by the specific values ​​of the row dimension and the column dimension. Next, the memory arrangement type of the data is clarified, that is, whether the data is stored in the memory in row-major or column-major order. For row-major data, since row data is accessed first and the memory area is arranged in the row direction first, selecting an arrangement with a column dimension value greater than or equal to the row dimension value can optimize the continuity of data access. By filtering out the arrangement that meets this condition from the initial arrangement set, a processing unit arrangement set corresponding to the row-major memory arrangement type can be formed. Similarly, for column-major data, since column data is accessed first and the memory area is arranged in the column direction first, selecting an arrangement method in which the row dimension value is greater than or equal to the column dimension value can optimize the continuity of data access. By filtering out the arrangement method that meets this condition from the initial arrangement method set, a processing unit arrangement method set corresponding to the column-major memory arrangement type can be formed.

[0053] Exemplarily, in a two-dimensional arrangement, assuming that the total number of processing units on the hardware is 16, the arrangements included in the initial arrangement set include 1×16, 2×8, 4×4, 8×2, 16×1, etc. When the memory arrangement type of the data is row-major sequence, the arrangement methods in which the column dimension value is greater than or equal to the row dimension value can be filtered out from the initial arrangement set, that is, the filtered processing unit arrangement set includes 1×16, 2×8, and 4×4. When the memory arrangement type of the data is column-major sequence, the arrangement methods in which the row dimension value is greater than or equal to the column dimension value can be filtered out from the initial arrangement set, that is, the filtered processing unit arrangement set includes 4×4, 8×2, and 16×1.

[0054] After obtaining a set of processing unit arrangement methods, the most appropriate arrangement method can be further optimized and selected based on actual application scenarios and performance requirements. This can be implemented based on factors such as processing unit utilization, data access latency, cache utilization, etc. Finally, the selected processing unit arrangement method is applied to the actual data processing process to optimize the efficiency of data access and processing.

[0055] Based on any of the above embodiments, in step 330, determining a target arrangement from the set of processing unit arrangement based on the shape of the data includes: Step 331, based on the data block dimension of the processing unit, determine the data dimension corresponding to each arrangement mode in the processing unit arrangement mode set; Specifically, the data block dimension of a processing unit refers to the size of the amount of data that each processing unit can process simultaneously in a specific dimension. When processing units perform parallel computing, they are usually assigned a portion of data to process, and the size and shape (i.e., dimension) of this portion of data constitutes the data block dimension of the processing unit.

[0056] When determining the arrangement of processing units, the data block dimensions of the processing units need to be considered to ensure that each processing unit can effectively process the data block assigned to it. For a two-dimensional arrangement, each processing unit may be assigned a rectangular data block whose size in rows and columns corresponds to the data block dimensions in the row and column dimensions of the processing unit in the arrangement, respectively.

[0057] For example, suppose the data block dimension of the processing unit is (m, n), and the row dimension of a certain arrangement is r and the column dimension is c. That is, in this arrangement, the processing units are arranged in r rows and c columns. Then the data dimension corresponding to this arrangement is Here, the data dimension corresponding to the arrangement mode refers to the total dimension size of the data that can be processed by the entire processing unit array under a given processing unit arrangement mode, which is calculated based on the data block dimension of the processing unit and the row dimension and column dimension of the arrangement mode.

[0058] Step 332, comparing the data dimensions corresponding to the various arrangement modes with the shape of the data, and determining a candidate arrangement mode from the various arrangement modes; Specifically, when determining the target arrangement from the set of processing unit arrangements, each arrangement in the set can be traversed, and for the currently traversed arrangement, the corresponding data dimension is calculated, and the data dimension is compared with the shape of the data. If the data dimension corresponding to a certain arrangement is the same as the shape of the data or can cover the shape of the data, the arrangement can be regarded as a candidate arrangement. Here, the shape of the data is the shape of the data to be processed. For example, in MMA, the shape of the data is the shape of the input matrix and the output matrix that need to be matrix multiplied.

[0059] Step 333: If there is one candidate arrangement, determine the candidate arrangement as the target arrangement.

[0060] Specifically, if it is determined that there is only one candidate arrangement, it means that in a given set of processing unit arrangements, only one arrangement can adapt to the shape of the data. In this case, the candidate arrangement can be determined as the optimal target arrangement, and during the operator operation process, this arrangement is applied to access data.

[0061] For example, assume that the shape of the data is (H, W), where H is the height dimension, W is the width dimension, the values ​​of H and W are both 64, the memory arrangement type of the data is column-major order, there are 16 processing units on the hardware, and the data block dimension of each processing unit is (16, 16), then the processing unit arrangement set includes three arrangements: 4×4, 8×2, and 16×1. For the first arrangement 4×4, the corresponding data dimension is (64, 64), which just matches the shape of the data; for the second arrangement 8×2, the corresponding data dimension is (128, 32), which cannot cover the shape of the data in the width dimension; for the third arrangement 16×1, the corresponding data dimension is (256, 16), which also cannot cover the shape of the data in the width dimension. Therefore, it can be determined that the arrangement 4×4 is the target arrangement.

[0062] Based on any of the above embodiments, after step 332, the method further includes: In the case where there are at least two candidate arrangements, the target arrangement is determined from the candidate arrangements based on the difference between the data dimension corresponding to each candidate arrangement and the shape of the data.

[0063] Specifically, if there are at least two candidate arrangements, it means that in a given set of processing unit arrangements, there are multiple arrangements that can adapt to the shape of the data. For example, if the shape of the data is (64, 64), the memory arrangement type is column-major, the number of processing units on the hardware is 16, and the data block dimension of the processing unit is (64, 32), then the candidate arrangements may include 4×4, 8×2, and 16×1.

[0064] When there are multiple candidate arrangements, the difference between the data dimension and the shape of the data corresponding to each candidate arrangement can be calculated to further determine the optimal target arrangement. Specifically, this can be achieved by comparing the difference between the data dimension corresponding to the candidate arrangement and the shape of the data in the corresponding dimension. For example, assuming that the data dimension of a candidate arrangement is (H1, W1), and the actual shape of the data is (H, W), the difference in the height dimension and the width dimension are calculated respectively, that is, H1-H and W1-W. In order to facilitate the comparison of different candidate arrangements and determine the target arrangement, the difference in the height dimension and the width dimension can be weighted and summed (if the importance of the two dimensions is the same, the weights can be equal; otherwise, the weights can be adjusted according to the actual situation) to obtain a comprehensive difference for evaluation. Alternatively, other methods can be used to synthesize the differences in the two dimensions, such as taking the maximum value, the sum of squares, etc.

[0065] It is understandable that when determining the target arrangement from the candidate arrangements, multiple factors can also be considered at the same time, including but not limited to data dimension matching, processing efficiency, resource utilization, etc. Among them, data dimension matching refers to selecting the candidate arrangement whose data dimension is closest to the actual shape of the data, which can be achieved by comparing the difference corresponding to different candidate arrangements. The smallest difference and the fewer idle processing units have a higher matching degree. Processing efficiency refers to considering factors such as the communication cost between processing units under different arrangements and the locality of data access, and selecting an arrangement that can improve processing efficiency. Resource utilization refers to ensuring that the selected arrangement can make full use of the available processing unit resources to avoid resource waste. Taking all the above factors into consideration, heuristic search, optimization algorithm or rule-based decision-making method can be used to determine the target arrangement from the candidate arrangements.

[0066] Based on any of the above embodiments, Figure 4 is a flow chart of a data access optimization method based on adaptive arrangement of processing units provided by the present invention, such as Figure 4 As shown, taking data access in a matrix multiplication operation as an example, the method includes: Step S1, obtain the Shape of the matrix; Step S2, obtaining the memory arrangement type of the matrix (row-major order or column-major order); Step S3, determining an initial processing unit arrangement set according to the number of processing units on the hardware, and filtering the initial processing unit arrangement set according to the memory arrangement type to obtain a processing unit arrangement set corresponding to the memory arrangement type; Step S4, according to different memory arrangement types, traverse each arrangement mode in the processing unit arrangement mode set, and select an arrangement mode that best suits the data Shape; Step S5, after selecting the optimal arrangement, execute the Kernel (kernel function) of the matrix multiplication operation, so that during the execution process, the optimal arrangement can be applied to access data.

[0067] Specifically, in step S4, if the memory arrangement type of the data is column-major order, the set of processing unit arrangement methods corresponding to the column-major order is traversed to select the optimal arrangement method therefrom; if the memory arrangement type of the data is row-major order, the set of processing unit arrangement methods corresponding to the row-major order is traversed to select the optimal arrangement method therefrom.

[0068] Figure 5 It is a schematic diagram of the process of selecting the optimal arrangement method provided by the present invention, such as Figure 5 As shown, according to the total number of processing units (PUs) obtained, they can be combined in two dimensions, that is, the total number of PUs can be divided into , thus we can get a variety of (x, y) combinations, each of which corresponds to a specific arrangement of processing units. By traversing each (x, y) combination, we can select the optimal arrangement.

[0069] It is understandable that, since the distribution and number of processing units in different hardware are different, the memory arrangement type of the matrix is ​​not necessarily the same, and the same processing unit arrangement method is difficult to achieve optimal performance in various scenarios. Therefore, the embodiment of the present invention designs an adaptive processing unit arrangement strategy according to the memory arrangement type of the matrix. According to this method, the MMA operator can automatically and efficiently obtain the optimal allocation of hardware resources, thereby achieving efficient access to data and improving the overall performance of the MMA operator.

[0070] Based on any of the above embodiments, Figure 6 is a schematic diagram of the structure of the data access optimization device provided by the present invention, such as Figure 6 As shown, the device comprises: An acquisition unit 610 is used to acquire a shape of data and a memory arrangement type of the data; A determination unit 620, configured to determine a set of processing unit arrangement modes based on the memory arrangement type of the data and the number of processing units on the hardware; The access unit 630 is used to determine a target arrangement from the processing unit arrangement set based on the shape of the data, and apply the target arrangement to access the data.

[0071] The device provided by the embodiment of the present invention can determine a set of processing unit arrangement modes by obtaining the shape and memory arrangement type of the data, and according to the memory arrangement type of the data and the number of processing units on the hardware, which provides a wealth of candidate solutions for subsequent optimization selection, so that the most suitable target arrangement mode can be found from different memory arrangement types, thereby realizing adaptive adjustment of the processing unit arrangement mode, thereby maximizing the utilization rate of hardware resources. By selecting the best target arrangement mode from the set of processing unit arrangement modes based on the shape of the data, and applying the target arrangement mode for data access, the efficiency of data access can be significantly improved, because the target arrangement mode can reduce the delay and conflict of memory access, improve the cache hit rate, thereby improving the execution speed of operator operations, and realizing the improvement of the overall performance of the computing platform.

[0072] Based on any of the above embodiments, the determining unit 620 includes: Determine a subunit, for determining an initial arrangement set of processing units on the hardware based on the number of processing units on the hardware; The screening subunit is used to screen the initial arrangement mode set based on the memory arrangement type of the data to obtain the processing unit arrangement mode set.

[0073] Based on any of the above embodiments, the determination subunit is specifically used for: Splitting the number of processing units on the hardware on a specified dimension to obtain a split result, wherein the specified dimension is two-dimensional or three-dimensional; Based on the splitting result, an initial arrangement set of processing units on the hardware is obtained.

[0074] Based on any of the above embodiments, when the specified dimension is two-dimensional, the specified dimension includes a row dimension and a column dimension; accordingly, the screening subunit is specifically used for: When the memory arrangement type of the data is row-major order, an arrangement method in which the value of the column dimension is greater than or equal to the value of the row dimension is selected from the initial arrangement method set to obtain the processing unit arrangement method set; When the memory arrangement type of the data is column-major order, arrangement methods whose row dimension values ​​are greater than or equal to column dimension values ​​are filtered out from the initial arrangement method set to obtain the processing unit arrangement method set.

[0075] Based on any of the above embodiments, the access unit 630 is specifically used for: Determining, based on the data block dimension of the processing unit, the data dimension corresponding to each arrangement mode in the processing unit arrangement mode set; Compare the data dimensions corresponding to the various arrangements with the shape of the data, and determine a candidate arrangement from the various arrangements; When there is one candidate arrangement, the candidate arrangement is determined as the target arrangement.

[0076] Based on any of the foregoing embodiments, the access unit 630 is further configured to: In the case where there are at least two candidate arrangements, the target arrangement is determined from the candidate arrangements based on the difference between the data dimension corresponding to each candidate arrangement and the shape of the data.

[0077] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the data access optimization method, which includes: obtaining the shape of the data and the memory arrangement type of the data; determining a processing unit arrangement mode set based on the memory arrangement type of the data and the number of processing units on the hardware; determining a target arrangement mode from the processing unit arrangement mode set based on the shape of the data, and applying the target arrangement mode to access the data.

[0078] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data access optimization method provided by the above methods, which method includes: obtaining the shape of the data and the memory arrangement type of the data; determining a set of processing unit arrangement methods based on the memory arrangement type of the data and the number of processing units on the hardware; determining a target arrangement method from the set of processing unit arrangement methods based on the shape of the data, and applying the target arrangement method to access the data.

[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the data access optimization method provided by the above-mentioned methods, the method comprising: obtaining the shape of the data and the memory arrangement type of the data; determining a set of processing unit arrangement methods based on the memory arrangement type of the data and the number of processing units on the hardware; determining a target arrangement method from the set of processing unit arrangement methods based on the shape of the data, and applying the target arrangement method to access the data.

[0081] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data access optimization method, characterized in that: include: Obtaining the shape of data and the memory arrangement type of the data; Determine a set of processing unit arrangement modes based on the memory arrangement type of the data and the number of processing units on the hardware; Based on the shape of the data, a target arrangement is determined from the set of processing unit arrangement modes, and the target arrangement is applied to access the data.

2. The data access optimization method according to claim 1, characterized in that: The determining of a processing unit arrangement mode set based on the memory arrangement type of the data and the number of processing units on the hardware includes: Based on the number of processing units on the hardware, determining a set of initial arrangements of the processing units on the hardware; Based on the memory arrangement type of the data, the initial arrangement mode set is screened to obtain the processing unit arrangement mode set.

3. The data access optimization method according to claim 2, characterized in that: The determining, based on the number of processing units on the hardware, a set of initial arrangement modes of the processing units on the hardware includes: Splitting the number of processing units on the hardware on a specified dimension to obtain a split result, wherein the specified dimension is two-dimensional or three-dimensional; Based on the splitting result, an initial arrangement set of processing units on the hardware is obtained.

4. The data access optimization method according to claim 3, characterized in that: In the case where the specified dimension is two-dimensional, the specified dimension includes a row dimension and a column dimension; The filtering of the initial arrangement mode set based on the memory arrangement type of the data to obtain the processing unit arrangement mode set includes: When the memory arrangement type of the data is row-major order, an arrangement method in which the value of the column dimension is greater than or equal to the value of the row dimension is selected from the initial arrangement method set to obtain the processing unit arrangement method set; When the memory arrangement type of the data is column-major order, arrangement methods whose row dimension values ​​are greater than or equal to column dimension values ​​are filtered out from the initial arrangement method set to obtain the processing unit arrangement method set.

5. The data access optimization method according to any one of claims 1 to 4, characterized in that: The step of determining a target arrangement from the set of processing unit arrangement modes based on the shape of the data includes: Determining, based on the data block dimension of the processing unit, the data dimension corresponding to each arrangement mode in the processing unit arrangement mode set; Compare the data dimensions corresponding to the various arrangements with the shape of the data, and determine a candidate arrangement from the various arrangements; When there is one candidate arrangement, the candidate arrangement is determined as the target arrangement.

6. The data access optimization method according to claim 5, characterized in that: The step of determining a candidate arrangement from the arrangement modes further comprises: In the case where there are at least two candidate arrangements, the target arrangement is determined from the candidate arrangements based on the difference between the data dimension corresponding to each candidate arrangement and the shape of the data.

7. A data access optimization device, characterized in that: include: An acquisition unit, used to acquire the shape of data and the memory arrangement type of the data; A determination unit, configured to determine a set of processing unit arrangement modes based on a memory arrangement type of the data and a number of processing units on the hardware; The access unit is used to determine a target arrangement from the processing unit arrangement set based on the shape of the data, and access the data using the target arrangement.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data access optimization method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data access optimization method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data access optimization method according to any one of claims 1 to 6 is implemented.

Images