A method and apparatus for optimizing convolution operations

Abstract

The application discloses a convolution operation optimization method and device, and the method comprises the following steps: generating a plurality of memory access addresses based on the first coordinates of the pixel points to be processed in the obtained output feature map and the convolution operation instruction; generating corresponding control signals according to the second coordinates of the input feature map corresponding to each memory access address; performing merging processing on each memory access address based on each control signal to generate a memory access request; and performing data extraction and data rearrangement processing on the data returned by the memory after the memory access operation based on the memory access request, so as to obtain target data. The application can realize the merging access of the memory access addresses and the efficient extraction of the effective data in the convolution calculation process, reduces the repeated memory access to the same memory address, improves the memory bandwidth utilization and the data access efficiency of the convolution operation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence processor technology, and in particular to a method and apparatus for optimizing convolution operations. Background Technology

[0002] With the development of artificial intelligence and deep learning technologies, Neural Processing Units (NPUs) are widely used in the inference and training of convolutional neural networks. Convolution, as a core computational operation in neural networks, is characterized by high computational cost, frequent data access, and high data reuse. To reduce on-chip storage resource consumption and minimize data interaction with external memory, feature map data is typically compressed for storage. Therefore, improving data retrieval efficiency and memory bandwidth utilization while ensuring the correctness of convolution calculations has become a crucial technical challenge in NPU design.

[0003] In existing technologies, convolution calculations typically generate corresponding memory access addresses based on the sliding position of the convolution kernel on the input feature map, and then read the data involved in the convolution calculation from memory. When convolution operations support parameters such as stride and insert, the data reading module usually generates memory access addresses independently according to the convolution calculation requirements, and retrieves the required data from memory in the order of convolution calculations for subsequent multiply-accumulate (MAC) operations to perform convolution calculations.

[0004] However, when feature maps are stored using compressed storage and the stride parameter is greater than 1, the distribution of convolutional sampling points in storage space is inconsistent with the convolutional computation order. Existing data reading methods typically generate memory access addresses and initiate memory access requests independently according to the convolutional computation order, making it difficult to consider the actual storage location of the data in memory. Especially in cross-line reading scenarios, data corresponding to multiple sampling points may reside in the same storage address but are still repeatedly read, leading to increased memory access frequency, reduced memory bandwidth utilization, and decreased data access efficiency.

[0005] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0006] This invention provides a convolution operation optimization method to achieve merged access of memory addresses and efficient extraction of effective data during convolution calculation, reduce repeated memory accesses to the same memory address, and improve memory bandwidth utilization and data access efficiency of convolution operation.

[0007] The optimization method for convolution operations includes: Multiple memory access addresses are generated based on the first coordinates of the pixel to be processed in the obtained output feature map and the convolution operation instructions. Based on the second coordinate of the input feature map corresponding to each memory access address, a corresponding control signal is generated; Based on the control signals, a merge process is performed on each memory access address to generate a memory access request. The data returned by the memory after performing a memory access operation based on the memory access request is extracted and rearranged to obtain the target data.

[0008] In some embodiments, generating multiple memory access addresses based on the first coordinates of the pixel to be processed in the obtained output feature map and the convolution operation instructions includes: The base address of the input feature map is determined based on the address information in the convolution operation instruction; Based on the convolution operation instruction and the first coordinate, determine multiple second coordinates corresponding to the input feature map; Multiple offset addresses are generated based on the convolution operation instructions and each of the second coordinates; Multiple memory access addresses are generated based on the offset addresses and the base addresses.

[0009] In some embodiments, the first coordinate includes a first horizontal coordinate and a first vertical coordinate, and the second coordinate includes a second horizontal coordinate and a second vertical coordinate; determining multiple second coordinates corresponding to the input feature map based on the convolution operation instruction and the first coordinate includes: Based on the offset of the convolution kernel in the width direction and the sliding step size in the convolution operation instruction, the first horizontal coordinate is mapped to multiple second horizontal coordinates in the input feature map; Based on the offset of the convolution kernel in the height direction and the sliding step size in the convolution operation instruction, the first ordinate is mapped to multiple second ordinates in the input feature map.

[0010] In some embodiments, generating multiple offset addresses based on the convolution operation instructions and each of the second coordinates includes: Based on the width of the input feature map in the convolution operation instruction, each of the second horizontal coordinates and each of the second vertical coordinates are compressed into multiple pixel coordinates; Multiple offset addresses are generated based on the pixel coordinates.

[0011] In some embodiments, the control signals include: a fill valid flag and a zero insertion valid flag; the step of performing a merge process on each memory access address based on each of the control signals to generate a memory access request includes: An address table is constructed based on each memory access address and its corresponding control signal; Determine whether the high-order bits of the memory access addresses corresponding to adjacent entries in the address table are the same; If the high-order address of the memory access address corresponding to adjacent entries is the same and the padding valid flag or the zero insertion valid flag is invalid, the memory access addresses corresponding to each entry are merged into one memory access request, and the valid data location index in each memory access address is retained.

[0012] In some embodiments, the step of extracting and rearranging data from the data returned after the memory performs a memory access operation based on the memory access request to obtain the target data includes: Generate a set of location indexes for the valid data corresponding to the storage address based on each of the valid data location indexes; Extract valid data from the data returned from the memory based on the location index set; If the bit width of the valid data is less than the preset data bit width, the valid data obtained from multiple memory access operations is concatenated through a shift register to obtain the target data with the preset data bit width.

[0013] This invention also provides a convolution operation optimization device to achieve merged access of memory addresses and efficient extraction of effective data during convolution calculation, reduce repeated memory accesses to the same memory address, and improve memory bandwidth utilization and data access efficiency of convolution operation.

[0014] The convolution operation optimization device includes: The address generation module is used to generate multiple memory access addresses based on the first coordinates of the pixel to be processed in the obtained output feature map and the convolution operation instructions. The control signal generation module is used to generate a corresponding control signal based on the second coordinate of the input feature map corresponding to each memory access address; The address merging module is used to perform merging processing on each memory access address based on each of the control signals, and generate a memory access request. The data processing module is used to extract and rearrange the data returned by the memory after performing a memory access operation based on the memory access request to obtain the target data.

[0015] This invention also provides a neural network processor, including a memory, a processor, and convolution operation instructions stored in the memory and executable on the processor. When the processor executes the convolution operation instructions, it implements the above-mentioned convolution operation optimization method.

[0016] This invention also provides a readable storage medium storing convolution operation instructions, which, when executed by a processor, implement the above-described convolution operation optimization method.

[0017] This invention also provides a convolution operation program product, which includes convolution operation instructions. When the convolution operation instructions are executed by a processor, they implement the above-described convolution operation optimization method.

[0018] The convolution operation optimization method and apparatus provided in this invention generate multiple memory access addresses based on the coordinates of the pixels to be processed in the output feature map, and perform address merging processing on the multiple memory access addresses in conjunction with the control signals corresponding to the sampling points in the input feature map. This achieves merged access to memory access requests corresponding to the same memory address, reducing the number of repeated memory accesses. Furthermore, by retaining the valid data location index and performing data extraction and rearrangement processing on the data returned from the memory, the effective data required for convolution calculation can be accurately extracted from the returned data, and the discretely distributed effective data can be reorganized. This method improves the effective utilization rate of data returned from a single memory access, reduces the bandwidth overhead caused by invalid data transmission, and reduces the number of memory accesses, thereby improving memory bandwidth utilization and data access efficiency during convolution operations. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart illustrating a method for optimizing convolution operations in one embodiment of the present invention; Figure 2 This is a flowchart illustrating the convolution operation optimization method in another embodiment of the present invention; Figure 3 This is a schematic diagram of the address generator in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the convolution operation optimization method in another embodiment of the present invention; Figure 5 This is a flowchart illustrating the memory access address determination method in an embodiment of the present invention; Figure 6 This is a flowchart illustrating the convolution operation optimization method in another embodiment of the present invention; Figure 7 This is a schematic diagram of the address table in an embodiment of the present invention; Figure 8 This is a flowchart illustrating the convolution operation optimization method in another embodiment of the present invention; Figure 9 This is a schematic diagram of the address merging process in an embodiment of the present invention; Figure 10 This is a flowchart illustrating the convolution operation optimization method in another embodiment of the present invention; Figure 11 This is a schematic diagram of the effective data extraction and rearrangement process in an embodiment of the present invention; Figure 12 This is a schematic diagram of the data flow and control flow in an embodiment of the present invention; Figure 13 This is a schematic diagram of the structure of the convolution operation optimization device in an embodiment of the present invention; Figure 14 This is a schematic diagram of the physical structure of a neural network processor provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with relevant laws and regulations. The user information in the embodiments of this application is obtained through legal and compliant means, and the acquisition, storage, use, and processing of user information have been authorized and agreed upon by the customer.

[0021] To facilitate understanding of the technical solution provided in this application, the relevant content of the technical solution in this application will be explained below.

[0022] To improve data access efficiency and memory bandwidth utilization during convolution operations, this application provides an optimized convolution operation method. This method generates multiple memory access addresses corresponding to the input feature map based on the coordinates of the pixels to be processed in the output feature map. Multiple memory access addresses accessing the same memory address are merged according to the control signals corresponding to those addresses, generating a memory access request, while simultaneously recording the valid data location index. After the memory completes the access and returns data, valid data is extracted from the returned data based on the valid data location index, and the extracted valid data is then rearranged and concatenated. By merging memory access requests corresponding to the same memory address and filtering and organizing the returned data based on the valid data location index, the bandwidth overhead caused by repeated memory accesses is reduced, the memory bandwidth occupied by invalid data transmission is decreased, and memory bandwidth utilization is improved, thereby enhancing data access efficiency during convolution operations.

[0023] This invention provides an optimization method for convolution operations, such as... Figure 1 As shown, the method includes S101 to S104.

[0024] S101: Generate multiple memory access addresses based on the first coordinates of the pixel to be processed in the obtained output feature map and the convolution operation instructions.

[0025] S102: Generate the corresponding control signal based on the second coordinate of the input feature map corresponding to each memory access address.

[0026] S103: Perform merge processing on each memory access address based on each control signal to generate a memory access request.

[0027] S104: Extract and rearrange the data returned by the memory after performing a memory access operation based on the memory access request to obtain the target data.

[0028] According to the above embodiments, multiple memory access addresses are generated based on the coordinates of the pixels to be processed in the output feature map. Then, address merging is performed on these multiple memory access addresses using control signals corresponding to sampling points in the input feature map. This achieves merged access to memory requests corresponding to the same memory address, reducing the number of repeated memory accesses. Furthermore, by retaining the valid data location index and performing data extraction and rearrangement on the data returned from the memory, the effective data required for convolution calculation can be accurately extracted from the returned data, and the discretely distributed effective data can be reorganized. This method improves the effective utilization rate of data returned from a single memory access, reduces the bandwidth overhead caused by invalid data transmission, and reduces the number of memory accesses, thereby improving memory bandwidth utilization and data access efficiency during convolution operations.

[0029] In some embodiments, such as Figure 2 As shown, S101 includes S201 to S204.

[0030] S201: Determine the base address base_addr of the input feature map based on the address information in the convolution operation instruction.

[0031] S202: Based on the convolution operation instruction and the first coordinate, determine multiple second coordinates corresponding to the input feature map.

[0032] S203: Generate multiple offset addresses based on the convolution operation instructions and each second coordinate. .

[0033] S204: Based on each offset address Multiple memory access addresses are generated from the base address. .

[0034] In embodiments of the present invention, such as Figure 3 As shown, the address generator uses the coordinates of the pixels to be processed in the obtained output feature map. The convolution operation instruction generates multiple corresponding memory access addresses. The address generator includes an output width register (bias_wo), an output height register (bias_ho), and a base address generation module (bias_addr). The output width register, output height register, and base address generation module are all within the same clock cycle.

[0035] The output width register receives the x-coordinate of the pixel to be processed in the output feature map transmitted from upstream. and the x-axis The data is stored. The output height register receives the ordinate of the pixel to be processed from the output feature map transmitted upstream. and the vertical axis Store it.

[0036] The base address generation module parses the input feature map address information in the convolution instruction and calculates the base address of the input feature map in memory based on this address information. The base address represents the starting storage location of the input feature map in memory. The memory includes: Double Data Rate Synchronous Dynamic Random Access Memory (DDR) and / or Static Random Access Memory (SRAM).

[0037] According to the above embodiments, by mapping the coordinates of the pixels to be processed in the output feature map to the coordinates, offset addresses, and memory access addresses of the corresponding sampling points in the input feature map step by step, the conversion process from the convolution computation logic space to the memory address space is realized. By determining the base address of the input feature map and generating the memory access address in combination with the offset address corresponding to the sampling point, the storage location of the data required for convolution computation in memory can be accurately located. At the same time, by generating corresponding memory access addresses for multiple sampling points within the coverage area of ​​the convolution kernel, a foundation is provided for subsequent address table construction, memory access request merging, and data reading. The above method establishes a mapping relationship between the coordinates of the output feature map, the coordinates of the input feature map sampling points, and the memory access address, improving the accuracy and standardization of memory access address generation, and providing support for subsequent memory bandwidth utilization optimization and improved data access efficiency for convolution computation.

[0038] In some embodiments, the first coordinate includes: a first horizontal coordinate. and the first ordinate The second coordinate includes: the second x-coordinate. and the second ordinate .like Figure 4 As shown, S202 includes S401 to S402.

[0039] S401: Based on the offset of the convolution kernel in the width direction in the convolution operation instruction. and sliding step size , set the first x-coordinate Mapped to multiple second x-coordinates in the input feature map .

[0040] S402: Based on the offset of the convolution kernel in the height direction in the convolution operation instruction. and sliding step size , set the first ordinate Mapped to multiple second ordinates in the input feature map .

[0041] In embodiments of the present invention, such as Figure 3 As shown, the address generator also includes an input width coordinate mapping module (bias_wi) and an input height coordinate mapping module (bias_hi). The input width coordinate mapping module and the input height coordinate mapping module operate within the same clock cycle.

[0042] like Figure 5 As shown, the input width coordinate mapping module offsets the kernel width direction in the convolution operation instruction. Width direction sliding step size Void coefficient in width direction and the x-coordinate of the pixel to be processed in the output feature map Substitute into the following formula (1) to calculate the x-coordinate of the sampling point in the input feature map. .

[0043]

[0044] in, This represents the x-coordinate of the sampling point in the input feature map. This represents the x-coordinate of the pixel to be processed in the output feature map. This represents the stride of the convolution kernel in the width direction. This represents the offset of the convolution kernel in the width direction (Kernel Width Offset). This represents the dilation coefficient of the convolution kernel in the width direction.

[0045] Similarly, the input height coordinate mapping module will offset the height direction of the convolution kernel in the convolution operation instruction. Sliding step size in the height direction Void coefficient in height direction and the ordinate of the pixel to be processed in the output feature map Substitute into the following formula (2) to calculate the ordinate of the sampling point in the input feature map. .

[0046]

[0047] in, This represents the ordinate of the sampling point in the input feature map. This represents the ordinate of the pixel to be processed in the output feature map. This represents the stride of the convolution kernel in the height direction. This represents the offset of the convolution kernel in the height direction. This represents the void factor in the height direction.

[0048] For a pixel to be processed in the output feature map, the corresponding convolutional kernel coverage area contains multiple sampling points. The input width coordinate mapping module and the input height coordinate mapping module are based on the width direction offset corresponding to different sampling positions of the convolutional kernel, respectively. and height direction offset Calculate the coordinates of each sampling point within the coverage area of ​​the convolution kernel. .

[0049] According to the above embodiments, by mapping the coordinates of the pixels to be processed in the output feature map to the coordinates of multiple corresponding sampling points in the input feature map, the coordinate association between the output and input feature maps is established. For each sampling position within the convolution kernel coverage area, the horizontal and vertical coordinates of the corresponding sampling points are determined by combining the convolution kernel offset and the stride, thereby accurately obtaining the coordinates of multiple sampling points required for convolution calculation. The above method can adapt to convolution operation scenarios with different convolution kernel sizes and different stride parameters, providing an accurate coordinate basis for subsequent pixel coordinate generation, memory access address generation, and convolution data reading, avoiding data reading errors caused by inaccurate coordinate mapping, and improving the accuracy and reliability of data access during convolution calculation.

[0050] In some embodiments, such as Figure 6 As shown, S203 includes S601 to S602.

[0051] S601: Based on the width of the input feature map in the convolution operation instruction , set each second x-coordinate and each second ordinate Compressed into multiple pixel coordinates .

[0052] S602: Based on the coordinates of each pixel Generate multiple offset addresses .

[0053] In embodiments of the present invention, such as Figure 3 As shown, the address generator also includes a pixel coordinate generation module (pixel_i) and a pixel coordinate mapping module (addr_pi).

[0054] like Figure 6 As shown, the pixel coordinate generation module takes the width of the input feature map from the convolution operation instruction. 1. Input the x-coordinates of each sampling point in the feature map and ordinate Substitute into the following formula (3) to calculate the corresponding pixel coordinates. .

[0055]

[0056] in, This indicates the width of the input feature map. This represents the ordinate of the sampling point in the input feature map. This represents the x-coordinate of the sampling point in the input feature map. This represents the one-dimensional pixel coordinates (Pixel Index) corresponding to the sampling point.

[0057] The input features are established using the pixel coordinate generation module based on the above formula (3). Figure 2 The mapping relationship between 2D coordinate space and 1D storage space, which corresponds to the 2D coordinates of each sampling point. Convert to corresponding one-dimensional pixel coordinates .

[0058] The pixel coordinate mapping module maps the one-dimensional pixel coordinates according to the data storage format and memory organization of the input feature map. Mapped to the corresponding offset address Among them, the offset address This represents the address offset of the corresponding sampling point relative to the base address of the input feature map.

[0059] According to the above embodiments, by converting the two-dimensional coordinates corresponding to each sampling point in the input feature map into one-dimensional pixel coordinates, a unified representation between the logical coordinate space and storage space of the input feature map is achieved, facilitating subsequent storage address calculation and management. Based on this, corresponding offset addresses are generated according to the one-dimensional pixel coordinates, enabling each sampling point to be quickly mapped to its actual storage location in memory during convolution calculation, providing a foundation for subsequent memory access address generation. This method avoids repeated two-dimensional coordinate calculations during address generation, reducing address computation complexity and improving the efficiency of coordinate-to-storage address conversion. This, in turn, improves data access efficiency during convolution calculation and supports subsequent memory access request merging and bandwidth utilization optimization.

[0060] In some embodiments, such as Figure 3 As shown, the address generator also includes a memory access address generation module (Addr).

[0061] The memory access address generation module generates the corresponding memory access address based on the offset address corresponding to each sampling point in the input feature map and the base address of the input feature map in memory.

[0062] Specifically, the memory access address generation module will input the offset address corresponding to each sampling point in the feature map. and the base address of the input feature map in memory Substitute into the following formula (4) to calculate the corresponding memory access address. .

[0063]

[0064] in, This represents the offset address corresponding to each sampling point in the input feature map, used to characterize the address offset of the corresponding sampling point relative to the starting storage location of the input feature map; The base address represents the input feature map in memory, and is used to characterize the starting storage location of the input feature map in memory; The memory access address represents the actual storage address of the data corresponding to the sampling point in memory.

[0065] According to the above embodiment, the memory access address generation module converts the offset address corresponding to the logical coordinates of the input feature map into the actual access address in memory, thereby establishing a mapping relationship between the sampling points of the input feature map and the physical storage location in memory. For multiple sampling points within the coverage area of ​​the convolution kernel, the memory access address generation module calculates the corresponding memory access address for each, thereby generating multiple memory access addresses. Subsequently, an address table is constructed based on the multiple memory access addresses, and memory access request merging and memory access operations are further performed.

[0066] In some embodiments, based on each memory access address Corresponding input feature map sampling point coordinates Determine the type of the corresponding sampling point and generate the corresponding control signal.

[0067] Specifically, based on the coordinates of the sampling points The positional relationship between the sampled point and the valid region, padding region, and insert region of the input feature map is used to determine whether the corresponding sampled point belongs to a valid data point, a padding data point, or a insert data point. Specifically, if the sampled point is located within the valid data region of the input feature map, it is determined to be a valid data point. If the sampled point is located within the padding region generated by the convolution operation, it is determined to be a padding data point. If the sampled point is located within the insert region, it is determined to be a insert data point.

[0068] After determining the sampling point type, a corresponding control signal is generated. This control signal includes at least a padding valid flag (pad_vld) and an insertion valid flag (insrt_vld). Specifically, when the sampling point is a padding data point, the padding valid flag is set to valid. When the sampling point is a insertion data point, the insertion valid flag is set to valid. When the sampling point is a valid data point, both the padding valid flag and the insertion valid flag are set to invalid. Subsequent modules then perform memory access request filtering and memory access request merging processing based on the control signals.

[0069] In some embodiments, such as Figure 8 As shown, S103 includes S801 to S803.

[0070] S801: Constructs an address table based on each memory access address and its corresponding control signal.

[0071] S802: Determine whether the high-order address of the memory access address corresponding to adjacent entries in the address table is the same.

[0072] S803: If the high-order address of the memory access address corresponding to adjacent entries is the same and the padding valid flag or the zero insertion valid flag is invalid, merge the memory access addresses corresponding to each entry into one memory access request, and retain the valid data location index in each memory access address.

[0073] In this embodiment of the invention, multiple memory access addresses generated in parallel and their corresponding control signals are written into the same address table for unified management. For example... Figure 7As shown, the address table includes multiple entries, each corresponding to a memory access address and its corresponding control signal. The control signal further includes an entry validity flag (table_vld) and an address validity flag (addr_vld). The entry validity flag indicates whether the corresponding entry is valid, and the address validity flag indicates whether the corresponding memory access address participates in subsequent memory access processing.

[0074] Based on the address mapping rules corresponding to the memory bit width, it is determined whether the high-order addresses of adjacent entries in the address table are the same. The high-order address refers to the address field in the memory access address excluding the preset low-order address. In this embodiment, this is determined by comparing the high-order addresses of adjacent entries. and Whether they are the same determines whether the corresponding sampling points are located in the same storage address.

[0075] when and When they are the same, determine and The memory access address points to the same memory address, and... and The memory access addresses are merged, and a memory access request is generated based on the merged memory access addresses. The memory then executes the corresponding memory access operation based on the memory access request to retrieve the data at the memory address. Simultaneously, a valid data location index (Map) is maintained for each memory access address; this valid data location index corresponds to the memory access address... This field is used to identify the location of valid data in the data returned from memory.

[0076] When the padding valid flag bit of the corresponding entry is valid (i.e., pad_vld=1) or the zero insertion valid flag bit is valid (i.e., insrt_vld=1), it is determined that the corresponding sampling point belongs to the padding region or the zero insertion region, and the memory does not need to perform a memory access operation.

[0077] For example, such as Figure 9 As shown, the address table contains 16 entries, corresponding to the memory access addresses generated from the 16 sampling points. After performing address merging on the address table, a memory access request is generated based on the high-order address of each entry and the control signal.

[0078] Among them, the four sampling points corresponding to item0 to item3 have the same high-order address, that is, they correspond to the same memory address. Therefore, item0 to item3 are merged into a single memory access request CC0, and the corresponding number of valid data is recorded. The pixel_num parameter represents the number of valid sampling points corresponding to the current memory access request.

[0079] The sampling points corresponding to item4 and item5 are located in the filled area, and their fill validity flag is valid. Therefore, no corresponding memory access request is generated; only the number of valid data items is recorded. .

[0080] The eight sampling points corresponding to item6 to item13 have the same high-order address, meaning they correspond to the same memory address. Therefore, item6 to item13 are grouped into a single memory access request CC2, and the corresponding number of valid data points is recorded. .

[0081] The sampling points corresponding to item14 and item15 are also located in the filled area, and their fill validity flag is valid. Therefore, no corresponding memory access request is generated; only the number of valid data items is recorded. .

[0082] After the above processing, the 16 memory access addresses in the original address table are merged into 2 memory access requests. For multiple sampling points located within the same memory address, only one memory access request is generated. For sampling points located in filled or zero-inserted regions, no corresponding memory access request is generated. This method reduces the number of repeated accesses to the same memory address and lowers the memory access overhead for invalid data, thereby improving memory bandwidth utilization.

[0083] According to the above embodiments, by comparing the high-order addresses of adjacent entries in the address table and merging multiple memory access addresses pointing to the same memory address into a single memory access request, a memory access method that performs a single memory access for multiple sampling points is achieved. While retaining the valid data location index corresponding to each memory access address, this ensures accurate extraction of valid data corresponding to each sampling point and reduces the number of repeated accesses to the same memory address. This method reduces the number of memory access requests and data transfer overhead, improves the utilization rate of data returned from a single memory access, thereby improving memory bandwidth utilization and enhancing data access efficiency during convolution operations.

[0084] In some embodiments, such as Figure 10 As shown, S104 includes S1001 to S1003.

[0085] S1001: Generate a set of location indexes of valid data corresponding to each storage address based on each valid data location index.

[0086] S1002: Extract valid data from the data returned from the memory based on the location index set.

[0087] S1003: If the width of the valid data is less than the preset data width, the valid data obtained from multiple memory access operations is concatenated through the shift register to obtain the target data with the preset data width.

[0088] In embodiments of the present invention, such as Figure 11 As shown, after merging multiple memory access addresses, the valid data location index (Map) corresponding to each memory access address is retained. This valid data location index is generated from a portion of the low-order bits of the memory access address. In this embodiment, the valid data location index corresponds to the memory access address... This is used to characterize the position of the corresponding valid data in the memory returned data.

[0089] Based on multiple valid data location indices corresponding to the same storage address, a set of valid data location indices (Map Set) for the corresponding storage address is generated and stored for use in the subsequent data extraction process.

[0090] After the memory completes the memory access operation and returns the data at the corresponding memory address, the set of valid data location indices is time-aligned with the returned data. Based on the set of valid data location indices, the valid data corresponding to each sampling point is extracted from the returned data through a shift register, and data alignment is performed on the extracted valid data so that the valid data is arranged continuously in a preset order to form aligned data.

[0091] like Figure 11 As shown, the shaded area represents the valid data location index information of the record. Step 1 represents the generation of the corresponding valid data location index set based on the memory access address. Step 2 represents multiple sets of valid data after valid data extraction and data rearrangement.

[0092] After the data rearrangement is completed, the aligned data is sent to a shift register for buffering. The output bit width of the shift register corresponds to the memory access bandwidth, so that valid data obtained from multiple memory accesses can be concatenated in subsequent processing.

[0093] Furthermore, when the effective data bit width obtained in a single memory access is less than the preset data bit width, the effective data obtained from multiple memory access operations is cached and concatenated using a shift register to form a data block that meets the preset data bit width.

[0094] Specifically, after extracting and rearranging the valid data, the aligned valid data is written into the shift register. If the total width of the valid data in the current shift register is less than the preset data width (e.g., 2048 bits), the system continues to receive valid data returned from subsequent memory access operations and rearranged, and then concatenates it with the currently stored valid data. Valid data written to the shift register first is located in the lower bit position, and valid data written later is located in the higher bit position, thus completing data organization according to the memory access return order.

[0095] like Figure 11 As shown, Step 3 represents the data block formed by concatenating the multiple sets of valid data using a shift register. This method allows discrete valid data obtained from multiple memory access operations to be organized into continuous data blocks, adapting to the data bit width requirements of subsequent modules.

[0096] When the total width of valid data stored in the shift register reaches the preset data width, or when memory access is complete, the data in the shift register is output to the subsequent buffer for storage. Otherwise, valid data returned by subsequent memory access operations continues to be received, and after being combined with the preset data width (2048 bits) through the shift register, it is sent to the subsequent buffer for buffering until the output condition is met.

[0097] For example, the two data entries in Step 2 are concatenated to obtain the target data in Step 3. The shaded area represents invalid data, and the white area represents valid data. When the shift register is full or a memory access is complete, the data in the shift register is sent to the next level; otherwise, the process continues to wait.

[0098] According to this embodiment, by recording the position index of each valid data point in the memory return data and extracting valid data from the return data based on the set of position indices, rapid separation of valid and invalid data in the memory return data is achieved, avoiding invalid data from participating in subsequent processing and improving the utilization rate of the return data. Furthermore, by concatenating the valid data obtained from multiple memory accesses, the discretely distributed valid data is organized into data blocks that meet the preset data bit width, thereby adapting to the data input requirements of subsequent cache modules or computing modules. The above method not only improves the effective utilization rate of the return data from a single memory access but also reduces the data transmission overhead caused by data bit width mismatch, which is beneficial for improving memory bandwidth utilization and data access efficiency during convolution operations.

[0099] In one embodiment, such as Figure 12As shown, the above convolution operation optimization method includes two main paths: control flow and data flow. The control flow is used to generate multiple memory access addresses and their corresponding control signals (such as...) in parallel based on the coordinates of the pixels to be processed in the output feature map and the convolution operation instructions. Figure 12 (As shown in ①), and then cache the generated memory access addresses and control signals into the input / output queue (FIFO) and summarize them to form an address table (such as... Figure 12 (As shown in note ②). Subsequently, the memory access addresses in the address table are merged, multiple memory access requests pointing to the same memory address are combined, while the valid data location index (Map) is retained, and the generated memory access requests are cached in the input / output queue.

[0100] The data stream is used to process the data returned from memory. After the memory access operation is completed, the valid data corresponding to each sampling point is extracted from the returned data according to the valid data location index (Map) generated by the control flow. The extracted valid data is then processed by data alignment, arranging the valid data consecutively to form aligned data blocks. Subsequently, the aligned target data is output to the subsequent data buffer module for use by the subsequent convolution calculation unit (MAC).

[0101] According to the above embodiments, by merging the memory access addresses generated during convolution operations and extracting and rearranging the data returned from memory using the effective data location index, a unified data reading mechanism is achieved, which can adapt to data access scenarios corresponding to different time-length parameters. This technical solution can still continuously acquire the data required for convolution calculations even when the number of input channels is not aligned with the maximum processing specification, reducing the decrease in data reading efficiency caused by feature map wrapping. Simultaneously, by merging multiple memory access requests located at the same memory address and extracting effective data from the returned data for subsequent computation units, the effective utilization rate of memory returned data and the utilization rate of multiply-accumulate operation units are improved. By reducing the number of repeated accesses to the same memory address, memory access overhead and power consumption are reduced, thereby improving data access efficiency and overall computational performance during convolution operations.

[0102] This application provides a convolution operation optimization device applied to the aforementioned convolution operation optimization method. This convolution operation optimization device and the convolution operation optimization method in one embodiment of this application are based on the same inventive concept and have similar problem-solving principles. Therefore, the implementation of the convolution operation optimization device is the same as the implementation of the convolution operation optimization method in one embodiment of this application, and repeated details will not be described again. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0103] like Figure 13 As shown, the present invention provides a convolution operation optimization device comprising: Address generation module 1301 is used to generate multiple memory access addresses based on the first coordinates of the pixel to be processed in the obtained output feature map and the convolution operation instructions. The control signal generation module 1302 is used to generate corresponding control signals based on the second coordinates of the input feature map corresponding to each memory access address. Address merging module 1303 is used to perform merging processing on each memory access address based on each control signal and generate a memory access request. The data processing module 1304 is used to extract and rearrange the data returned by the memory after the memory performs a memory access operation based on the memory access request, so as to obtain the target data.

[0104] In some embodiments, the address generation module 1301 includes: The base address determination submodule is used to determine the base address of the input feature map based on the address information in the convolution operation instruction; The coordinate mapping submodule is used to determine multiple second coordinates corresponding to the input feature map based on the convolution operation instructions and the first coordinates; The offset address generation submodule is used to generate multiple offset addresses based on the convolution operation instructions and each second coordinate. The memory access address generation submodule is used to generate multiple memory access addresses based on each offset address and the base address.

[0105] In some embodiments, the first coordinate includes a first abscissa and a first ordinate, and the second coordinate includes a second abscissa and a second ordinate; the coordinate mapping submodule includes: The first coordinate mapping unit is used to map the first horizontal coordinate into multiple second horizontal coordinates in the input feature map according to the offset of the convolution kernel in the width direction and the sliding step size in the convolution operation instruction. The second coordinate mapping unit is used to map the first ordinate to multiple second ordinates in the input feature map according to the offset of the convolution kernel in the height direction and the sliding step size in the convolution operation instruction.

[0106] In some embodiments, the offset address generation submodule includes: The pixel coordinate generation unit is used to compress each second horizontal coordinate and each second vertical coordinate into multiple pixel coordinates based on the width of the input feature map in the convolution operation instruction. Offset address mapping unit, used to generate multiple offset addresses based on the coordinates of each pixel.

[0107] In some embodiments, the control signal includes: a fill valid flag bit and a zero insertion valid flag bit.

[0108] Address merging module 1303 includes: The address table construction submodule is used to construct an address table based on each memory access address and its corresponding control signal. The address comparison submodule is used to determine whether the high-order bits of the memory access addresses corresponding to adjacent entries in the address table are the same. The memory access request generation submodule is used to merge the memory access addresses corresponding to each table entry into a single memory access request if the high-order address of the memory access address corresponding to adjacent table entries is the same and the padding valid flag or the zero insertion valid flag is invalid, and to retain the valid data location index in each memory access address.

[0109] In some embodiments, the data processing module 1304 includes: The index set generation submodule is used to generate a set of location indexes for valid data at corresponding storage addresses based on each valid data location index. The valid data extraction submodule is used to extract valid data from the data returned from the memory based on the set of location indices; The target data generation submodule is used to concatenate the valid data obtained from multiple memory access operations through a shift register if the bit width of the valid data is less than the preset data bit width, so as to obtain the target data with the preset data bit width.

[0110] Figure 14 This is a schematic diagram of the physical structure of a neural processing unit (NPU) provided in an embodiment of the present invention, as shown below. Figure 14 As shown, the neural network processor includes: processor 1401, memory 1402 and bus 1403.

[0111] The processor 1401 and the memory 1402 communicate with each other via the bus 1403.

[0112] The processor 1401 is used to call program instructions in the memory 1402 to execute the methods provided in the above-described method embodiments.

[0113] This invention also provides a readable storage medium storing convolution operation instructions, which, when executed by a processor, implement the above-described convolution operation optimization method.

[0114] This invention also provides a convolution operation program product, which includes convolution operation instructions. When the convolution operation instructions are executed by a processor, they implement the above-described convolution operation optimization method.

[0115] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a program product implemented on one or more available storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing convolution operation instructions.

[0116] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by convolution operation instructions. The convolution operation instructions can be provided to a processor of a graphics processor, artificial intelligence accelerator, central processing unit, neural network processor, digital signal processor, tensor processor, or other convolution operation processing device for execution to produce a convolution operation machine, such that the convolution operation instructions executed by the processor produce methods for implementing convolution operations in the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These convolution operation instructions may also be stored in a readable storage medium that can guide or other neural network processing devices to operate in a specific manner, such that the convolution operation instructions stored in the readable storage medium produce an article of manufacture including an instruction means, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These convolution operation instructions can also be loaded onto other neural network processing devices, causing a series of operational steps to be performed on the other neural network processing device to produce the implemented processing, thereby providing the convolution operation instructions executed on the other neural network processing device for implementation in the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0120] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing convolution operations, characterized in that, include: Multiple memory access addresses are generated based on the first coordinates of the pixel to be processed in the obtained output feature map and the convolution operation instructions. Based on the second coordinate of the input feature map corresponding to each memory access address, a corresponding control signal is generated; Based on the control signals, a merge process is performed on each memory access address to generate a memory access request. The data returned by the memory after performing a memory access operation based on the memory access request is extracted and rearranged to obtain the target data.

2. The method according to claim 1, characterized in that, The process generates multiple memory access addresses based on the first coordinates of the pixel to be processed in the obtained output feature map and the convolution operation instructions, including: The base address of the input feature map is determined based on the address information in the convolution operation instruction; Based on the convolution operation instruction and the first coordinate, determine multiple second coordinates corresponding to the input feature map; Multiple offset addresses are generated based on the convolution operation instructions and each of the second coordinates; Multiple memory access addresses are generated based on the offset addresses and the base addresses.

3. The method according to claim 2, characterized in that, The first coordinate includes a first horizontal coordinate and a first vertical coordinate, and the second coordinate includes a second horizontal coordinate and a second vertical coordinate; determining multiple second coordinates corresponding to the input feature map based on the convolution operation instruction and the first coordinate includes: Based on the offset of the convolution kernel in the width direction and the sliding step size in the convolution operation instruction, the first horizontal coordinate is mapped to multiple second horizontal coordinates in the input feature map; Based on the offset of the convolution kernel in the height direction and the sliding step size in the convolution operation instruction, the first ordinate is mapped to multiple second ordinates in the input feature map.

4. The method according to claim 3, characterized in that, The step of generating multiple offset addresses based on the convolution operation instructions and each of the second coordinates includes: Based on the width of the input feature map in the convolution operation instruction, each of the second horizontal coordinates and each of the second vertical coordinates are compressed into multiple pixel coordinates; Multiple offset addresses are generated based on the pixel coordinates.

5. The method according to claim 1, characterized in that, The control signals include: a fill valid flag and a zero insertion valid flag; the step of performing a merge process on each memory access address based on each of the control signals to generate a memory access request includes: An address table is constructed based on each memory access address and its corresponding control signal; Determine whether the high-order bits of the memory access addresses corresponding to adjacent entries in the address table are the same; If the high-order address of the memory access address corresponding to adjacent entries is the same and the padding valid flag or the zero insertion valid flag is invalid, the memory access addresses corresponding to each entry are merged into one memory access request, and the valid data location index in each memory access address is retained.

6. The method according to claim 5, characterized in that, The step of extracting and rearranging data from the data returned after the memory performs a memory access operation based on the memory access request to obtain the target data includes: Generate a set of location indexes for the valid data corresponding to the storage address based on each of the valid data location indexes; Extract valid data from the data returned from the memory based on the location index set; If the bit width of the valid data is less than the preset data bit width, the valid data obtained from multiple memory access operations is concatenated through a shift register to obtain the target data with the preset data bit width.

7. A convolution operation optimization device, characterized in that, include: The address generation module is used to generate multiple memory access addresses based on the first coordinates of the pixel to be processed in the obtained output feature map and the convolution operation instructions. The control signal generation module is used to generate corresponding control signals based on the second coordinates of the input feature map corresponding to each memory access address. The address merging module is used to perform merging processing on each memory access address based on each of the control signals, and generate a memory access request. The data processing module is used to extract and rearrange the data returned by the memory after performing a memory access operation based on the memory access request to obtain the target data.

8. A neural network processor, comprising a memory, a processor, and convolution operation instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the convolution operation instruction, it implements the method of any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores convolution operation instructions, which, when executed by a processor, implement the method of any one of claims 1 to 6.

10. A convolution operation program product, characterized in that, The convolution operation program product includes convolution operation instructions, which, when executed by a processor, implement the method of any one of claims 1 to 6.

Images