Data multiplexing method and device of convolutional neural network, electronic equipment and medium

Abstract

Embodiments of the present application provide a data multiplexing method and device of a convolutional neural network, an electronic device and a medium. The method comprises obtaining network layer parameters and hardware constraint parameters of multiple convolutional layers of a convolutional neural model, generating multiple candidate block schemes, obtaining, according to the network layer parameters and the multiple candidate block schemes, the number of off-chip memory accesses of input data, the number of off-chip memory accesses of weight data and the number of off-chip memory accesses of output data of the multiple candidate block schemes in the multiple convolutional layers, respectively, determining the target block scheme of the multiple convolutional layers and the corresponding target multiplexing strategy from the multiple candidate block schemes, respectively, and performing data multiplexing of the multiple convolutional layers according to the target multiplexing strategy and the target block scheme. The method realizes the selection of the optimal block scheme and the optimal multiplexing strategy for each convolutional layer, avoids the problem that the existing technology is misjudged as needing to repeatedly transport data and generates redundant memory accesses due to a fixed multiplexing strategy in a scenario where cache resources are abundant, and thus saves energy consumption through data multiplexing.

Description

Technical Field

[0001] This invention relates to the technical field of data multiplexing in convolutional neural networks, and in particular to a data multiplexing method, apparatus, electronic device, and readable storage medium for convolutional neural networks. Background Technology

[0002] With the rapid development of artificial intelligence technology, Convolutional Neural Networks (CNNs) have been widely used in fields such as image recognition and natural language processing. Their model size and computational complexity continue to grow, placing higher demands on the computing power and energy efficiency of hardware acceleration platforms. In-memory arrays, as a novel computing architecture, significantly improve computational energy efficiency by reducing data transfer overhead, becoming an important technical path for accelerating neural networks. However, when deploying CNNs on in-memory arrays, the choice of data reuse strategy directly determines the number of off-chip memory accesses and the overall computational energy efficiency.

[0003] In related technologies, hardware acceleration solutions typically employ fixed data reuse strategies, such as fixed weighted reuse or fixed input reuse, to uniformly schedule data flow across all convolutional layers. However, when network layer structures are diverse and hardware cache configurations are limited, fixed strategies cannot adaptively adjust to the characteristics of each convolutional layer, leading to increased off-chip memory accesses, higher computational latency, and inefficient energy consumption, making it difficult to fully leverage the energy efficiency advantages of in-memory computing arrays. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a data multiplexing method, apparatus, electronic device and readable storage medium for convolutional neural networks that overcomes or at least partially solves the above problems.

[0005] In a first aspect, embodiments of the present invention provide a data reuse method for convolutional neural networks, the method comprising: Obtain the network layer parameters and hardware constraint parameters of multiple convolutional layers in the convolutional neural model; Based on the hardware constraint parameters, multiple candidate block partitioning schemes are generated; Based on the network layer parameters, the number of times the input data is reused, the number of times the weight data is reused, and the number of times the output data is reused for each of the multiple convolutional layers are obtained respectively. Based on the network layer parameters and the multiple candidate block segmentation schemes, the number of memory accesses outside the input data slices, the number of memory accesses outside the weight data slices, and the number of memory accesses outside the output data slices of the multiple candidate block segmentation schemes in the multiple convolutional layers are obtained respectively. Based on the number of input data reuses, the number of weight data reuses, the number of output data reuses, the number of input data accesses outside the memory slice, the number of weight data accesses outside the memory slice, and the number of output data accesses outside the memory slice, the total number of outside memory accesses under the input reuse strategy, the total number of outside memory accesses under the weight reuse strategy, and the total number of outside memory accesses under the output reuse strategy of the multiple convolutional layers are output respectively. Based on the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy, the target block scheme and the corresponding target reuse strategy of the multiple convolutional layers are determined from the multiple candidate block schemes respectively. Data reuse of the multiple convolutional layers is performed according to the target reuse strategy and the target block segmentation scheme.

[0006] Optionally, the hardware constraint parameters include: input buffer capacity, weight buffer capacity, output buffer capacity, and data bit width; generating multiple candidate block partitioning schemes based on the hardware constraint parameters includes: The input feature map, weight feature map, and output feature map of the multiple convolutional layers are obtained respectively. The input feature map of each convolutional layer is divided into blocks to obtain the input blocks of the convolutional layer; the data size of the input blocks is less than or equal to the input buffer capacity of the convolutional layer. The weight feature map of each convolutional layer is divided into blocks to obtain the weight blocks of the convolutional layer; the data size of the weight blocks is less than or equal to the weight cache capacity of the convolutional layer. The output feature map of each convolutional layer is divided into blocks to obtain the output blocks of the convolutional layer; the data size of the output blocks is less than or equal to the output buffer capacity of the convolutional layer. Based on the input blocks, weight blocks, and output blocks of multiple convolutional layers, multiple candidate block segmentation schemes are generated; each candidate block segmentation scheme includes an input block, a weight block, and an output block of a convolutional layer.

[0007] Optionally, the network layer parameters include: input feature map height, input feature map width, number of input channels, number of output channels, kernel size, stride, and loading amount per pass; the step of obtaining the number of input data reuses, weight data reuses, and output data reuses for the multiple convolutional layers based on the network layer parameters includes: The number of input multiplexing times is obtained based on the kernel size, the stride, and the number of output channels, and the number of input multiplexing times is calculated according to the following formula:

[0008] The number of weight reuses is obtained based on the input feature map height, the input feature map width, the convolution kernel size, and the convolution stride. The number of weight reuses is calculated using the following formula:

[0009] The number of output multiplexing operations is obtained based on the convolution kernel size and the number of input channels, and the number of output multiplexing operations is calculated according to the following formula:

[0010] Among them, RF I n Indicates the number of times the input data of the nth convolutional layer is reused; RF w n Indicates the number of times the weight data of the nth convolutional layer is reused; RF o n K represents the number of times the output data of the nth convolutional layer is reused; n H represents the kernel size of the nth convolutional layer, M represents the number of output channels, and H represents the kernel size of the nth convolutional layer. n L represents the height of the input feature map of the nth convolutional layer. n represents the width of the input feature map of the nth convolutional layer, N represents the number of input channels, and str represents the amount of data loaded each time.

[0011] Optionally, the hardware constraint parameters further include: the maximum data volume of a single burst transmission; the step of obtaining the number of out-of-memory accesses of the input data slices, the number of out-of-memory accesses of the weight data slices, and the number of out-of-memory accesses of the output data slices of the multiple candidate block partitioning schemes at the multiple convolutional layers according to the network layer parameters and the multiple candidate block partitioning schemes includes: The number of external memory accesses for the input data slice is obtained based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume in a single burst transmission. The number of external memory accesses for the input data slice is calculated according to the following formula:

[0012] The number of off-memory accesses for the weighted data slice is obtained based on the candidate block partitioning scheme, the convolution kernel size, and the maximum data volume in a single burst transmission. The number of off-memory accesses for the weighted data slice is calculated according to the following formula:

[0013] The number of external memory accesses for the output data slice is obtained based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume in a single burst transmission. The number of external memory accesses for the output data slice is calculated according to the following formula:

[0014] Among them, MA DRAM_I MA represents the number of off-chip memory accesses for the input data.DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O Indicates the number of off-chip memory accesses for the output data; t h Indicates the number of blocks in the height direction, t l Indicates the number of blocks divided in the width direction, t n Indicates the number of blocks segmented in the input channel direction, t m Indicates the number of blocks in the output channel direction, t r Indicates the number of blocks segmented in the output height direction, t c WN represents the number of blocks in the output width direction; M represents the number of output channels; and N represents the number of input channels.

[0015] Optionally, the step of outputting the total number of out-of-chip memory accesses under the input reuse strategy, the total number of out-of-chip memory accesses under the weight reuse strategy, and the total number of out-of-chip memory accesses under the output reuse strategy for the multiple convolutional layers, based on the number of input data reuses, the number of weight data reuses, the number of output data reuses, the number of input data out-of-chip memory accesses, the number of weight data out-of-chip memory accesses, and the number of output data out-of-chip memory accesses, respectively, includes: The total number of off-chip memory accesses under the input multiplexing strategy is obtained using the following formula:

[0016] The total number of off-chip memory accesses under the weighted reuse strategy is obtained using the following formula:

[0017] The total number of off-chip memory accesses under the output multiplexing strategy is obtained using the following formula:

[0018] Among them, MA DRAM_ir MA represents the total number of off-chip memory accesses under the input multiplexing strategy; DRAM_wr Total number of off-chip memory accesses under the representation right reuse strategy; MA DRAM_or The output multiplexing strategy represents the total number of off-chip memory accesses; H represents the height of the input feature map; L represents the width of the input feature map; N represents the number of input channels; M represents the number of output channels; R represents the height of the output feature map; C represents the width of the output feature map; T l T represents the block size in the width direction of the input feature map; n Indicates the block size in the input channel direction; T m Indicates the block size in the output channel direction; T h T represents the block size in the height direction of the input feature map. r T represents the block size in the height direction of the output feature map; cIndicates the block size in the width direction of the output feature map; RF I n Indicates the number of times the input data of the nth layer is reused; RF w n Represents the number of times the weight data of the nth layer is reused; RF o n MA represents the number of times the output data of the nth layer is reused. DRAM_I MA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O This indicates the number of off-chip memory accesses for the output data.

[0019] Optionally, determining the target block scheme and corresponding target reuse strategy for the multiple convolutional layers from the multiple candidate block schemes based on the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy includes: For each candidate block partitioning scheme, compare the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy, and select the reuse strategy corresponding to the minimum total number of off-chip memory accesses as the target reuse strategy of the candidate block partitioning scheme. Compare the total number of off-chip memory accesses corresponding to the target reuse strategies of all the candidate block segmentation schemes, and select the candidate block segmentation scheme with the minimum total number of off-chip memory accesses as the target block segmentation scheme of the current convolutional layer.

[0020] Optionally, performing data reuse of the multiple convolutional layers according to the target reuse strategy and the target chunking scheme includes: When the target reuse strategy is an input reuse strategy, the input data is fixed in the storage unit to perform the reuse of the input data; When the target reuse strategy is a weight reuse strategy, the weight data is fixed in the storage unit to perform the reuse of the weight data; When the target reuse strategy is the output reuse strategy, the output data is sent to the storage unit as the input data of the next convolutional layer to perform the reuse of the output data.

[0021] In a second aspect, embodiments of the present invention provide a data reuse apparatus for convolutional neural networks, the apparatus comprising: The parameter acquisition module is used to acquire the network layer parameters and hardware constraint parameters of multiple convolutional layers in the convolutional neural model. The candidate scheme generation module is used to generate multiple candidate block schemes based on the hardware constraint parameters. The reuse count determination module is used to obtain the number of times the input data is reused, the number of times the weight data is reused, and the number of times the output data is reused for the multiple convolutional layers according to the network layer parameters. The off-chip memory access count determination module is used to obtain, based on the network layer parameters and the multiple candidate block partitioning schemes, the off-chip memory access counts of the input data, the off-chip memory access counts of the weight data, and the off-chip memory access counts of the output data of the multiple candidate block partitioning schemes in the multiple convolutional layers, respectively. The total off-chip memory access count determination module is used to output the total off-chip memory access count under the input reuse strategy, the total off-chip memory access count under the weight reuse strategy, and the total off-chip memory access count under the output reuse strategy of the multiple convolutional layers, based on the input data reuse count, the weight data reuse count, the output data reuse count, the input data off-chip memory access count, the weight data off-chip memory access count, and the output data off-chip memory access count, respectively. The target scheme determination module is used to determine the target block scheme and corresponding target reuse strategy of the multiple convolutional layers from the multiple candidate block schemes based on the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy. The data reuse module is used to perform data reuse of the multiple convolutional layers according to the target reuse strategy and the target block partitioning scheme.

[0022] Optionally, the hardware constraint parameters include: input buffer capacity, weight buffer capacity, output buffer capacity, and data bit width; the candidate solution generation module includes: The feature map acquisition submodule is used to acquire the input feature map, weight feature map and output feature map of the multiple convolutional layers respectively; The input segmentation submodule is used to segment the input feature map of each convolutional layer into blocks to obtain the input blocks of the convolutional layer; the data size of the input block is less than or equal to the input buffer capacity of the convolutional layer. The weight segmentation submodule is used to segment the weight feature map of each convolutional layer into weight segments of the convolutional layer; the data size of the weight segment is less than or equal to the weight cache capacity of the convolutional layer. The output block submodule is used to divide the output feature map of each convolutional layer into blocks to obtain the output blocks of the convolutional layer; the data size of the output block is less than or equal to the output buffer capacity of the convolutional layer. The candidate scheme output submodule generates multiple candidate block schemes based on the input blocks, weight blocks, and output blocks of multiple convolutional layers; each candidate block scheme includes an input block, a weight block, and an output block of a convolutional layer.

[0023] Optionally, the network layer parameters include: input feature map height, input feature map width, number of input channels, number of output channels, kernel size, stride, and loading amount per pass; the reuse count determination module includes: The input multiplexing count determination submodule is used to obtain the input multiplexing count based on the convolution kernel size, the convolution stride, and the number of output channels. The input multiplexing count is obtained according to the following formula:

[0024] The weight reuse count determination submodule is used to determine the weight reuse count based on the input feature map height, the input feature map width, the convolution kernel size, and the convolution stride. The weight reuse count is obtained according to the following formula:

[0025] The output multiplexing count determination submodule is used to obtain the output multiplexing count based on the convolution kernel size and the number of input channels. The output multiplexing count is obtained according to the following formula:

[0026] Among them, RF I n Indicates the number of times the input data of the nth convolutional layer is reused; RF w n Indicates the number of times the weight data of the nth convolutional layer is reused; RF o n K represents the number of times the output data of the nth convolutional layer is reused; n H represents the kernel size of the nth convolutional layer, M represents the number of output channels, and H represents the kernel size of the nth convolutional layer. n L represents the height of the input feature map of the nth convolutional layer. n represents the width of the input feature map of the nth convolutional layer, N represents the number of input channels, and str represents the amount of data loaded each time.

[0027] Optionally, the hardware constraint parameters further include: the maximum data size in a single burst transmission; the off-chip memory access count determination module includes: The input off-chip memory access determination submodule is used to determine the number of off-chip memory accesses for the input data based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume of a single burst transmission. The number of off-chip memory accesses for the input data is obtained according to the following formula:

[0028] The weighted data slice external memory access determination submodule is used to obtain the number of weighted data slice external memory accesses based on the candidate block partitioning scheme, convolution kernel size, and maximum data volume of a single burst transmission. The number of weighted data slice external memory accesses is obtained according to the following formula:

[0029] The output off-chip memory access determination submodule is used to determine the number of off-chip memory accesses for the output data based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume of a single burst transmission. The number of off-chip memory accesses for the output data is obtained according to the following formula:

[0030] Among them, MA DRAM_I MA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O Indicates the number of off-chip memory accesses for the output data; t h Indicates the number of blocks in the height direction, t l Indicates the number of blocks divided in the width direction, t n Indicates the number of blocks segmented in the input channel direction, t m Indicates the number of blocks in the output channel direction, t r Indicates the number of blocks segmented in the output height direction, t c WN represents the number of blocks in the output width direction; M represents the number of output channels; and N represents the number of input channels.

[0031] Optionally, the total number of off-chip memory accesses determination module includes: The first total off-chip memory access determination submodule is used to obtain the total number of off-chip memory accesses under the input multiplexing strategy according to the following formula:

[0032] The second total off-chip memory access determination submodule is used to obtain the total number of off-chip memory accesses under the weighted reuse strategy according to the following formula:

[0033] The third submodule for determining the total number of off-chip memory accesses is used to obtain the total number of off-chip memory accesses under the output multiplexing strategy according to the following formula:

[0034] Among them, MA DRAM_ir MA represents the total number of off-chip memory accesses under the input multiplexing strategy; DRAM_wr Total number of off-chip memory accesses under the representation right reuse strategy; MA DRAM_or The output multiplexing strategy represents the total number of off-chip memory accesses; H represents the height of the input feature map; L represents the width of the input feature map; N represents the number of input channels; M represents the number of output channels; R represents the height of the output feature map; C represents the width of the output feature map; T l T represents the block size in the width direction of the input feature map;n Indicates the block size in the input channel direction; T m Indicates the block size in the output channel direction; T h T represents the block size in the height direction of the input feature map. r T represents the block size in the height direction of the output feature map; c Indicates the block size in the width direction of the output feature map; RF I n Indicates the number of times the input data of the nth layer is reused; RF w n Represents the number of times the weight data of the nth layer is reused; RF o n MA represents the number of times the output data of the nth layer is reused. DRAM_I MA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O This indicates the number of off-chip memory accesses for the output data.

[0035] Optionally, the target solution determination module includes: The target strategy determination submodule is used to compare the total number of off-chip memory accesses under the input multiplexing strategy, the total number of off-chip memory accesses under the weighted multiplexing strategy, and the total number of off-chip memory accesses under the output multiplexing strategy for each candidate block partitioning scheme, and select the multiplexing strategy corresponding to the minimum total number of off-chip memory accesses as the target multiplexing strategy of the candidate block partitioning scheme. The target block segmentation scheme determination submodule is used to compare the total number of off-chip memory accesses corresponding to the target reuse strategies of all the candidate block segmentation schemes, and select the candidate block segmentation scheme with the minimum total number of off-chip memory accesses as the target block segmentation scheme of the current convolutional layer.

[0036] Optionally, the data multiplexing module includes: The first multiplexing submodule is used to fix the input data in the storage unit when the target multiplexing strategy is an input multiplexing strategy, so as to perform the multiplexing of the input data; The second reuse submodule is used to fix the weight data in the storage unit when the target reuse strategy is a weight reuse strategy, so as to perform the reuse of the weight data; The third multiplexing submodule is used to send the output data as the input data of the next convolutional layer into the storage unit when the target multiplexing strategy is the output multiplexing strategy, so as to perform the multiplexing of the output data.

[0037] Thirdly, embodiments of the present invention provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0038] Fourthly, embodiments of the present invention provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, the steps of the data reuse method for a convolutional neural network as described in the first aspect are implemented.

[0039] The embodiments of the present invention have the following advantages: By acquiring the network layer parameters and hardware constraint parameters of multiple convolutional layers in a convolutional neural model, multiple candidate block partitioning schemes are generated. Based on the network layer parameters and the multiple candidate block partitioning schemes, the number of off-chip memory accesses for the input data, weight data, and output data of each candidate block partitioning scheme in multiple convolutional layers are obtained. From the multiple candidate block partitioning schemes, the target block partitioning schemes and corresponding target reuse strategies for each convolutional layer are determined. Data reuse of multiple convolutional layers is executed according to the target reuse strategy and target block partitioning scheme. This achieves the selection of the optimal block partitioning scheme and optimal reuse strategy for each convolutional layer. It can avoid the problem of existing technologies causing repeated data movement and redundant memory accesses, and the cumulative increase in off-chip memory accesses and ineffective energy consumption in scenarios with sufficient cache resources due to fixed reuse strategies. It can minimize the number of off-chip memory accesses under different network layer structures and cache configurations, and effectively reduce memory access energy consumption and computation latency through data reuse. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the steps of a data reuse method for convolutional neural networks provided in an embodiment of the present invention. Figure 2 This is a flowchart of another data reuse method for convolutional neural networks provided in an embodiment of the present invention; Figure 3 This is an application flowchart of a data reuse method for convolutional neural networks provided in an embodiment of the present invention; Figure 4 This is a structural block diagram of a data multiplexing device for a convolutional neural network provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0044] With the rapid development of artificial intelligence technology, Convolutional Neural Networks (CNNs) have been widely used in fields such as image recognition and natural language processing. Their model size and computational complexity continue to grow, placing higher demands on the computing power and energy efficiency of hardware acceleration platforms. In related technologies, hardware acceleration solutions typically employ fixed data reuse strategies, such as fixed weighted reuse or fixed input reuse, to uniformly schedule data flow across all convolutional layers. However, when network layer structures are diverse and hardware cache configurations are limited, fixed strategies cannot adaptively adjust to the characteristics of each convolutional layer, leading to increased off-chip memory accesses, higher computational latency, and ineffective energy consumption, making it difficult to fully leverage the energy efficiency advantages of in-memory computing arrays.

[0045] Therefore, how to adaptively select the optimal data reuse strategy for different network layer structures and cache configurations has become a pressing technical problem to be solved in the field of in-memory array neural network acceleration.

[0046] One of the core concepts of this invention is to generate multiple candidate block partitioning schemes by obtaining the network layer parameters and hardware constraint parameters of multiple convolutional layers of a convolutional neural model. Based on the network layer parameters and the multiple candidate block partitioning schemes, the number of off-chip memory accesses for the input data, the number of off-chip memory accesses for the weight data, and the number of off-chip memory accesses for the output data of the multiple candidate block partitioning schemes in the multiple convolutional layers are obtained respectively. From the multiple candidate block partitioning schemes, the target block partitioning schemes and corresponding target reuse strategies for the multiple convolutional layers are determined respectively. Data reuse of the multiple convolutional layers is performed according to the target reuse strategy and the target block partitioning scheme. This realizes the selection of the optimal block partitioning scheme and the optimal reuse strategy for each convolutional layer. It can avoid the problem of existing technologies where fixed reuse strategies lead to the misjudgment that data needs to be repeatedly moved in scenarios with sufficient cache resources, resulting in redundant memory accesses, an increase in the number of off-chip memory accesses, and ineffective energy consumption. It can minimize the number of off-chip memory accesses under different network layer structures and cache configurations, and effectively reduce memory access energy consumption and computation latency through data reuse.

[0047] Figure 1 This is a flowchart of the steps of a data reuse method for convolutional neural networks provided in an embodiment of the present invention.

[0048] Step 101: Obtain the network layer parameters and hardware constraint parameters of multiple convolutional layers of the convolutional neural model; In this embodiment of the invention, the convolutional neural model can be a deep learning network model containing multiple convolutional layers, pooling layers, fully connected layers, etc. It extracts features from the input data through convolutional operations and is widely used in artificial intelligence tasks such as image recognition, object detection, and semantic segmentation. In the convolutional computation scenario of in-memory arrays, the input feature map, convolutional kernel weights, and output feature map need to be moved from off-chip storage to the in-memory unit. By fixing the reused data under the target reuse strategy in the in-memory unit, the data is reused during the computation process, thereby reducing the number of times data is moved from off-chip storage.

[0049] By obtaining the aforementioned network layer parameters and hardware constraint parameters, it is possible to evaluate the data reuse capability of each convolutional layer directly using the specifications of the convolutional neural model and the in-memory array hardware platform without relying on additional measurement overhead.

[0050] Step 102: Generate multiple candidate block partitioning schemes based on the hardware constraint parameters; In this embodiment of the invention, the candidate block segmentation scheme can characterize the way in which the input feature map, convolutional kernel weights, and output feature map of the current convolutional layer are segmented in the spatial dimension and the channel dimension, respectively. In the hardware deployment scenario of convolutional neural networks, due to the limited capacity of the in-memory computing unit, it is impossible to accommodate the complete input feature map, weights, and output feature map at one time. Therefore, the above data needs to be segmented into multiple smaller blocks and loaded into the in-memory computing unit in batches for calculation.

[0051] By generating multiple candidate block partitioning schemes, embodiments of the present invention can select the scheme that best matches the memory access characteristics of the current convolutional layer from different block partitioning granularities in subsequent steps.

[0052] Step 103: Based on the network layer parameters, obtain the number of times the input data is reused, the number of times the weight data is reused, and the number of times the output data is reused for the multiple convolutional layers, respectively. In this embodiment of the invention, the number of data reuses can characterize the theoretical number of times a certain data is reused during the convolution calculation process. The number of input data reuses can characterize the number of times each data point of the input feature map is repeatedly read during the sliding of the convolution kernel; the number of weight data reuses can characterize the number of times each weight parameter is repeatedly used on the output feature map plane; and the number of output data reuses can characterize the number of times each output result is repeatedly read internally during the accumulation process.

[0053] In some examples, the number of times the input data is reused by multiple output channel kernels during the sliding of the convolution window is determined based on the kernel size, stride, and number of output channels. The number of times the weight parameters are reused as they slide across the output feature map plane is determined based on the input feature map height, width, kernel size, and stride. The number of times the output result is repeatedly read internally during the accumulation process is determined based on the kernel size and number of input channels.

[0054] By calculating the three reuse times mentioned above, the embodiments of the present invention can quantify the reuse parameters of each type of data in the current convolutional layer.

[0055] Step 104: Based on the network layer parameters and the multiple candidate block partitioning schemes, obtain the number of memory accesses outside the input data slices, the number of memory accesses outside the weight data slices, and the number of memory accesses outside the output data slices of the multiple candidate block partitioning schemes in the multiple convolutional layers, respectively. In this embodiment of the invention, the number of off-chip memory accesses can characterize the number of burst transfers of data from off-chip storage required to complete the computation of the current convolutional layer under the current block partitioning scheme. The off-chip operation type for input data and weight data is a read operation; the off-chip operation type for output data includes two types: write-back to off-chip storage and read-back from off-chip storage. When the output block data volume exceeds the output cache capacity, it is necessary to first write part of the computation results back to off-chip storage, and then read them back to the on-chip in-memory computing unit to continue computation, thus incurring twice the memory access overhead.

[0056] By calculating the number of off-chip memory accesses for each of the three types of data, embodiments of the present invention can transform the block-based scheme into quantifiable memory access overhead.

[0057] Step 105: Based on the number of input data reuses, the number of weight data reuses, the number of output data reuses, the number of input data accesses outside the memory slice, the number of weight data accesses outside the memory slice, and the number of output data accesses outside the memory slice, output the total number of out-of-memory accesses under the input reuse strategy, the total number of out-of-memory accesses under the weight reuse strategy, and the total number of out-of-memory accesses under the output reuse strategy for the multiple convolutional layers, respectively. In this embodiment of the invention, the reuse strategy can characterize which data is selected as the primary data object to be repeatedly used and fixed in the computing unit during the convolution calculation process. The input reuse strategy can characterize fixing the input feature map data on the chip, and feeding the weight data into the computing unit in a sliding manner according to the convolution window to complete the calculation of all output channels; the weight reuse strategy can characterize fixing the weight data on the chip, and feeding the input feature map data into the computing unit in a sliding manner according to the convolution window to complete the calculation of all spatial locations; the output reuse strategy can characterize summing the output part on the chip to reduce the number of times it is written back to off-chip storage.

[0058] Based on the number of input data reuses, weight data reuses, and output data reuses, as well as the number of off-chip memory accesses for input data, weight data, and output data, the total number of off-chip memory accesses is calculated for each of the three strategies. In the input reuse strategy, the memory access count is not amplified because the input data is fixed on-chip, while the memory access counts for weight data and output data are amplified by the number of input reuses. In the weight reuse strategy, the memory access count is not amplified because the weight data is fixed on-chip, while the memory access count for input data is amplified by the number of weight reuses, and the memory access count for output data is amplified by the number of weight reuses. In the output reuse strategy, the memory access count is not amplified because the output data is accumulated on-chip, while the memory access count for input data is amplified by the number of output reuses, and the memory access count for output data is also amplified by the number of output reuses.

[0059] By calculating the total number of off-chip memory accesses under the three strategies, the embodiments of the present invention can quantify the memory access efficiency of each strategy under the current convolutional layer and block partitioning scheme.

[0060] Step 106: Based on the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy, determine the target block scheme and the corresponding target reuse strategy for the multiple convolutional layers from the multiple candidate block schemes. In this embodiment of the invention, the target block scheme can characterize the block size combination that best matches the current convolutional layer selected from multiple candidate block schemes; the target reuse strategy can characterize the reuse method selected from three reuse strategies under the current convolutional layer and block scheme.

[0061] By selecting a target reuse strategy and a target partitioning scheme, this embodiment of the invention ensures that the final determined combination is globally optimal among all feasible partitioning schemes and all reuse strategies in the current convolutional layer.

[0062] Step 107: Perform data reuse of the multiple convolutional layers according to the target reuse strategy and the target block segmentation scheme.

[0063] In this embodiment of the invention, data reuse can be characterized as the operation of fixing input data, weight data, or output data in the memory unit and using them repeatedly to reduce the number of off-chip memory accesses during the convolution calculation process.

[0064] This invention generates multiple candidate block partitioning schemes by acquiring network layer parameters and hardware constraint parameters of multiple convolutional layers in a convolutional neural model. Based on the network layer parameters and the multiple candidate block partitioning schemes, it obtains the number of off-chip memory accesses for input data, weight data, and output data in each of the multiple candidate block partitioning schemes. From the multiple candidate block partitioning schemes, it determines the target block partitioning schemes and corresponding target reuse strategies for each convolutional layer. Data reuse of multiple convolutional layers is then performed according to the target reuse strategy and target block partitioning scheme. This achieves the selection of the optimal block partitioning scheme and optimal reuse strategy for each convolutional layer. It avoids the problem in existing technologies where fixed reuse strategies lead to misjudgments of data transfer in scenarios with ample cache resources, resulting in redundant memory accesses, increased off-chip memory accesses, and ineffective energy consumption. This invention minimizes the number of off-chip memory accesses under different network layer structures and cache configurations, effectively reducing memory access energy consumption and computation latency through data reuse.

[0065] Figure 2 This is a flowchart of another data reuse method for convolutional neural networks provided in an embodiment of the present invention.

[0066] Step 201: Obtain the network layer parameters and hardware constraint parameters of multiple convolutional layers of the convolutional neural model; In this embodiment of the invention, the convolutional neural model can be a deep learning network model containing multiple convolutional layers, pooling layers, fully connected layers, etc. It extracts features from the input data through convolutional operations and is widely used in artificial intelligence tasks such as image recognition, object detection, and semantic segmentation. In the convolutional computation scenario of in-memory arrays, the input feature map, convolutional kernel weights, and output feature map need to be moved from off-chip storage to the in-memory unit and then sent to the in-memory array for computation. By fixing the reused data under the target reuse strategy in the in-memory unit, the data is reused during the computation process, thereby reducing the number of times data is moved from off-chip storage.

[0067] In some examples, convolutional layers can be the core computational units for feature extraction in convolutional neural models. Each convolutional layer contains a set of learnable convolutional kernels, which perform multiplication and addition operations by sliding across the input feature map to produce an output feature map. Network layer parameters characterize the structural configuration information of each convolutional layer, including the height and width of the input feature map, the number of input channels, the number of output channels, the kernel size, and the convolution stride. Hardware constraint parameters characterize the storage resource limitations of the in-memory computing array hardware platform, and may include input cache capacity, weight cache capacity, output cache capacity, data bit width, and the maximum data volume in a single burst transfer. These hardware constraint parameters can be determined by the chip design specifications, affecting the maximum size of data blocks and the granularity of off-chip memory access.

[0068] By obtaining the aforementioned network layer parameters and hardware constraint parameters, it is possible to evaluate the data reuse capability of each convolutional layer directly using the specifications of the convolutional neural model and the in-memory array hardware platform without relying on additional measurement overhead.

[0069] Step 202: Generate multiple candidate block partitioning schemes based on the hardware constraint parameters; In this embodiment of the invention, the candidate block segmentation scheme can characterize the way the input feature map, convolutional kernel weights, and output feature map of the current convolutional layer are segmented in both spatial and channel dimensions. Since the storage unit has limited capacity and cannot accommodate the complete input feature map, weights, and output feature map at once, the data needs to be segmented into multiple smaller blocks and loaded into the storage unit in batches for computation. Block size design variables may include the input block height and width, the input channel block size, the convolutional kernel block size, the output channel block size, and the output block height and width.

[0070] In some embodiments, hardware constraint parameters may include: input buffer capacity, weight buffer capacity, output buffer capacity, and data bit width; step 202 may include the following sub-steps: Sub-step S11: Obtain the input feature map, weight feature map, and output feature map of the multiple convolutional layers respectively; In this embodiment of the invention, due to the limited capacity of the in-memory computing unit, it is impossible to accommodate the complete input feature map, weights, and output feature map at once. Therefore, the above data needs to be divided into multiple smaller blocks and loaded into the on-chip cache in batches for computation. The block size design variables may include the height and width of the input block, the size of the input channel block, the size of the convolution kernel block, the size of the output channel block, and the height and width of the output block.

[0071] Sub-step S12 involves dividing the input feature map of each convolutional layer into blocks to obtain the input blocks of the convolutional layer; the data size of the input blocks is less than or equal to the input buffer capacity of the convolutional layer. In this embodiment of the invention, the input block can represent the sub-blocks obtained by dividing the input feature map in the height direction, width direction, and input channel direction, respectively. By constraining the data size of the input block to not exceed the input cache capacity, it is ensured that each input block can be fully loaded into the input cache for computation, avoiding cache overflow due to excessively large blocks and triggering additional off-chip memory access operations.

[0072] Sub-step S13: Divide the weight feature map of each convolutional layer into blocks to obtain weight blocks of the convolutional layer; the data size of the weight blocks is less than or equal to the weight cache capacity of the convolutional layer; In this embodiment of the invention, weight blocks can represent sub-blocks obtained by dividing the weight parameters in the convolution kernel space dimension, input channel direction, and output channel direction, respectively. By constraining the data size of the weight blocks to not exceed the weight cache capacity, it is ensured that each weight block can be completely loaded into the weight cache, so that the weight data can be repeatedly read during the calculation process without having to be moved from off-chip.

[0073] Sub-step S14: The output feature map of each convolutional layer is divided into blocks to obtain the output blocks of the convolutional layer; the data size of the output blocks is less than or equal to the output buffer capacity of the convolutional layer; In this embodiment of the invention, the output block can represent the sub-blocks obtained by dividing the output feature map in the height direction, width direction, and output channel direction, respectively. By constraining the data size of the output block to not exceed the output cache capacity, it is ensured that the output block can be temporarily stored on-chip without frequent write-back to off-chip storage, effectively reducing the additional memory access overhead of write-back and read-back.

[0074] Sub-step S15: Based on the input blocks, weight blocks, and output blocks of multiple convolutional layers, generate multiple candidate block schemes; each candidate block scheme includes an input block, a weight block, and an output block of a convolutional layer.

[0075] In this embodiment of the invention, a candidate block partitioning scheme can represent a complete set of block size combinations, including input block size, weight block size, and output block size. By traversing all block size combinations that satisfy the cache capacity constraint, multiple candidate block partitioning schemes are generated, providing a candidate set for subsequent selection of the block partitioning scheme that best matches the memory access characteristics of the current convolutional layer.

[0076] Step 203: Based on the network layer parameters, obtain the number of times the input data is reused, the number of times the weight data is reused, and the number of times the output data is reused for the multiple convolutional layers, respectively. In this embodiment of the invention, the number of data reuses can characterize the theoretical number of times a certain type of data is reused during the convolution calculation process. The number of input data reuses can characterize the number of times each data point of the input feature map is repeatedly read during the sliding of the convolution kernel; the number of weight data reuses can characterize the number of times each weight parameter is repeatedly used on the output feature map plane; and the number of output data reuses can characterize the number of times each output result is repeatedly read internally during the accumulation process.

[0077] In some embodiments, network layer parameters may include: input feature map height, input feature map width, number of input channels, number of output channels, kernel size, stride, and amount of data loaded per cycle. Step 203 may include the following sub-steps: Sub-step S21: The number of input multiplexing times is obtained based on the kernel size, the stride, and the number of output channels. The number of input multiplexing times is obtained according to the following formula:

[0078] In this embodiment of the invention, the input multiplexing count RF is... I n The calculation method is as follows: Each data point on the input feature map falls into multiple convolution windows during the sliding process of the convolution kernel, and performs multiplication and addition operations with the convolution kernel within each window. Each convolution kernel covers K×K input points, and the stride str determines the window sliding interval. Therefore, the number of times a single data point is covered is (K... n / str)×(K n / str). For each output channel, there is an independent convolution kernel that operates on that input point, so it needs to be multiplied by the number of output channels M.

[0079] Sub-step S22: The number of weight reuses is obtained based on the input feature map height, the input feature map width, the convolution kernel size, and the convolution stride. The number of weight reuses is obtained according to the following formula:

[0080] In this embodiment of the invention, the number of times the weight is reused is RF. w n The calculation method is as follows: each weight parameter participates in the calculation at each position on the output feature map as it slides on the output feature map plane. The height of the output feature map is (HK) + 1 / str, and the width is (LK) + 1 / str. Therefore, each weight parameter is reused (HK) + 1 / str times in the height direction and (LK) + 1 / str times in the width direction. The number of times the weight is reused is the product of the two.

[0081] Sub-step S23: The number of output multiplexing times is obtained based on the convolution kernel size and the number of input channels. The number of output multiplexing times is obtained according to the following formula:

[0082] In this embodiment of the invention, the calculation method for the output multiplexing count RF_O^n is as follows: each output result is obtained by calculating K... n ×K n This is obtained by multiplying and adding N input data with their corresponding weights. Therefore, during the accumulation process, each output result needs to be read from the input channel direction K. n ×K n After processing N data points, the number of times the data is reused is output as K. n ×K n ×N.

[0083] Among them, RF I n Indicates the number of times the input data of the nth convolutional layer is reused; RF w n Indicates the number of times the weight data of the nth convolutional layer is reused; RF o n K represents the number of times the output data of the nth convolutional layer is reused; n H represents the kernel size of the nth convolutional layer, M represents the number of output channels, and H represents the kernel size of the nth convolutional layer. n L represents the height of the input feature map of the nth convolutional layer. n represents the width of the input feature map of the nth convolutional layer, N represents the number of input channels, and str represents the amount of data loaded each time.

[0084] Step 204: Based on the network layer parameters and the multiple candidate block partitioning schemes, obtain the number of memory accesses outside the input data slices, the number of memory accesses outside the weight data slices, and the number of memory accesses outside the output data slices of the multiple candidate block partitioning schemes in the multiple convolutional layers, respectively. In some embodiments, the hardware constraint parameters may further include: the maximum data size for a single burst transmission; step 204 may further include the following sub-steps: Sub-step S31: Based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume in a single burst transmission, the number of external memory accesses for the input data slice is obtained according to the following formula:

[0085] In this embodiment of the invention, the number of external memory accesses for the input data slice can be calculated as follows: when the input block covers the entire input feature map, it only needs to be moved in blocks along the input channel direction, and the number of moves is t. n ×[(t h ×t lWhen the input block needs to be further divided along the height and width directions, the number of transfers is t. l ×t m ×[t h / WN].

[0086] Sub-step S32: Based on the candidate block partitioning scheme, convolution kernel size, and maximum data volume in a single burst transmission, the number of external memory accesses for the weighted data slice is obtained. The number of external memory accesses for the weighted data slice is calculated according to the following formula:

[0087] In this embodiment of the invention, the number of external memory accesses for weighted data slices can be calculated as follows: when the output channel is divided into blocks t m When the number of output channels is less than the total number of output channels M, the weights need to be moved in batches along the output channel direction, with the number of moves being K×K×t. n +[t m / WN]; otherwise, directly move all weights, with t being the number of moves. n ×[(t m ×K×K) / WN].

[0088] Sub-step S33: Based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume in a single burst transmission, the number of external memory accesses for the output data slice is obtained according to the following formula:

[0089] In this embodiment of the invention, the number of external memory accesses for the output data slice can be calculated as follows: when the output width is divided into blocks t... c Less than the total width C and the input channel is divided into blocks t n When the number of transports is less than the total number of input channels N, the number of transports is t. r ×t m ×[t c / WN]; when the output width is divided into blocks t c The output height is less than the total width C and the output height is divided into blocks t. r Equal to the total height R and the input channel is divided into blocks t n When the total number of input channels N is equal to the number of transports, the number of transports is t. m ×[(t c ×t r ) / WN]; When the input channel is divided into blocks t n When the number of input channels is less than the total number of input channels N, the output buffer overflows and needs to be written back and read back, with t being the number of times the buffer is moved. r ×t m +[t c / WN]×2 or t m +[(t c ×t r[) / WN]×2. Where ×2 represents the additional overhead of one write-back to off-chip memory and one read-back to off-chip memory.

[0090] In the above sub-steps S31-S33, MA DRAM_I MA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O Indicates the number of off-chip memory accesses for the output data; t h Indicates the number of blocks in the height direction, t l Indicates the number of blocks divided in the width direction, t n Indicates the number of blocks segmented in the input channel direction, t m Indicates the number of blocks in the output channel direction, t r Indicates the number of blocks segmented in the output height direction, t c WN represents the number of blocks in the output width direction; M represents the number of output channels; and N represents the number of input channels.

[0091] Step 205: Based on the number of input data reuses, the number of weight data reuses, the number of output data reuses, the number of input data accesses outside the memory slice, the number of weight data accesses outside the memory slice, and the number of output data accesses outside the memory slice, output the total number of out-of-memory accesses under the input reuse strategy, the total number of out-of-memory accesses under the weight reuse strategy, and the total number of out-of-memory accesses under the output reuse strategy for the multiple convolutional layers, respectively. In some embodiments, step 205 may further include the following sub-steps: Sub-step S41: The total number of off-chip memory accesses under the input multiplexing strategy is obtained according to the following formula:

[0092] In this embodiment of the invention, under the input reuse strategy, the input data is fixed in the on-chip cache and used repeatedly. Therefore, the off-chip memory access frequency of the input data is not amplified, and the coefficient is 1. The memory access frequency of the weight data and output data is amplified as the input data is reused, and is multiplied by the input reuse frequency RF. I n The weighted sum within the parentheses is then multiplied by the total number of blocks in the input feature map (H×L×N) / (T). h ×T i ×T n ), thus obtaining the total number of off-chip memory accesses under the input reuse strategy.

[0093] Sub-step S42: The total number of off-chip memory accesses under the weighted reuse strategy is obtained according to the following formula:

[0094] In this embodiment of the invention, under the weight reuse strategy, the weight data is fixed in the on-chip cache and used repeatedly. Therefore, the off-chip memory access frequency of the weight data is not amplified, and the coefficient is 1. The memory access frequency of input data and output data is amplified with the reuse of weight data, and is multiplied by the weight reuse frequency RF. w n The weighted sum within parentheses is multiplied by the weight, resulting in a total number of blocks (M×N) / (T). m ×T n ), and obtain the total number of off-site memory accesses under the right reuse strategy.

[0095] Sub-step S43: The total number of off-chip memory accesses under the output multiplexing strategy is obtained according to the following formula:

[0096] In this embodiment of the invention, under the output multiplexing strategy, the output data is accumulated and multiplexed in the on-chip cache, so the off-chip memory access count of the output data is not amplified, and the coefficient is 1; the input data is repeatedly read along with the multiplexing of the output data, and needs to be multiplied by the output multiplexing count RF. o n The weighted sum within the parentheses is then multiplied by the total number of blocks in the output feature map (C×R×M) / (T). c ×T r ×T m ), which gives the total number of off-chip memory accesses under the output reuse strategy.

[0097] Among them, MA DRAM_ir MA represents the total number of off-chip memory accesses under the input multiplexing strategy; DRAM_wr Total number of off-chip memory accesses under the representation right reuse strategy; MA DRAM_or The output multiplexing strategy represents the total number of off-chip memory accesses; H represents the height of the input feature map; L represents the width of the input feature map; N represents the number of input channels; M represents the number of output channels; R represents the height of the output feature map; C represents the width of the output feature map; T l T represents the block size in the width direction of the input feature map; n Indicates the block size in the input channel direction; T m Indicates the block size in the output channel direction; T h T represents the block size in the height direction of the input feature map. r T represents the block size in the height direction of the output feature map; c Indicates the block size in the width direction of the output feature map; RF I n Indicates the number of times the input data of the nth layer is reused; RF w n Represents the number of times the weight data of the nth layer is reused; RF o nMA represents the number of times the output data of the nth layer is reused. DRAM_I MA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O This indicates the number of off-chip memory accesses for the output data.

[0098] Step 206: For each candidate block partitioning scheme, compare the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy, and select the reuse strategy corresponding to the minimum total number of off-chip memory accesses as the target reuse strategy of the candidate block partitioning scheme. In this embodiment of the invention, the target reuse strategy can characterize the reuse method that minimizes the number of off-chip memory accesses among the input reuse strategy, weighted reuse strategy, and output reuse strategy under the current candidate block partitioning scheme. Each candidate block partitioning scheme corresponds to the total number of off-chip memory accesses under the input reuse, weighted reuse, or output reuse strategy. By comparing the magnitudes of the three values, the strategy corresponding to the minimum value is selected as the target reuse strategy for that candidate block partitioning scheme. For example, if MA under a certain candidate block partitioning scheme DRAM_ir If the minimum is found, then the input multiplexing strategy is the most efficient under this block partitioning scheme; if MA DRAM_wr If the minimum value is found, then the most efficient strategy is to reuse the weights; if MA is found... DRAM_or If the minimum value is reached, then the output reuse strategy is the most efficient.

[0099] Step 207: Compare the total number of off-chip memory accesses corresponding to the target reuse strategies of all the candidate block partitioning schemes, and select the candidate block partitioning scheme with the smallest total number of off-chip memory accesses as the target block partitioning scheme of the current convolutional layer. In this embodiment of the invention, the target block partitioning scheme can represent the block size combination that best matches the current convolutional layer, selected from all candidate block partitioning schemes. After step 206, each candidate block partitioning scheme has determined its target reuse strategy and corresponding minimum total off-chip memory accesses. This step compares all candidate block partitioning schemes and selects the scheme with the minimum total off-chip memory accesses as the target block partitioning scheme for the current convolutional layer. The reuse strategy corresponding to this scheme is the target reuse strategy for the current convolutional layer.

[0100] Step 208: When the target multiplexing strategy is an input multiplexing strategy, the input data is fixed in the storage unit to perform the multiplexing of the input data; In this embodiment of the invention, the in-memory computing unit can represent a hardware unit that integrates storage and computing functions, including an on-chip cache for caching data blocks and an in-memory computing array for performing convolution multiplication-accumulation operations. The core of the data reuse strategy lies in fixing a certain type of data within the in-memory computing unit for repeated use, thereby reducing the number of times data is moved from off-chip storage. The input reuse strategy is applicable to scenarios where the input feature map size is small but the number of convolution kernels is large.

[0101] In some embodiments, when the target multiplexing strategy is an input multiplexing strategy, the input data is fixed in the on-chip cache of the in-memory unit, and the weight data in the in-memory array is switched sequentially. Taking the convolution calculation of a 9×9 input feature map with 9 3×3 convolution kernels as an example, the 9×9 input feature map is first divided into 3×3 input blocks and fixed in the on-chip cache. The 3×3 input blocks are then sequentially convolved with the 9 convolution kernels in the in-memory array. Only the weight data in the in-memory array needs to be switched in between, and the input data remains unchanged in the on-chip cache.

[0102] By fixing the input data and switching the weight data, the input data can be loaded once and used multiple times, thereby reducing the number of times input data is moved from off-chip storage.

[0103] Step 209: When the target reuse strategy is a weight reuse strategy, the weight data is fixed in the storage unit to perform the reuse of the weight data. In this embodiment of the invention, the weight reuse strategy can be applied to scenarios where the amount of weight data is smaller than the input feature map or the weight data has a large sliding range on the output feature map plane. When the target reuse strategy is the weight reuse strategy, the weight data can be fixed in the in-memory array, and the input data can be sent from the on-chip cache to the in-memory array in a convolution window sliding manner.

[0104] In some examples, taking the calculation of a 9×9 input feature map with a 3×3 convolution kernel and a stride of 1 as an example, after the 3×3 window in the upper left corner of the input feature map is calculated, it moves one step to the right. At this time, the data in the second and third columns of the on-chip cache remain unchanged, and only the data in the fourth column of the newly slid-in window is added to the on-chip cache. The weight data remains unchanged in the in-memory array.

[0105] By retaining the non-new data from the previous window in the on-chip cache and only updating the new data in the sliding-in window, weight data can be loaded once and used multiple times, thereby reducing the number of times weight data is moved from off-chip storage.

[0106] Step 210: When the target reuse strategy is the output reuse strategy, the output data is sent to the storage unit as the input data of the next convolutional layer to perform the reuse of the output data.

[0107] In this embodiment of the invention, the application is suitable for scenarios where the output feature map size is small or used as an intermediate layer. When the target multiplexing strategy is an output multiplexing strategy, the output portion generated by the in-memory array is accumulated in the on-chip cache, and the accumulated output data is directly sent to the in-memory unit as the input data of the next convolutional layer. At this time, the output data does not need to be written back to off-chip storage and then read back, but is directly sent from the on-chip cache to the in-memory array of the next convolutional layer to participate in the calculation, thereby skipping the read and write operations of off-chip storage and reducing the off-chip memory access overhead of intermediate data.

[0108] This invention, through obtaining network layer parameters and hardware constraint parameters of multiple convolutional layers in a convolutional neural model, generates multiple candidate block partitioning schemes based on the hardware constraint parameters. It then obtains the number of input data reuses, weight data reuses, and output data reuses for each convolutional layer based on the network layer parameters. Finally, it calculates the number of out-of-slice memory accesses for the input data, weight data, and output data of each candidate block partitioning scheme within the multiple convolutional layers, based on the network layer parameters and the multiple candidate block partitioning schemes. Based on each reuse count and each out-of-slice memory access count, it outputs the total number of out-of-slice memory accesses under three reuse strategies. The total number of off-chip memory accesses is determined by selecting the target block scheme and corresponding target reuse strategy for multiple convolutional layers from multiple candidate block schemes. Data reuse of multiple convolutional layers is performed according to the target reuse strategy and target block scheme. This enables each convolutional layer to independently select the optimal block scheme and optimal reuse strategy. It can avoid the problem of existing technologies that, due to a fixed single strategy, are misjudged as needing repeated data movement in scenarios with sufficient cache resources, resulting in redundant memory accesses, increased off-chip memory accesses, and ineffective energy consumption. It can minimize the number of off-chip memory accesses under different network layer structures and cache configurations, effectively reducing memory access energy consumption and computation latency, thereby improving computational energy efficiency.

[0109] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0110] Figure 3 This is an application flowchart of a data reuse method for convolutional neural networks provided in an embodiment of the present invention.

[0111] To enable those skilled in the art to better understand the embodiments of the present invention, see below. Figure 3 An example will be used to illustrate an embodiment of the present invention: 1. The original data is input into the neural network from the outside, and the optimization process is completed to initialize it; In this embodiment of the invention, the original data of the neural network is equivalent to the network layer parameters of multiple convolutional layers of the convolutional neural model, including the height and width of the input feature map, the number of input channels, the number of output channels, the kernel size, and the stride of the convolution; the optimization process initialization is equivalent to the parameter preset steps of completing the initial variables of the optimization algorithm, the hardware resource threshold, and the computing cache space of each layer.

[0112] 2. Bind the current network layer parameters and hardware resource constraints, conduct data reuse strategy analysis, and output multi-dimensional reuse parameters including the number of input / weight / output reuses; In this embodiment of the invention, hardware resource constraints are equivalent to hardware constraint parameters, including input buffer capacity, weight buffer capacity, output buffer capacity, data bit width, and maximum data volume in a single burst transmission; data reuse strategy analysis is equivalent to selecting multiple candidate block partitioning schemes that meet the buffer capacity limit based on the hardware constraint parameters, and obtaining the number of input data reuses, weight data reuses, and output data reuses of multiple convolutional layers based on the network layer parameters.

[0113] 3. Determine if a next layer network exists: If it exists, switch to the next layer and re-execute the hardware constraint matching and reuse analysis loop; if it does not exist, generate the hardware configurations corresponding to all network layers. In this embodiment of the invention, switching to the next layer is equivalent to loading the network parameters of the next layer and re-entering the hardware constraint matching and reuse analysis loop when all network layers have not been optimized; generating hardware configuration is equivalent to determining the target block scheme and corresponding target reuse strategy of the multiple convolutional layers from multiple candidate block schemes based on the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy.

[0114] 4. The entire set of hierarchical configuration information is sent to the host computer, which then distributes the configuration to the underlying in-memory computing neural network accelerator; In this embodiment of the invention, the hierarchical configuration information is equivalent to the control signals of the target reuse strategy and the target block scheme; the upper computer issuing configuration is equivalent to the instruction transmission step of packaging and issuing the operation configuration parameters of each layer of the entire network to the hardware execution unit.

[0115] 5. The accelerator reads the ifmaps input feature map, Weights weights, and Ofmaps output feature map data from the off-chip DRAM and stores them in the on-chip data cache; In this embodiment of the invention, off-chip DRAM is equivalent to off-chip storage; on-chip data cache is equivalent to a storage and computing unit.

[0116] 6. The data cache supplies reused data to the in-memory array, which performs parallel convolution operations. Intermediate results are written back to the cache. After completing a single-layer computation, the array waits for the next layer to be scheduled, and the entire network inference is completed in a loop.

[0117] In this embodiment of the invention, the in-memory array is equivalent to a hardware acceleration unit for performing convolutional neural network inference computation; the intermediate result write-back cache is equivalent to the output part and the accumulation process inside the in-memory unit.

[0118] It should be noted that the data multiplexing method for convolutional neural networks provided in this embodiment of the invention can be executed by a data multiplexing device for convolutional neural networks, or by a control module within that data multiplexing device for executing the data multiplexing method for loading convolutional neural networks. This embodiment of the invention uses the execution of the data multiplexing method for loading convolutional neural networks by a data multiplexing device as an example to illustrate the data multiplexing method for convolutional neural networks provided in this embodiment of the invention.

[0119] Figure 4 This is a structural block diagram of a data multiplexing device for a convolutional neural network provided in an embodiment of the present invention.

[0120] like Figure 4 As shown in the figure, the data multiplexing device for convolutional neural networks provided in this embodiment of the invention may specifically include the following modules: The parameter acquisition module 301 is used to acquire the network layer parameters and hardware constraint parameters of multiple convolutional layers of the convolutional neural model. The candidate scheme generation module 302 is used to generate multiple candidate block schemes based on the hardware constraint parameters. The reuse count determination module 303 is used to obtain the number of times the input data is reused, the number of times the weight data is reused, and the number of times the output data is reused for the multiple convolutional layers according to the network layer parameters. The off-chip memory access count determination module 304 is used to obtain the off-chip memory access counts of the input data, the off-chip memory access counts of the weight data, and the off-chip memory access counts of the output data of the multiple candidate block partitioning schemes in the multiple convolutional layers, respectively, based on the network layer parameters and the multiple candidate block partitioning schemes. The total off-chip memory access count determination module 305 is used to output the total off-chip memory access count under the input reuse strategy, the total off-chip memory access count under the weight reuse strategy, and the total off-chip memory access count under the output reuse strategy of the multiple convolutional layers, based on the input data reuse count, the weight data reuse count, the output data reuse count, the input data off-chip memory access count, the weight data off-chip memory access count, and the output data off-chip memory access count, respectively. The target scheme determination module 306 is used to determine the target block scheme and the corresponding target reuse strategy of the multiple convolutional layers from the multiple candidate block schemes based on the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weight reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy. The data reuse module 307 is used to perform data reuse of the multiple convolutional layers according to the target reuse strategy and the target block scheme.

[0121] In this embodiment of the invention, the hardware constraint parameters include: input buffer capacity, weight buffer capacity, output buffer capacity, and data bit width; the candidate solution generation module includes: The feature map acquisition submodule is used to acquire the input feature map, weight feature map and output feature map of the multiple convolutional layers respectively; The input segmentation submodule is used to segment the input feature map of each convolutional layer into blocks to obtain the input blocks of the convolutional layer; the data size of the input block is less than or equal to the input buffer capacity of the convolutional layer. The weight segmentation submodule is used to segment the weight feature map of each convolutional layer into weight segments of the convolutional layer; the data size of the weight segment is less than or equal to the weight cache capacity of the convolutional layer. The output block submodule is used to divide the output feature map of each convolutional layer into blocks to obtain the output blocks of the convolutional layer; the data size of the output block is less than or equal to the output buffer capacity of the convolutional layer. The candidate scheme output submodule generates multiple candidate block schemes based on the input blocks, weight blocks, and output blocks of multiple convolutional layers; each candidate block scheme includes an input block, a weight block, and an output block of a convolutional layer.

[0122] In this embodiment of the invention, the network layer parameters include: input feature map height, input feature map width, number of input channels, number of output channels, convolution kernel size, convolution stride, and loading amount per pass; the reuse count determination module includes: The input multiplexing count determination submodule is used to obtain the input multiplexing count based on the convolution kernel size, the convolution stride, and the number of output channels. The input multiplexing count is obtained according to the following formula:

[0123] The weight reuse count determination submodule is used to determine the weight reuse count based on the input feature map height, the input feature map width, the convolution kernel size, and the convolution stride. The weight reuse count is obtained according to the following formula:

[0124] The output multiplexing count determination submodule is used to obtain the output multiplexing count based on the convolution kernel size and the number of input channels. The output multiplexing count is obtained according to the following formula:

[0125] Among them, RF I n Indicates the number of times the input data of the nth convolutional layer is reused; RF w n Indicates the number of times the weight data of the nth convolutional layer is reused; RF o n K represents the number of times the output data of the nth convolutional layer is reused; n H represents the kernel size of the nth convolutional layer, M represents the number of output channels, and H represents the kernel size of the nth convolutional layer. n L represents the height of the input feature map of the nth convolutional layer. n represents the width of the input feature map of the nth convolutional layer, N represents the number of input channels, and str represents the amount of data loaded each time.

[0126] In this embodiment of the invention, the hardware constraint parameters further include: the maximum data volume in a single burst transmission; the off-chip memory access count determination module includes: The input off-chip memory access determination submodule is used to determine the number of off-chip memory accesses for the input data based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume of a single burst transmission. The number of off-chip memory accesses for the input data is obtained according to the following formula:

[0127] The weighted data slice external memory access determination submodule is used to obtain the number of weighted data slice external memory accesses based on the candidate block partitioning scheme, convolution kernel size, and maximum data volume of a single burst transmission. The number of weighted data slice external memory accesses is obtained according to the following formula:

[0128] The output off-chip memory access determination submodule is used to determine the number of off-chip memory accesses for the output data based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume of a single burst transmission. The number of off-chip memory accesses for the output data is obtained according to the following formula:

[0129] Among them, MA DRAM_I MA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O Indicates the number of off-chip memory accesses for the output data; t h Indicates the number of blocks in the height direction, t lIndicates the number of blocks divided in the width direction, t n Indicates the number of blocks segmented in the input channel direction, t m Indicates the number of blocks in the output channel direction, t r Indicates the number of blocks segmented in the output height direction, t c WN represents the number of blocks in the output width direction; M represents the number of output channels; and N represents the number of input channels.

[0130] In this embodiment of the invention, the total off-chip memory access count determination module includes: The first total off-chip memory access determination submodule is used to obtain the total number of off-chip memory accesses under the input multiplexing strategy according to the following formula:

[0131] The second total off-chip memory access determination submodule is used to obtain the total number of off-chip memory accesses under the weighted reuse strategy according to the following formula:

[0132] The third submodule for determining the total number of off-chip memory accesses is used to obtain the total number of off-chip memory accesses under the output multiplexing strategy according to the following formula:

[0133] Among them, MA DRAM_ir MA represents the total number of off-chip memory accesses under the input multiplexing strategy; DRAM_wr Total number of off-chip memory accesses under the representation right reuse strategy; MA DRAM_or The output multiplexing strategy represents the total number of off-chip memory accesses; H represents the height of the input feature map; L represents the width of the input feature map; N represents the number of input channels; M represents the number of output channels; R represents the height of the output feature map; C represents the width of the output feature map; T l T represents the block size in the width direction of the input feature map; n Indicates the block size in the input channel direction; T m Indicates the block size in the output channel direction; T h T represents the block size in the height direction of the input feature map. r T represents the block size in the height direction of the output feature map; c Indicates the block size in the width direction of the output feature map; RF I n Indicates the number of times the input data of the nth layer is reused; RF w n Represents the number of times the weight data of the nth layer is reused; RF o n MA represents the number of times the output data of the nth layer is reused. DRAM_IMA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O This indicates the number of off-chip memory accesses for the output data.

[0134] In this embodiment of the invention, the target solution determination module includes: The target strategy determination submodule is used to compare the total number of off-chip memory accesses under the input multiplexing strategy, the total number of off-chip memory accesses under the weighted multiplexing strategy, and the total number of off-chip memory accesses under the output multiplexing strategy for each candidate block partitioning scheme, and select the multiplexing strategy corresponding to the minimum total number of off-chip memory accesses as the target multiplexing strategy of the candidate block partitioning scheme. The target block segmentation scheme determination submodule is used to compare the total number of off-chip memory accesses corresponding to the target reuse strategies of all the candidate block segmentation schemes, and select the candidate block segmentation scheme with the minimum total number of off-chip memory accesses as the target block segmentation scheme of the current convolutional layer.

[0135] In this embodiment of the invention, the data multiplexing module includes: The first multiplexing submodule is used to fix the input data in the storage unit when the target multiplexing strategy is an input multiplexing strategy, so as to perform the multiplexing of the input data; The second reuse submodule is used to fix the weight data in the storage unit when the target reuse strategy is a weight reuse strategy, so as to perform the reuse of the weight data; The third multiplexing submodule is used to send the output data as the input data of the next convolutional layer into the storage unit when the target multiplexing strategy is the output multiplexing strategy, so as to perform the multiplexing of the output data.

[0136] This invention generates multiple candidate block partitioning schemes by acquiring network layer parameters and hardware constraint parameters of multiple convolutional layers in a convolutional neural model. Based on the network layer parameters and the multiple candidate block partitioning schemes, it obtains the number of off-chip memory accesses for input data, weight data, and output data in each of the multiple candidate block partitioning schemes. From the multiple candidate block partitioning schemes, it determines the target block partitioning schemes and corresponding target reuse strategies for each convolutional layer. Data reuse of multiple convolutional layers is then performed according to the target reuse strategy and target block partitioning scheme. This achieves the selection of the optimal block partitioning scheme and optimal reuse strategy for each convolutional layer. It avoids the problem in existing technologies where fixed reuse strategies lead to misjudgments of data transfer in scenarios with ample cache resources, resulting in redundant memory accesses, increased off-chip memory accesses, and ineffective energy consumption. This invention minimizes the number of off-chip memory accesses under different network layer structures and cache configurations, effectively reducing memory access energy consumption and computation latency through data reuse.

[0137] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0138] This invention also provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described data reuse method embodiment of the convolutional neural network and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0139] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0140] This invention also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data reuse method embodiment for convolutional neural networks and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0141] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0143] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0148] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0149] The present invention provides a detailed description of a data reuse method, apparatus, electronic device, and computer-readable storage medium for convolutional neural networks. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data reuse method for convolutional neural networks, characterized in that, include: Obtain the network layer parameters and hardware constraint parameters of multiple convolutional layers in the convolutional neural model; Based on the hardware constraint parameters, multiple candidate block partitioning schemes are generated; Based on the network layer parameters, the number of times the input data is reused, the number of times the weight data is reused, and the number of times the output data is reused for each of the multiple convolutional layers are obtained respectively. Based on the network layer parameters and the multiple candidate block segmentation schemes, the number of memory accesses outside the input data slices, the number of memory accesses outside the weight data slices, and the number of memory accesses outside the output data slices of the multiple candidate block segmentation schemes in the multiple convolutional layers are obtained respectively. Based on the number of input data reuses, the number of weight data reuses, the number of output data reuses, the number of input data accesses outside the memory slice, the number of weight data accesses outside the memory slice, and the number of output data accesses outside the memory slice, the total number of outside memory accesses under the input reuse strategy, the total number of outside memory accesses under the weight reuse strategy, and the total number of outside memory accesses under the output reuse strategy of the multiple convolutional layers are output respectively. Based on the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy, the target block scheme and the corresponding target reuse strategy of the multiple convolutional layers are determined from the multiple candidate block schemes respectively. Data reuse of the multiple convolutional layers is performed according to the target reuse strategy and the target block segmentation scheme.

2. The data reuse method for convolutional neural networks according to claim 1, characterized in that, The hardware constraint parameters include: input buffer capacity, weight buffer capacity, output buffer capacity, and data bit width; the step of generating multiple candidate block partitioning schemes based on the hardware constraint parameters includes: The input feature map, weight feature map, and output feature map of the multiple convolutional layers are obtained respectively. The input feature map of each convolutional layer is divided into blocks to obtain the input blocks of the convolutional layer; the data size of the input blocks is less than or equal to the input buffer capacity of the convolutional layer. The weight feature map of each convolutional layer is divided into blocks to obtain the weight blocks of the convolutional layer; the data size of the weight blocks is less than or equal to the weight cache capacity of the convolutional layer. The output feature map of each convolutional layer is divided into blocks to obtain the output blocks of the convolutional layer; the data size of the output blocks is less than or equal to the output buffer capacity of the convolutional layer. Based on the input blocks, weight blocks, and output blocks of multiple convolutional layers, multiple candidate block segmentation schemes are generated; each candidate block segmentation scheme includes an input block, a weight block, and an output block of a convolutional layer.

3. The data reuse method for convolutional neural networks according to claim 1, characterized in that, The network layer parameters include: input feature map height, input feature map width, number of input channels, number of output channels, convolution kernel size, convolution stride, and load per iteration. The process of obtaining the number of input data reuses, weight data reuses, and output data reuses for the multiple convolutional layers based on these network layer parameters includes: The number of input multiplexing times is obtained based on the kernel size, the stride, and the number of output channels, and the number of input multiplexing times is calculated according to the following formula: The number of weight reuses is obtained based on the input feature map height, the input feature map width, the convolution kernel size, and the convolution stride. The number of weight reuses is calculated using the following formula: The number of output multiplexing operations is obtained based on the convolution kernel size and the number of input channels, and the number of output multiplexing operations is calculated according to the following formula: Among them, RF I n Indicates the number of times the input data of the nth convolutional layer is reused; RF w n Indicates the number of times the weight data of the nth convolutional layer is reused; RF o n K represents the number of times the output data of the nth convolutional layer is reused; n H represents the kernel size of the nth convolutional layer, M represents the number of output channels, and H represents the kernel size of the nth convolutional layer. n L represents the height of the input feature map of the nth convolutional layer. n represents the width of the input feature map of the nth convolutional layer, N represents the number of input channels, and str represents the amount of data loaded each time.

4. The data reuse method for convolutional neural networks according to claim 1, characterized in that, The hardware constraint parameters also include: the maximum data volume in a single burst transmission; the step of obtaining the number of external memory accesses for the input data slice, the number of external memory accesses for the weight data slice, and the number of external memory accesses for the output data slice at the multiple convolutional layers according to the network layer parameters and the multiple candidate block partitioning schemes includes: The number of external memory accesses for the input data slice is obtained based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume in a single burst transmission. The number of external memory accesses for the input data slice is calculated according to the following formula: The number of off-memory accesses for the weighted data slice is obtained based on the candidate block partitioning scheme, the convolution kernel size, and the maximum data volume in a single burst transmission. The number of off-memory accesses for the weighted data slice is calculated according to the following formula: The number of external memory accesses for the output data slice is obtained based on the candidate block partitioning scheme, the current convolutional layer parameters, and the maximum data volume in a single burst transmission. The number of external memory accesses for the output data slice is calculated according to the following formula: Among them, MA DRAM_I MA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O Indicates the number of off-chip memory accesses for the output data; t h Indicates the number of blocks in the height direction, t l Indicates the number of blocks divided in the width direction, t n Indicates the number of blocks segmented in the input channel direction, t m Indicates the number of blocks in the output channel direction, t r Indicates the number of blocks segmented in the output height direction, t c WN represents the number of blocks in the output width direction; M represents the number of output channels; and N represents the number of input channels.

5. The data reuse method for convolutional neural networks according to claim 1, characterized in that, The step of outputting the total off-chip memory accesses under the input reuse strategy, the total off-chip memory accesses under the weight reuse strategy, and the total off-chip memory accesses under the output reuse strategy for the multiple convolutional layers, based on the input data reuse count, the weight data reuse count, the output data reuse count, the input data off-chip memory access count, the weight data off-chip memory access count, and the output data off-chip memory access count, respectively, includes: The total number of off-chip memory accesses under the input multiplexing strategy is obtained using the following formula: The total number of off-chip memory accesses under the weighted reuse strategy is obtained using the following formula: The total number of off-chip memory accesses under the output multiplexing strategy is obtained using the following formula: Among them, MA DRAM_ir MA represents the total number of off-chip memory accesses under the input multiplexing strategy; DRAM_wr Total number of off-chip memory accesses under the representation right reuse strategy; MA DRAM_or The output multiplexing strategy represents the total number of off-chip memory accesses; H represents the height of the input feature map; L represents the width of the input feature map; N represents the number of input channels; M represents the number of output channels; R represents the height of the output feature map; C represents the width of the output feature map; T l T represents the block size in the width direction of the input feature map; n Indicates the block size in the input channel direction; T m Indicates the block size in the output channel direction; T h T represents the block size in the height direction of the input feature map. r T represents the block size in the height direction of the output feature map; c Indicates the block size in the width direction of the output feature map; RF I n Indicates the number of times the input data of the nth layer is reused; RF w n Represents the number of times the weight data of the nth layer is reused; RF o n MA represents the number of times the output data of the nth layer is reused. DRAM_I MA represents the number of off-chip memory accesses for the input data. DRAM_W MA represents the number of off-chip memory accesses for weighted data. DRAM_O This indicates the number of off-chip memory accesses for the output data.

6. The data reuse method for convolutional neural networks according to claim 1, characterized in that, The step of determining the target block scheme and corresponding target reuse strategy for the multiple convolutional layers from the multiple candidate block schemes based on the total off-chip memory accesses under the input reuse strategy, the total off-chip memory accesses under the weighted reuse strategy, and the total off-chip memory accesses under the output reuse strategy includes: For each candidate block partitioning scheme, compare the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy, and select the reuse strategy corresponding to the minimum total number of off-chip memory accesses as the target reuse strategy of the candidate block partitioning scheme. Compare the total number of off-chip memory accesses corresponding to the target reuse strategies of all the candidate block segmentation schemes, and select the candidate block segmentation scheme with the minimum total number of off-chip memory accesses as the target block segmentation scheme of the current convolutional layer.

7. The data reuse method for convolutional neural networks according to claim 1, characterized in that, The process of reusing the data of the multiple convolutional layers according to the target reuse strategy and the target chunking scheme includes: When the target reuse strategy is an input reuse strategy, the input data is fixed in the storage unit to perform the reuse of the input data; When the target reuse strategy is a weight reuse strategy, the weight data is fixed in the storage unit to perform the reuse of the weight data; When the target reuse strategy is the output reuse strategy, the output data is sent to the storage unit as the input data of the next convolutional layer to perform the reuse of the output data.

8. A data multiplexing device for convolutional neural networks, characterized in that, include: The parameter acquisition module is used to acquire the network layer parameters and hardware constraint parameters of multiple convolutional layers in the convolutional neural model. The candidate scheme generation module is used to generate multiple candidate block schemes based on the hardware constraint parameters. The reuse count determination module is used to obtain the number of times the input data is reused, the number of times the weight data is reused, and the number of times the output data is reused for the multiple convolutional layers according to the network layer parameters. The off-chip memory access count determination module is used to obtain, based on the network layer parameters and the multiple candidate block partitioning schemes, the off-chip memory access counts of the input data, the off-chip memory access counts of the weight data, and the off-chip memory access counts of the output data of the multiple candidate block partitioning schemes in the multiple convolutional layers, respectively. The total off-chip memory access count determination module is used to output the total off-chip memory access count under the input reuse strategy, the total off-chip memory access count under the weight reuse strategy, and the total off-chip memory access count under the output reuse strategy of the multiple convolutional layers, based on the input data reuse count, the weight data reuse count, the output data reuse count, the input data off-chip memory access count, the weight data off-chip memory access count, and the output data off-chip memory access count, respectively. The target scheme determination module is used to determine the target block scheme and corresponding target reuse strategy of the multiple convolutional layers from the multiple candidate block schemes based on the total number of off-chip memory accesses under the input reuse strategy, the total number of off-chip memory accesses under the weighted reuse strategy, and the total number of off-chip memory accesses under the output reuse strategy. The data reuse module is used to perform data reuse of the multiple convolutional layers according to the target reuse strategy and the target block partitioning scheme.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory, wherein when the program or instructions are executed by the processor, they implement the data reuse steps of the convolutional neural network as described in claims 1-7.

10. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, they implement the data reuse steps of the convolutional neural network according to claims 1-7.

Images