Automatic synthesis method of in-memory computing oriented convolutional neural network accelerator architecture

Abstract

The application provides an automatic synthesis method of a convolutional neural network accelerator for in-memory computing, which can obtain a convolutional neural network weight replication candidate strategy under each parameter combination by using an analog annealing method based on a convolutional neural network inference task, a peak power consumption constraint of an in-memory computing accelerator and a hardware parameter look-up table of the in-memory computing accelerator, obtain a macro cell mapping scheme with optimal performance by using a genetic method, and obtain a convolutional neural network weight replication candidate strategy, a hardware parameter combination of the in-memory computing accelerator and a hardware resource allocation scheme corresponding to the scheme. The method provided by the application greatly reduces the professional threshold of architecture design, improves the efficiency of in-memory computing accelerator design, saves the cost of expert design, and effectively enriches the optimization strategy of accelerator architecture configuration scheme to guarantee the design quality of the in-memory accelerator configuration scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of high-efficiency computing and automation design, in particular to the field of in-memory computing accelerator, convolutional neural network inference and automatic synthesis, and more particularly to an automatic synthesis method for an in-memory computing-oriented convolutional neural network accelerator architecture. BACKGROUND

[0002] In the traditional von Neumann architecture, the separation of computation and storage results in the need to load data from storage to computing units for each operation, which brings a large amount of energy consumption and performance loss to convolutional neural network (CNN) computation. However, unlike the traditional von Neumann architecture, in-memory computing (PIM) binds computation and storage units through new non-volatile memory devices (NVM), such as resistive random-access memories (ReRAM), effectively reducing data access in the accelerator architecture and alleviating the performance bottleneck problem caused by the current "memory wall". In addition, compared with the traditional CMOS-based convolutional neural network accelerator, the in-memory computing accelerator used for convolutional neural network inference can improve the energy efficiency by 2 to 3 orders of magnitude, and has great potential to optimize the energy efficiency of convolutional neural network inference.

[0003] The existing in-memory computing accelerator architecture configuration schemes for reasoning convolutional neural networks are mostly artificially designed, and usually have the following three characteristics: (1) inter-layer computation flow, that is, in the in-memory computing accelerator, the weights of each network layer in the convolutional neural network are written in the form of resistance into the ReRAM array in advance, wherein the weights of any network layer can be copied and stored in the ReRAM array (i.e., the "weight replication" strategy). Each network layer is executed in a pipeline manner in parallel, effectively improving the parallelism of convolutional neural network reasoning. (2) Communication becomes the key performance bottleneck, that is, the in-memory computing accelerator is usually composed of multiple macro units (i.e., Macro) interconnected by a Network-on-Chip (NoC) or a bus, wherein each macro unit is usually composed of a storage unit, a computing array and a peripheral circuit. Although the in-memory computing accelerator avoids the communication overhead of the weights of the convolutional neural network, there is still a large amount of communication demand for input and intermediate activation within the macro unit and between the macro units in the in-memory computing accelerator, which makes communication the key performance bottleneck. (3) The peripheral circuit (also known as peripheral functional component) becomes the main source of power consumption, that is, in the in-memory computing accelerator, the power consumption of the peripheral circuit such as the Digital-to-Analog Converter (DAC) and the Analog-to-Digital Converter can account for more than 60% of the total power consumption.

[0004] In summary, the existing in-memory computing accelerator architecture configuration schemes for reasoning convolutional neural networks have the following disadvantages: (1) high design cost, the in-memory computing accelerator architecture configuration schemes for convolutional neural network reasoning currently mostly rely on artificial design, which usually requires rich architecture design experience and sufficient time, resulting in high design cost of the in-memory computing accelerator architecture configuration scheme; (2) great design difficulty, when the convolutional neural network reasoning task is complex, the size of the design space of the corresponding in-memory computing accelerator architecture configuration scheme becomes larger, greatly increasing the design difficulty, and it is almost impossible to design an accelerator architecture configuration scheme with optimal area, power consumption and performance only by relying on the experience of experts; (3) the design quality is difficult to guarantee, the in-memory computing accelerator architecture configuration schemes for convolutional neural network reasoning currently lack comprehensive optimization and exploration of the accelerator architecture, and cannot realize the specific mapping method from the convolutional neural network reasoning task to the in-memory computing accelerator architecture to realize the accelerated reasoning of the convolutional neural network, therefore, the design quality of the in-memory computing accelerator architecture configuration scheme for convolutional neural network reasoning currently still has a lot of room for improvement.

[0005] It should be noted that: the background technology is only used to introduce the related information of the application, so as to help understand the technical scheme of the application, but it does not mean that the related information must be prior art. In the absence of evidence that the related information has been disclosed before the application date of the application, the related information should not be regarded as prior art. SUMMARY

[0006] Therefore, the purpose of the present application is to overcome the defects of the prior art, to provide a convolutional neural network inference scheme search method based on in-memory computing accelerator.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] According to the first aspect of the present application, an automatic synthesis method for an in-memory computing-oriented convolutional neural network accelerator architecture is provided, the in-memory computing accelerator includes a plurality of macro units, the method comprises: S1, obtaining a convolutional neural network inference task, an in-memory computing accelerator peak power consumption constraint and a hardware parameter lookup table of the in-memory computing accelerator, wherein the hardware parameter lookup table of the in-memory computing accelerator includes a plurality of design parameters of a non-volatile memory and a plurality of accuracies of a digital-to-analog converter; S2, constructing a plurality of optional parameter combinations based on the plurality of design parameters of the non-volatile memory, and determining the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator based on each parameter combination and the in-memory computing accelerator peak power consumption constraint; S3, based on the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator determined in the step S2 and the convolutional neural network inference task, a first preset method is used to filter out a first preset number of convolutional neural network weight replication candidate strategies corresponding to each parameter combination; S4, based on each accuracy of the digital-to-analog converter and each convolutional neural network weight replication candidate strategy, a convolutional neural network inference data flow directed acyclic graph corresponding to each convolutional neural network weight replication candidate strategy is compiled; S5, based on each convolutional neural network inference data flow directed acyclic graph, the number of macro units occupied by each network layer in the convolutional neural network and the number of macro units shared between network layers are determined to obtain a macro unit mapping scheme corresponding to each convolutional neural network inference data flow directed acyclic graph; S6, a second preset method is used to perform performance evaluation on each macro unit mapping scheme, and the macro unit mapping scheme with the best performance is selected based on the performance evaluation result to obtain the convolutional neural network weight replication candidate strategy, the in-memory computing accelerator hardware parameter combination and the hardware resource allocation scheme corresponding to the scheme, which are used to determine the in-memory accelerator architecture configuration to realize the convolutional neural network acceleration inference, wherein the in-memory computing accelerator hardware parameter combination includes a plurality of design parameters of a non-volatile memory and an accuracy of a digital-to-analog converter.

[0009] In some embodiments of the present application, the plurality of design parameters of the non-volatile memory comprises: a proportion of power consumption of the non-volatile memory to a peak power consumption constraint of the in-memory computing accelerator, a precision of the non-volatile memory unit, and a size of the non-volatile memory array, and in the step S2, the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator is determined in the following manner:

[0010]

[0011] wherein #crossbars represents the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator, TotalPower represents the peak power consumption constraint of the in-memory computing accelerator, RatioRram represents the proportion of power consumption of the non-volatile memory to the peak power consumption constraint of the in-memory computing accelerator, CrossbarPower(·) represents a power consumption function of the non-volatile memory array, XbSize represents the size of the non-volatile memory array, and ResRram represents the precision of the non-volatile memory unit.

[0012] In some embodiments of the present application, the convolutional neural network inference task comprises: weights, input channel number, output channel number, and weight kernel size of each network layer in the convolutional neural network, and in the step S3, the first preset method is simulated annealing method, and the simulated annealing method is used to screen out the first preset number of convolutional neural network weight duplication candidate strategies corresponding to each parameter combination in the following manner:

[0013]

[0014] wherein WtDup represents the convolutional neural network weight duplication candidate strategy corresponding to each parameter combination, Performance(·) is an energy function of simulated annealing, WrDup i represents the weight duplication candidate strategy of the i-th network layer in the convolutional neural network under each parameter combination, set i represents the number of non-volatile memory arrays required to duplicate the weight of the i-th network layer in the convolutional neural network under each parameter combination, and

[0015]

[0016]

[0017]

[0018] represents the weight kernel size of the i-th network layer in the convolutional neural network, represents the number of input channels of the i-th network layer in the convolutional neural network, represents the number of output channels of the i-th network layer in the convolutional neural network, PrecWt represents the weight precision, and stdev(·) is a standard deviation function, represents the standard deviation of the computation time of all network layers in the convolutional neural network under each parameter combination, and a is a preset empirical parameter, represents the standard deviation of the data access amount of all network layers in the convolutional neural network under each parameter combination.

[0019] In some embodiments of the present application, the convolutional neural network inference data flow directed acyclic graph corresponding to each convolutional neural network weight replication candidate strategy is compiled in the following manner: based on each precision of the digital-to-analog converter, each convolutional neural network weight replication candidate strategy, and a preset computation rule of the convolutional neural network, the inferences of the network layers in the convolutional neural network are respectively compiled into a group of operation nodes; the dependency relationship between the operation nodes of the inferences of all network layers in the convolutional neural network is determined based on a preset dependency relationship rule between the operation nodes; and the convolutional neural network inference data flow directed acyclic graph corresponding to each convolutional neural network weight replication candidate strategy is generated based on the dependency relationship between the operation nodes of the inferences of all network layers in the convolutional neural network.

[0020] In some embodiments of the present application, the second preset method is a genetic method, and the performance of each macro cell mapping scheme is evaluated to obtain the macro cell mapping scheme with the optimal performance in the following manner using the genetic method: S61, each macro cell mapping scheme is taken as an individual in a preset macro cell mapping scheme population; S62, the preset macro cell mapping scheme population is randomly initialized and each individual in the population is evaluated for performance, wherein the hardware resource allocation scheme corresponding to each convolutional neural network inference data flow directed acyclic graph is obtained using the third preset method when each individual is evaluated for performance; S63, a second preset number of individuals with the optimal performance are selected as parents based on the performance evaluation values of each individual in the preset macro cell mapping scheme population; S64, new individuals are generated as offspring after the parents are mutated and the offspring are evaluated for performance; S65, the parents and the offspring are combined to form a new macro cell mapping scheme population, and steps S62-S64 are repeatedly executed for a preset number of iterations of the genetic method to search for the macro cell mapping scheme with the optimal performance.

[0021] In some embodiments of the present invention, S62.1, based on the preset macro unit mapping scheme, each individual in the population improves each convolutional neural network inference data flow directed acyclic graph; S62.2, using the preset third method to design a corresponding hardware resource allocation scheme for each improved convolutional neural network inference data flow directed acyclic graph; S62.3, based on each improved convolutional neural network inference data flow directed acyclic graph and its corresponding hardware allocation scheme, and the hardware parameter lookup table of the in-memory computing accelerator, the performance of each individual in the population is evaluated.

[0022] In some embodiments of the present invention, the hardware parameter lookup table of the in-memory computing accelerator further includes: the operating frequency and power of each peripheral functional component, wherein the peripheral functional components in the in-memory computing accelerator include at least: an arithmetic logic unit, an analog-to-digital converter, and a digital-to-analog converter, and in step S62.2, the preset third method is a heuristic method, and the heuristic method is used to design a corresponding hardware resource allocation scheme for each improved convolutional neural network inference data flow directed acyclic graph in the following manner:

[0023] 1≤i≤L,c∈peripheral functional components

[0024]

[0025] in, Freq represents the computational workload that the peripheral functional component c needs to perform when reasoning about a single operation node of the i-th layer in a convolutional neural network. c Indicates the operating frequency of the peripheral functional component c, P c represents the power of the peripheral functional component c, represents the number of peripheral functional components c assigned to the i-th network layer in the convolutional neural network.

[0026] In some embodiments of the present application, the performance of each individual in the population is evaluated in the following manner: topological sorting is performed on the perfected each convolutional neural network inference data flow directed acyclic graph to determine the number of calculation cycles of the convolutional neural network inference; determining the inference delay of each operation node in the perfected each convolutional neural network inference data flow directed acyclic graph based on the hardware allocation scheme corresponding to the perfected each convolutional neural network inference data flow directed acyclic graph, wherein the maximum inference delay of the operation node is taken as the delay of a single calculation cycle of the convolutional neural network inference; determining the energy consumption of the peripheral functional components in the in-memory computing accelerator based on the perfected each convolutional neural network inference data flow directed acyclic graph, the hardware parameter lookup table of the in-memory computing accelerator, and the power consumption and delay parameters of the peripheral functional components in the hardware parameter lookup table; determining the energy consumption of the non-volatile memory in the in-memory computing accelerator based on each convolutional neural network weight replication candidate strategy and the power consumption and delay parameters of the non-volatile memory under each combination parameter in the hardware parameter lookup table; determining the inference delay of the convolutional neural network based on the number of calculation cycles of the convolutional neural network inference and the delay of a single calculation cycle; and determining the energy efficiency of the in-memory computing accelerator architecture based on the energy consumption of the peripheral functional components and the non-volatile memory in the in-memory computing accelerator, the inference delay of the convolutional neural network, the capacity and bandwidth of the cache in the hardware parameter lookup table of the in-memory computing accelerator, and the power consumption and bandwidth of the network on chip to achieve performance evaluation of each individual in the population.

[0027] According to a second aspect of the present application, a convolutional neural network inference method is provided, which comprises: obtaining the macro unit mapping scheme with optimal performance and the convolutional neural network weight replication candidate strategy, the in-memory computing accelerator hardware parameter combination, and the hardware resource allocation scheme corresponding to the scheme obtained by the method in the above embodiments; and determining the in-memory computing accelerator architecture configuration based on the macro unit mapping scheme with optimal performance and the convolutional neural network weight replication candidate strategy, the in-memory computing accelerator hardware parameter combination, and the hardware resource allocation scheme corresponding to the scheme to implement the inference of the convolutional neural network.

[0028] According to a third aspect of the present application, there is provided an in-memory computing oriented convolutional neural network accelerator architecture search system, comprising: a design space search module configured to store a convolutional neural network inference task, an in-memory computing accelerator peak power consumption constraint, and a hardware parameter lookup table of an in-memory computing accelerator, wherein the hardware parameter lookup table of the in-memory computing accelerator comprises a plurality of design parameters of a non-volatile memory and a plurality of precisions of a digital-to-analog converter, and a plurality of selectable parameter combinations are constructed based on the plurality of design parameters of the non-volatile memory, and an array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator is determined based on each parameter combination and the in-memory computing accelerator peak power consumption constraint; a weight replication module configured to filter out a first preset number of convolutional neural network weight replication candidate strategies corresponding to each parameter combination using a simulated annealing method based on the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator of the design space search module and the convolutional neural network inference task; a data flow compilation module configured to compile a convolutional neural network inference data flow directed acyclic graph corresponding to each convolutional neural network weight replication candidate strategy based on each precision of the digital-to-analog converter and each convolutional neural network weight replication candidate strategy filtered out by the weight replication module; a macro cell mapping module configured to determine a number of macro cells occupied by each network layer in the convolutional neural network and a number of macro cells shared between network layers based on each convolutional neural network inference data flow directed acyclic graph obtained by the data flow compilation module to obtain a macro cell mapping scheme corresponding to each convolutional neural network inference data flow directed acyclic graph; and a hardware allocation module configured to perform performance evaluation on each macro cell mapping scheme using a genetic method, and select a macro cell mapping scheme with optimal performance based on a performance evaluation result to obtain a convolutional neural network weight replication candidate strategy, an in-memory computing accelerator hardware parameter combination, and a hardware resource allocation scheme corresponding to the scheme, which are used to determine an in-memory accelerator architecture configuration to implement convolutional neural network acceleration inference, wherein the in-memory computing accelerator hardware parameter combination comprises a plurality of design parameters of a non-volatile memory and a precision of a digital-to-analog converter.

[0029] According to a fourth aspect of the present application, there is provided an electronic device, comprising: one or more processors; and a storage device storing one or more programs, which when executed by the one or more processors, cause the electronic device to implement the steps of the method of the first aspect and the third aspect.

[0030] Compared with the prior art, the present application has the following advantages:

[0031] The application utilizes a convolutional neural network inference scheme search method based on an in-memory computing accelerator to generate an in-memory computing accelerator architecture configuration scheme, greatly reduces the professional threshold of architecture design, improves the efficiency of in-memory computing accelerator design, saves the cost of expert design, and effectively enriches the optimization strategy of accelerator architecture configuration scheme to guarantee the design quality of the in-memory accelerator configuration scheme. Further, the in-memory computing accelerator designed by the generated in-memory computing accelerator architecture configuration scheme can improve the inference efficiency of the convolutional neural network. BRIEF DESCRIPTION OF DRAWINGS

[0032] The embodiments of the application will be further described below with reference to the drawings, in which:

[0033] Figure 1 It is a hardware component structure framework diagram of an in-memory computing accelerator;

[0034] Figure 2 It is a non-volatile memory array structure diagram;

[0035] Figure 3 It is a flowchart of an automatic synthesis method of an in-memory computing-oriented convolutional neural network accelerator skeleton according to the embodiment of the application;

[0036] Figure 4 It is a part of data flow diagram in a convolutional neural network corresponding data flow directed acyclic graph;

[0037] Figure 5 It is a framework diagram of an in-memory computing-oriented convolutional neural network accelerator architecture search system according to the embodiment of the application;

[0038] Figure 6 It is a pseudo code description flowchart of obtaining a performance-optimal in-memory computing accelerator architecture configuration scheme according to the embodiment of the application;

[0039] Figure 7 It is a pseudo code flowchart of obtaining an optimal macro cell mapping scheme using a genetic method according to the embodiment of the application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0041] As described in the background, the existing in-memory computing accelerator architecture configuration scheme for reasoning convolutional neural network has the following disadvantages: (1) high design cost, most of the in-memory computing accelerator architecture configuration schemes for convolutional neural network reasoning currently rely on manual design, which usually requires rich architecture design experience and sufficient time, resulting in high design cost of in-memory computing accelerator architecture configuration scheme; (2) high design difficulty, when the convolutional neural network reasoning task is complex, the design space of the corresponding in-memory computing accelerator architecture configuration scheme becomes larger, greatly increasing the design difficulty, and it is almost impossible to design an accelerator architecture configuration scheme with optimal area, power consumption and performance by relying only on the experience of experts; (3) the design quality is difficult to guarantee, the current in-memory computing accelerator architecture configuration scheme for convolutional neural network reasoning lacks comprehensive optimization and exploration of the accelerator architecture, and cannot realize the mapping method from the convolutional neural network reasoning task to the in-memory computing accelerator architecture to realize the accelerated reasoning of the convolutional neural network, therefore, the design quality of the current in-memory computing accelerator architecture configuration scheme for convolutional neural network reasoning still has a lot of room for improvement.

[0042] In order to solve the above problems, the inventors design an in-memory computing accelerator based convolutional neural network reasoning scheme search scheme, which can directly generate an in-memory computing accelerator architecture configuration scheme with optimal performance by mapping from a convolutional neural network reasoning task to an in-memory computing accelerator architecture to realize the accelerated reasoning of the convolutional neural network, greatly reducing the design cost and difficulty of the in-memory computing accelerator architecture configuration scheme, while effectively enriching the optimization strategy of the accelerator architecture configuration scheme to guarantee the design quality of the in-memory accelerator configuration scheme. Further, designing an in-memory computing accelerator by using the generated in-memory computing accelerator architecture configuration scheme can improve the reasoning efficiency of the convolutional neural network.

[0043] Before specifically introducing the embodiments of the present application, the hardware composition structure of the in-memory computing accelerator and the definition of the variables used in the formulas involved in the present application are introduced in detail.

[0044] I. In-memory computing accelerator

[0045] As shown in Figure 1 , it shows the hardware composition structure of the in-memory computing accelerator, which is composed of three layers of topology structure of macro unit (Macro), computing unit (PE) and non-volatile memory array (such as ReRAM array). As can be seen from Figure 1 (a), the in-memory computing accelerator includes a plurality of macro units, each macro unit is connected to each other through on-chip network, the calculation of each network layer is mapped to different macro unit group, and is executed in parallel in the form of network layer pipeline, and the communication between each network layer will cause the synchronization between each macro unit.Figure 1 As shown in (b), each macrocell includes: a buffer, a processing element (PE), an analog-to-digital converter (ADC), an arithmetic logic unit (ALU), a register, and a controller. Figure 1 As shown in (c), the processing element includes: an input register, a digital-to-analog converter (DAC), a non-volatile memory array, a sample-and-hold unit (S&H), and a multiplexer.

[0046] In addition, in the inference task of the convolutional neural network, the functions performed by each component in the above-mentioned in-memory computing accelerator are as follows: the PE is responsible for performing a matrix-vector multiplication operation and generating an analog result and outputting, wherein the analog result of the PE is converted into a digital signal by the ADC group, and further subsequent vector operations (such as shift addition and pooling, etc.) are completed by the ALU component. As shown in (d), a structural diagram of a non-volatile memory array is shown, wherein the NVM array is usually composed of NVM cells (such as ReRAM) in the form of a cross array. Figure 2 As shown in (d), a structural diagram of a non-volatile memory array is shown, wherein the NVM array is usually composed of NVM cells (such as ReRAM) in the form of a cross array.

[0047] II. Variable definition

[0048] As shown in Table 1, the partial variable definitions used in the formula are shown.

[0049] Table 1

[0050]

[0051] In order to better understand the present application, the present application will be described in detail below according to the drawings and specific embodiments.

[0052] According to one embodiment of the present application, the present application proposes an automatic synthesis method for an in-memory computing-oriented convolutional neural network accelerator architecture, as shown in (a), which includes steps S1-S6. Steps S1-S6 will be described in detail below. Figure 3 As shown in (a), the method includes steps S1-S6. Steps S1-S6 will be described in detail below.

[0053] In the step S1, a convolutional neural network inference task, an in-memory computing accelerator peak power consumption constraint, and a hardware parameter lookup table of the in-memory computing accelerator are obtained, wherein the hardware parameter lookup table of the in-memory computing accelerator includes a plurality of design parameters of a non-volatile memory and a plurality of precisions of a digital-to-analog converter. The plurality of precisions of the digital-to-analog converter includes 1 bit, 2 bits, and 4 bits.

[0054] In the step S2, a plurality of optional parameter combinations are constructed based on a plurality of design parameters of the non-volatile memory, and an array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator is determined based on each parameter combination and an in-memory computing accelerator peak power consumption constraint. According to an embodiment of the present application, the plurality of design parameters of the non-volatile memory include: a proportion of power consumption of the non-volatile memory to the in-memory computing accelerator peak power consumption constraint, an accuracy of the non-volatile memory unit, and a size of the non-volatile memory array, and in the step S2, the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator is determined in the following manner:

[0055]

[0056] wherein #crossbars represents the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator, TotalPower represents the in-memory computing accelerator peak power consumption constraint, RatioRram represents the proportion of power consumption of the non-volatile memory to the in-memory computing accelerator peak power consumption constraint, CrossbarPower(·) represents a power consumption function of the non-volatile memory array, XbSize represents the size of the non-volatile memory array, and ResRram represents the accuracy of the non-volatile memory unit.

[0057] Illustratively, taking the non-volatile memory ReRAM as an example, the plurality of design parameters of the ReRAM include: a proportion of power consumption of the ReRAM to the in-memory computing accelerator peak power consumption constraint, an accuracy of the ReRAM unit, and a size of the ReRAM array, wherein the proportion of power consumption of the ReRAM to the in-memory computing accelerator peak power consumption constraint is set to be in the range of 0.1-0.4; the design enumeration values of the accuracy of the ReRAM unit are 1 bit, 2 bits and 4 bits; and the design enumeration values of the size of the ReRAM array are 128, 256 and 512. It should be noted that the present application only takes the ReRAM as an example to describe the scheme, and the non-volatile memory can also be FRAM, MRAM, PCRAM or STT-RAM, etc. Further, with the development of non-volatile memory, a non-volatile memory with better performance can also be selected for design, which is not limited here.

[0058] In the step S3, based on the number of non-volatile memory arrays corresponding to each parameter combination in the in-memory computing accelerator and the convolutional neural network inference task determined in the step S2, a first preset number of convolutional neural network weight duplication candidate strategies corresponding to each parameter combination are screened out by using a first preset method. The convolutional neural network weight duplication candidate strategy is the weight duplication multiple of each network layer in the convolutional neural network, and the number of non-volatile memory arrays occupied by each network layer in the convolutional neural network in the in-memory computing can be obtained through the convolutional neural network weight duplication candidate strategy.

[0059] Further, the weights of each network layer in the convolutional neural network share all the non-volatile memory arrays (such as ReRAM arrays) in the in-memory computing accelerator. Therefore, solving the convolutional neural network weight duplication candidate strategy is equivalent to solving the allocation problem of the non-volatile memory arrays among each network layer in the convolutional neural network. Therefore, according to an embodiment of the present application, the convolutional neural network inference task includes the weights, the input channel number, the output channel number, and the weight kernel size of each network layer in the convolutional neural network, and in the step S3, the first preset method is the simulated annealing method, and the simulated annealing method is used to screen out the first preset number of convolutional neural network weight duplication candidate strategies corresponding to each parameter combination in the following manner:

[0060]

[0061] Wherein, WtDup represents the convolutional neural network weight duplication candidate strategy corresponding to each parameter combination, PerFormance(·) is the energy function of simulated annealing, WtDup i represents the weight duplication candidate strategy of the i-th network layer in the convolutional neural network under each parameter combination, set i represents the number of non-volatile memory arrays required to occupy one copy of the weight of the i-th network layer in the convolutional neural network under each parameter combination, and

[0062]

[0063]

[0064]

[0065] represents the weight kernel size of the i-th network layer in the convolutional neural network, represents the input channel number of the i-th network layer in the convolutional neural network, represents the output channel number of the i-th network layer in the convolutional neural network, PrecWt represents the weight precision, and stdev(·) is the standard deviation function, denotes the standard deviation of the computation time of all network layers in the convolutional neural network under each parameter combination, and a is a preset empirical parameter, denotes the standard deviation of the data access amount of all network layers in the convolutional neural network under each parameter combination. It should be noted that the above embodiment uses the simulated annealing method to solve the optimization problem, the weight duplication strategy is a feasible solution of the simulated annealing method, the simplified performance function is an energy function of the simulated annealing, and after multiple iterations of random disturbance, evaluation and selection of the feasible solution, 30 convolutional neural network weight duplication candidate strategies with the best performance are screened out.

[0066] In the step S4, based on each precision of the digital-to-analog converter and each convolutional neural network weight duplication candidate strategy, a directed acyclic graph of the convolutional neural network inference data stream corresponding to each convolutional neural network weight duplication candidate strategy is compiled. According to an embodiment of the present application, the process of compiling the directed acyclic graph of the convolutional neural network inference data stream corresponding to each convolutional neural network weight duplication candidate strategy is as follows: based on each precision of the digital-to-analog converter, each convolutional neural network weight duplication candidate strategy and a preset computation rule of the convolutional neural network, the inferences of the network layers in the convolutional neural network are respectively compiled into a group of operation nodes; the dependency relationship between the operation nodes of the inferences of all network layers in the convolutional neural network is determined based on a preset dependency relationship rule between the operation nodes; and the directed acyclic graph of the convolutional neural network inference data stream corresponding to each convolutional neural network weight duplication candidate strategy is generated based on the dependency relationship between the operation nodes of the inferences of all network layers in the convolutional neural network.

[0067] Illustratively, the operation nodes involved in the present application are shown in Table 2, and the computations of the network layers in the convolutional neural network are respectively compiled into a group of operation nodes / intermediate representations (IR). Specifically, first, any operation node of any network layer in the convolutional neural network is defined by three parameters, namely layer, cnt and bit. Layer represents the network layer to which the operation node belongs; cnt represents the serial number of the calculation block to which the operation node belongs; and bit represents the bit position executed by the current operation node. Second, the dependency relationship between the generated operation nodes is determined, wherein the dependency relationship between the operation nodes can be mainly divided into four types, namely inter-layer dependency, inter-calculation block dependency, inter-bit position dependency and inter-operation dependency. Among them, the inter-layer dependency is determined by the topology of the convolutional neural network; the inter-calculation block dependency and the inter-bit position dependency conform to the sequential execution dependency, i.e. the cnt+1 calculation block depends on the cnt calculation block, and the bit+1 execution depends on the bit execution; and the inter-operation dependency is responsible for defining the execution order of the operation nodes in a calculation block, such as Load-MVM-ADC-ALU-Store. Figure 4As shown, it shows an example of part of the data flow in the directed acyclic graph of the data flow corresponding to the convolutional neural network, from Figure 4 As can be seen in the figure, each node represents an operation, which can be described by a corresponding operation node. The color of the operation node indicates the type of operation node (e.g., MVM, ADC, etc.), the sequence number of the operation node indicates the sequence number of the computation block to which it belongs, and the depth of the operation node color indicates the order of the bits to which it belongs. In addition, the inter-block pipeline reflects the dependencies between computation blocks. As can be seen in the figure, inter-block dependency reflects the dependencies between adjacent nodes of the same type; similarly, inter-bit dependency reflects the dependencies between adjacent bits of the same type of operation nodes belonging to the same computation block.

[0068] Table 2

[0069]

[0070] It should be noted that the macro units of different network layers in the convolutional neural network can use the same or different configurations. When performing convolutional neural network inference, the weights of each network layer will be written into the NVM unit, and the weights from the same weight core will be mapped to the same column of one or more NVM arrays. A network layer requires a total of C O Column and W K ×W K ×C I Considering the limitation of NVM array size and NVM unit precision, mapping the weight of a convolution layer usually requires set NVM arrays, as shown in the calculation formula of the number of non-volatile memory arrays in the in-memory computing accelerator. The calculation of one network layer requires W K ×W K ×C I The input is loaded onto the array word line in the form of voltage, and after O(1) time, C o Output. When the network layer weight replication multiple is WtDup, WtDup×C can be obtained by parallel calculation. O Output. The present invention will generate WtDup×C O The calculation process of an output is called a "computation block". Due to the limited accuracy of the DAC, obtaining a calculation block usually requires multiple iterations of the input bit level. The different bits of the input activation are loaded onto the array word line in a serial manner, and the number of iterations required is determined by the ratio of the input activation accuracy to the DAC accuracy. In addition, the weight of a layer of network is replicated WtDup times, which means that one calculation can obtain WtDup×C O output. The computation time of a network layer can be calculated by W O ×H O / WtDup is roughly estimated.O , H O are width and height of the network layer output feature map, respectively.

[0071] In the step S5, the number of macro units occupied by each network layer in the convolutional neural network and the number of macro units shared between network layers are determined based on each convolutional neural network inference data flow directed acyclic graph to obtain a macro unit mapping scheme corresponding to each convolutional neural network inference data flow directed acyclic graph.

[0072] In the step S6, a second preset method is used to perform performance evaluation on each macro unit mapping scheme, and the macro unit mapping scheme with the optimal performance is selected based on the performance evaluation result to obtain a convolutional neural network weight duplication candidate strategy, an in-memory computing accelerator hardware parameter combination and a hardware resource allocation scheme corresponding to the scheme, which are used to determine an in-memory accelerator architecture configuration to implement convolutional neural network accelerated inference. The in-memory computing accelerator hardware parameter combination includes a plurality of design parameters of the non-volatile memory and the accuracy of the digital-to-analog converter. According to an embodiment of the present application, the second preset method is a genetic method, and the genetic method is used to perform steps S61-S65 to obtain the macro unit mapping scheme with the optimal performance. Steps S61-S65 are described in detail below.

[0073] In the step S61, each macro unit mapping scheme is taken as an individual in a preset macro unit mapping scheme population. Illustratively, each macro unit mapping scheme MacAlloc is taken as an individual in a genetic algorithm population, where MacAlloc i = i x 1000 + #Macro i When the hth layer and the ith layer network layer (j < i) in the convolutional neural network share the same set of macro units, MacAlloc i = j x 1000 + #Macro i , where #Macro i represents the number of macro units allocated to the ith layer network layer i in the convolutional neural network.

[0074] In the step S62, the preset macro unit mapping scheme population is randomly initialized and performance evaluation is performed on each individual in the population. When performance evaluation is performed on each individual, a third preset method is used to obtain a hardware resource allocation scheme corresponding to each convolutional neural network inference data flow directed acyclic graph. Further, when performance evaluation is performed on each macro unit mapping scheme, complete architecture implementation details of the in-memory computing accelerator under the mapping scheme need to be obtained. To this end, according to an embodiment of the present application, steps S62.1-S62.3 are performed to implement performance evaluation on each macro unit mapping scheme. Steps S62.1-S62.3 are described in detail below.

[0075] In step S62.1, each convolutional neural network inference data flow directed acyclic graph is improved based on each individual in the population of the preset macro unit mapping scheme. Specifically, the memory access operations within and between macro units of each network layer in the convolutional neural network are determined based on the macro unit mapping scheme, and the parameters of the intra-macro communication and the parameters of the inter-macro communication of each network layer in the convolutional neural network are further determined to improve the convolutional neural network inference data flow directed acyclic graph obtained in step S4.

[0076] In step S62.2, a preset third method is used to design a corresponding hardware resource allocation scheme for each improved convolutional neural network inference data flow directed acyclic graph. According to one embodiment of the present invention, the hardware parameter lookup table of the in-memory computing accelerator also includes: the operating frequency and power of each peripheral functional component, wherein the peripheral functional components in the in-memory computing accelerator include at least: an arithmetic logic unit, an analog-to-digital converter, and a digital-to-analog converter. The preset third method is a heuristic method, and the heuristic method is used to design a corresponding hardware resource allocation scheme for each improved convolutional neural network inference data flow directed acyclic graph in the following manner:

[0077] 1≤i≤L,c∈peripheral functional components

[0078]

[0079] in, Freq represents the computational workload that the peripheral functional component c needs to perform when reasoning about a single operation node of the i-th layer in a convolutional neural network. c Indicates the operating frequency of the peripheral functional component c, P c represents the power of the peripheral functional component c, represents the number of peripheral functional components c assigned to the i-th network layer in the convolutional neural network.

[0080] From the above, we can see that under the peak power consumption constraint of the in-memory computing accelerator, the optimal hardware resource allocation scheme obtained by the above formula needs to balance the delay of each operation node to the greatest extent. The hardware resource allocation scheme is mainly responsible for allocating a certain number of peripheral functional components, such as DAC, ADC, and ALU, to each computing IR in the directed acyclic graph of the convolutional neural network inference data flow. The delay of the computing IR related to the peripheral functional component c of the i-th network layer in the convolutional neural network can be calculated by Calculated.

[0081] In the step S62.3, the performance of each individual in the population is evaluated according to the refined each convolutional neural network inference data flow directed acyclic graph and its corresponding hardware allocation scheme, and the hardware parameter lookup table of the in-memory computing accelerator. According to an embodiment of the present application, the performance evaluation process of each individual in the population is as follows: topological sorting is performed on the refined each convolutional neural network inference data flow directed acyclic graph to determine the number of calculation cycles of the convolutional neural network inference; the inference delay of each operation node in each convolutional neural network inference data flow directed acyclic graph is determined based on the hardware allocation scheme corresponding to the refined each convolutional neural network inference data flow directed acyclic graph, wherein the maximum inference delay of the operation node is taken as the delay of a single calculation cycle of the convolutional neural network inference; the energy consumption of the peripheral functional components in the in-memory computing accelerator is determined based on the hardware parameter lookup table of the in-memory computing accelerator, the power consumption and delay parameters of the peripheral functional components in the refined each convolutional neural network inference data flow directed acyclic graph and its corresponding hardware allocation scheme, wherein the power consumption and delay parameters of the peripheral functional components at each precision include the power consumption and delay parameters of the digital-to-analog converter and the analog-to-digital converter at each precision, and the power consumption and delay of the arithmetic logic unit; the energy consumption of the non-volatile memory in the in-memory computing accelerator is determined based on the hardware parameter lookup table and the power consumption and delay parameters of the non-volatile memory at each combination parameter in each convolutional neural network weight replication candidate strategy; the inference delay of the convolutional neural network is determined based on the number of calculation cycles of the convolutional neural network inference and the delay of a single calculation cycle; and the energy efficiency of the in-memory computing accelerator architecture is determined based on the energy consumption of the peripheral functional components and the non-volatile memory in the in-memory computing accelerator, the inference delay of the convolutional neural network, the capacity and bandwidth of the cache in the hardware parameter lookup table of the in-memory computing accelerator, and the power consumption and bandwidth of the network on chip to achieve the performance evaluation of each individual in the population. Wherein the inference delay of the convolutional neural network = the number of calculation cycles of the convolutional neural network inference x the delay of a single calculation cycle of the convolutional neural network inference, and the energy efficiency of the in-memory computing accelerator architecture = the energy consumption of the peripheral functional components and the non-volatile memory in the in-memory computing accelerator / the inference delay of the convolutional neural network.

[0082] In the step S63, the second preset number of individuals with the optimal performance are selected as the parents based on the performance evaluation values of each individual in the population according to the preset macro unit mapping scheme.

[0083] In the step S64, new individuals are generated as offspring after mutation of the parents, and the performance of the offspring is evaluated. Illustratively, each parent individual is mutated and a corresponding offspring is generated, one of which changes the number of macro units occupied by each network layer in each convolutional neural network and generates a new offspring; the other changes the strategy of sharing macro units among network layers in the convolutional neural network and generates a corresponding offspring. Each offspring needs to go through the execution of the hardware allocation scheme and the performance evaluation of the accelerator.

[0084] In the step S65, the parent and the child are combined to form a new macro cell mapping scheme population, and the steps S62-S64 are repeatedly executed according to the preset iteration number of the genetic method to search for a macro cell mapping scheme with optimal performance.

[0085] From the above embodiment, it can be seen that the automatic synthesis method of the in-memory computing-oriented convolutional neural network accelerator architecture of the application can greatly reduce the design cost and design difficulty of the in-memory computing accelerator architecture configuration scheme, and effectively enriches the optimization strategy of the accelerator architecture configuration scheme to guarantee the design quality of the in-memory accelerator configuration scheme.

[0086] Next, the in-memory computing accelerator architecture configuration scheme obtained by the application is introduced from the system side, according to an embodiment of the application, as shown in Figure 5As shown, the application proposes an in-memory computing-oriented convolutional neural network accelerator architecture search system, which comprises: a design space search module for storing a convolutional neural network inference task, an in-memory computing accelerator peak power consumption constraint and a hardware parameter lookup table of an in-memory computing accelerator, wherein the hardware parameter lookup table of the in-memory computing accelerator comprises a plurality of design parameters of a non-volatile memory and a plurality of precisions of a digital-to-analog converter, and a plurality of optional parameter combinations are constructed based on the plurality of design parameters of the non-volatile memory, and the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator is determined based on each parameter combination and the in-memory computing accelerator peak power consumption constraint; a weight replication module for screening a first preset number of convolutional neural network weight replication candidate strategies corresponding to each parameter combination based on the array number of the non-volatile memory corresponding to each parameter combination in the in-memory computing accelerator of the design space search module and the convolutional neural network inference task by using an analog annealing method; a data flow compilation module for compiling a convolutional neural network inference data flow directed acyclic graph corresponding to each convolutional neural network weight replication candidate strategy based on each precision of the digital-to-analog converter and each convolutional neural network weight replication candidate strategy screened by the weight replication module; a macro cell mapping module for determining the number of macro cells occupied by each network layer in the convolutional neural network and the number of macro cells shared between network layers based on each convolutional neural network inference data flow directed acyclic graph obtained by the data flow compilation module to obtain a macro cell mapping scheme corresponding to each convolutional neural network inference data flow directed acyclic graph; and a hardware allocation module for performing performance evaluation on each macro cell mapping scheme by using a genetic method, and selecting a macro cell mapping scheme with optimal performance based on the performance evaluation result to obtain a convolutional neural network weight replication candidate strategy, an in-memory computing accelerator hardware parameter combination and a hardware resource allocation scheme corresponding to the scheme, which are used to determine an in-memory accelerator architecture configuration to realize convolutional neural network acceleration inference, wherein the in-memory computing accelerator hardware parameter combination comprises a plurality of design parameters of a non-volatile memory and a precision of a digital-to-analog converter.

[0087] Further, according to an embodiment of the present application, the hardware allocation module is configured to perform the process of performance evaluation on each macro unit mapping scheme and select the optimal macro unit mapping scheme by using a genetic method, which includes: (1) regarding each macro unit mapping scheme as an individual in a preset macro unit mapping scheme population; (2) randomly initializing the preset macro unit mapping scheme population and perfecting each convolutional neural network inference data flow directed acyclic graph based on each individual in the preset macro unit mapping scheme population; (3) designing a corresponding hardware resource allocation scheme for each perfected convolutional neural network inference data flow directed acyclic graph by using a heuristic method; (4) performing performance evaluation on each individual in the population according to the perfected convolutional neural network inference data flow directed acyclic graph and the corresponding hardware allocation scheme, and a hardware parameter lookup table of the in-memory computing accelerator, wherein the process of performance evaluation on each individual in the population includes: performing topological sorting on the perfected convolutional neural network inference data flow directed acyclic graph to determine the number of calculation periods of the convolutional neural network inference; determining the inference delay of each operation node in the convolutional neural network inference data flow directed acyclic graph based on the corresponding hardware allocation scheme of the perfected convolutional neural network inference data flow directed acyclic graph, wherein the maximum inference delay of the operation node is taken as the delay of a single calculation period of the convolutional neural network inference; determining the energy consumption of the peripheral functional components in the in-memory computing accelerator based on the power consumption and delay parameters of the peripheral functional components in the hardware parameter lookup table of the in-memory computing accelerator and the perfected convolutional neural network inference data flow directed acyclic graph and the corresponding hardware allocation scheme; determining the energy consumption of the non-volatile memory in the in-memory computing accelerator based on the power consumption and delay parameters of the non-volatile memory at each combination parameter in the hardware parameter lookup table and each convolutional neural network weight replication candidate strategy; determining the inference delay of the convolutional neural network based on the number of calculation periods of the convolutional neural network inference and the delay of a single calculation period; and determining the energy efficiency of the in-memory computing accelerator architecture based on the energy consumption of the peripheral functional components and the non-volatile memory in the in-memory computing accelerator, the inference delay of the convolutional neural network, the capacity and bandwidth of the cache in the hardware parameter lookup table of the in-memory computing accelerator, and the power consumption and bandwidth of the network on chip to achieve performance evaluation on each individual in the population.

[0088] From the above embodiments, it can be seen that the in-memory computing-oriented convolutional neural network accelerator architecture search system based on the convolutional neural network, the in-memory computing accelerator peak power consumption constraint and each hardware parameter in the in-memory computing accelerator can automatically generate the in-memory computing accelerator architecture configuration scheme and the convolutional neural network inference computing data flow graph, therefore, the system of the embodiments of the present application can greatly reduce the design cost and difficulty of the in-memory computing accelerator architecture configuration scheme, and effectively enrich the optimization strategy of the accelerator architecture configuration scheme to guarantee the design quality of the in-memory accelerator configuration scheme.

[0089] In order to more intuitively understand the in-memory computing accelerator convolutional neural network inference scheme search scheme proposed in the above embodiments of the present application, the implementation process of the in-memory computing accelerator architecture configuration scheme obtained by the above embodiments of the present application will be described in detail below in combination with the pseudo code.

[0090] As shown in Figure 6 and Figure 7 , wherein, Figure 6 the pseudo code description flow of obtaining the in-memory computing accelerator architecture configuration scheme with the optimal performance is shown, Figure 7 the pseudo code description flow of obtaining the optimal macro cell mapping scheme by using the genetic method is shown. In Figure 6 , the pseudo code description of obtaining the optimal macro cell mapping scheme in Figure 7 is included in the pseudo code description of obtaining the in-memory computing accelerator architecture configuration scheme with the optimal performance in Figure 6 and Figure 7 , the functions of each line of the pseudo code will be described in detail.

[0091] As shown in Figure 6As shown, firstly, the pseudo code line 1-2 is to initialize the optimal in-memory computing accelerator architecture configuration and its performance; then, the pseudo code line 3-5 starts to traverse the multiple design parameters of the non-volatile memory in the in-memory computing accelerator, for example, the multiple design parameters of the ReRAM include the ReRAM power consumption ratio (i.e. RatioRram), the ReRAM cell precision (i.e. ResRram) and the ReRAM array size (i.e. XbSize); the pseudo code line 6 is to screen 30 weight duplication candidate strategies for each parameter combination of the in-memory computing accelerator under any (RatioRram, ResRram, XbSize) design parameter combination by using the simulated annealing method; the pseudo code line 7 is to traverse all the weight duplication candidate strategies; the pseudo code line 8 is to traverse the DAC precision; the pseudo code line 10 is to determine the data flow graph after determining the weight duplication strategy WtDup and the DAC precision ResDAC of the current iteration, and determine the optimal macro cell mapping scheme and the hardware resource allocation scheme by using the genetic method in line 10. At this time, the optimal in-memory computing accelerator architecture configuration obtained is the optimal in-memory computing accelerator architecture configuration under the current (RatioRram, ResRram, XbSize, WtDup, ResDAC) loop iteration, and its performance will be updated to Best Perf; finally, by traversing all (RatioRram, ResRram, XbSize, WtDup, ResDAC) combinations, the method executed by the pseudo code can finally determine the optimal in-memory computing accelerator architecture configuration scheme, wherein the genetic method in Figure 7 is executed to perform performance evaluation on all the macro cell mapping schemes obtained after executing the pseudo code line 3-line 9 in Figure 6 , and the macro cell mapping scheme with the optimal performance is selected based on the performance evaluation result to obtain the convolutional neural network weight duplication candidate strategy, the in-memory computing accelerator hardware parameter combination and the hardware resource allocation scheme corresponding to the scheme to obtain the optimal in-memory computing accelerator architecture configuration scheme.

[0092] Further, as shown in Figure 7As shown, first, pseudo code lines 1-2 are to randomly initialize a MacAlloc (macro unit mapping scheme) population and return a performance evaluation value of an in-memory computing accelerator architecture configurator according to each initial MacAlloc solution, wherein the MacAlloc population is a set of all macro unit mapping schemes; then, pseudo code line 4 selects a parent according to the performance evaluation value of each MacAlloc; pseudo code line 5 is to randomly mutate the number of macro units in each network layer of the convolutional neural network and generate a new child; pseudo code line 6 is to randomly mutate the sharing of macro units between network layers in the convolutional neural network and generate a new child; pseudo code line 7 is to insert the child into the original population and perform performance evaluation; pseudo code lines 4-8 represent the steps in one iteration of the genetic method, and the entire method needs to repeat lines 4-8 until the maximum number of iterations of the genetic method is reached, as shown in line 3. Finally, the method executed by the pseudo code returns the optimal macro unit mapping scheme searched by the genetic method.

[0093] In summary, compared with the existing manual design of in-memory computing accelerator architecture configuration for reasoning convolutional neural networks according to the experience of experts, the scheme for generating the in-memory computing accelerator architecture configuration by using the above-mentioned embodiments of the present application greatly reduces the professional threshold of architecture design, improves the efficiency of in-memory computing accelerator design, saves the cost of expert design, and effectively enriches the optimization strategy of accelerator architecture configuration scheme to ensure the design quality of the in-memory accelerator configuration scheme.

[0094] In actual applications, according to an embodiment of the present application, a convolutional neural network reasoning method is provided, which comprises: obtaining the performance-optimal macro unit mapping scheme, the convolutional neural network weight duplication candidate strategy corresponding to the scheme, the in-memory computing accelerator hardware parameter combination, and the hardware resource allocation scheme obtained by the method or system in the above-mentioned embodiments; and determining the in-memory computing accelerator architecture configuration based on the performance-optimal macro unit mapping scheme, the convolutional neural network weight duplication candidate strategy corresponding to the scheme, the in-memory computing accelerator hardware parameter combination, and the hardware resource allocation scheme to implement the reasoning of the convolutional neural network.

[0095] As can be seen from the above-mentioned embodiments, the scheme for generating the in-memory computing accelerator architecture configuration by using the automatic synthesis method of the convolutional neural network accelerator architecture for in-memory computing proposed in the embodiments of the present application improves the execution efficiency of the convolutional neural network.

[0096] It should be noted that although the above describes the steps in a specific order, it does not mean that the steps must be performed in the above-mentioned specific order, in fact, some of the steps can be executed concurrently, or even in a changed order, as long as the required functions can be achieved.

[0097] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0098] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se.

[0099] Having described above several embodiments of the present application, any modifications and variations that fall within the scope of the described embodiments are also to be considered as falling within the scope of the present application. The above description is that of current embodiments of the application, and is not to be construed as limiting the scope of the application, for the application will encompass any modifications or variations of the application that are apparent to those skilled in the art upon attaining a full understanding of the application by reading the foregoing description. The use of the terms "including", "containing", "comprising", "having" or "with" are used herein to mean inclusive (i.e., open-ended) and do not exclude the presence of additional elements or additional steps. Where the disclosure expresses a combination of features it is understood that each individual feature can be replaced by an alternative. Where an indefinite or dependent claim is recited with features in the form "a," "an," or "the," those features should be interpreted under 35 U.S.C. § 112(f) instead of as limiting based on 35 U.S.C. § 101.

Claims

1. A method for automatically synthesizing an in-memory computation-based convolutional neural network accelerator architecture, wherein the in-memory computation accelerator includes a plurality of macrocells, the method comprising: S1. Obtaining a convolutional neural network inference task, a peak power consumption constraint of an in-memory computing accelerator, and a hardware parameter lookup table of the in-memory computing accelerator, wherein the hardware parameter lookup table of the in-memory computing accelerator includes: multiple design parameters of a non-volatile memory and multiple precisions of a digital-to-analog converter; S2. constructing multiple optional parameter combinations based on multiple design parameters of the non-volatile memory, and determining the number of non-volatile memory arrays corresponding to each parameter combination in the in-memory computing accelerator based on each parameter combination and a peak power consumption constraint of the in-memory computing accelerator; S3. Based on the number of non-volatile memory arrays corresponding to each parameter combination in the in-memory computing accelerator and the convolutional neural network inference task determined in step S2, a first preset method is used to screen out a first preset number of convolutional neural network weight replication candidate strategies corresponding to each parameter combination; S4, based on each precision of the digital-to-analog converter and each candidate strategy for copying the weight of the convolutional neural network, compile and generate a directed acyclic graph of the convolutional neural network inference data flow corresponding to each candidate strategy for copying the weight of the convolutional neural network; S5. Determine the number of macro units occupied by each network layer in the convolutional neural network and the number of macro unit sharing between each network layer based on each convolutional neural network inference data flow directed acyclic graph to obtain a macro unit mapping scheme corresponding to each convolutional neural network inference data flow directed acyclic graph; S6. Use the second preset method to perform performance evaluation on each macro unit mapping scheme, and select the macro unit mapping scheme with the best performance based on the performance evaluation results to obtain the convolutional neural network weight replication candidate strategy, in-memory computing accelerator hardware parameter combination and hardware resource allocation scheme corresponding to the scheme, which is used to determine the in-memory accelerator architecture configuration to achieve convolutional neural network accelerated reasoning, wherein the in-memory computing accelerator hardware parameter combination includes: multiple design parameters of the non-volatile memory and the accuracy of the digital-to-analog converter.

2. The method according to claim 1, characterized in that The multiple design parameters of the non-volatile memory include: a ratio of the power consumption of the non-volatile memory to the peak power consumption constraint of the in-memory computing accelerator, the precision of the non-volatile memory cells, and the size of the non-volatile memory array. In step S2, the number of non-volatile memory arrays corresponding to each parameter combination in the in-memory computing accelerator is determined in the following manner: Where #crossbars represents the number of non-volatile memory arrays corresponding to each parameter combination in the in-memory computing accelerator, TotalPower represents the peak power consumption constraint of the in-memory computing accelerator, RatioRram represents the ratio of the power consumed by the non-volatile memory to the peak power consumption constraint of the in-memory computing accelerator, CrossbarPower(·) represents the power consumption function of the non-volatile memory array, XbSize represents the size of the non-volatile memory array, and ResRram represents the precision of the non-volatile memory unit.

3. The method according to claim 2, characterized in that The convolutional neural network inference task includes: the weights of each network layer in the convolutional neural network, the number of input channels, the number of output channels, and the weight kernel size, and in step S3, the first preset method is a simulated annealing method, and the simulated annealing method is used to screen out a first preset number of convolutional neural network weight replication candidate strategies corresponding to each parameter combination in the following manner: Among them, WtDup represents the corresponding convolutional neural network weight replication candidate strategy under each parameter combination, Performance(·) is the energy function of simulated annealing, and WtDup i represents the candidate strategy for weight replication of the i-th layer in the convolutional neural network under each parameter combination, set i represents the number of non-volatile memory arrays required to copy the weights of the i-th layer in the convolutional neural network under each parameter combination, and Represents the weight kernel size of the i-th layer in the convolutional neural network, Represents the number of input channels of the i-th layer in the convolutional neural network, represents the number of output channels of the i-th layer in the convolutional neural network, PrecWt represents the weight accuracy, stdev(·) is the standard deviation function, It represents the standard deviation of the computation time of all network layers in the convolutional neural network under each parameter combination, α is the preset empirical parameter, Represents the standard deviation of data access amount of all network layers in convolutional neural network under each parameter combination.

4. The method according to claim 2, characterized in that In step S4, the convolutional neural network reasoning data flow directed acyclic graph corresponding to each convolutional neural network weight replication candidate strategy is compiled and generated in the following manner: Based on each precision of the digital-to-analog converter, each candidate strategy for weight replication of the convolutional neural network, and the preset computational rules of the convolutional neural network, the reasoning of each network layer in the convolutional neural network is compiled into a set of operation nodes. Determine the dependency between the operation nodes of all network layer reasoning in the convolutional neural network based on the preset dependency rules between the operation nodes; Based on the dependency relationship between the operation nodes of all network layer reasoning in the convolutional neural network, a directed acyclic graph of the convolutional neural network reasoning data flow corresponding to each convolutional neural network weight replication candidate strategy is generated.

5. The method according to claim 4, characterized in that In step S6, the second preset method is a genetic method, and the genetic method is used to evaluate the performance of each macrocell mapping scheme in the following manner to obtain the macrocell mapping scheme with the best performance: S61, taking each macro-cell mapping scheme as an individual in a population of preset macro-cell mapping schemes; S62, randomly initializing a population of preset macro unit mapping schemes and performing a performance evaluation on each individual in the population, wherein a third preset method is used to obtain a hardware resource allocation scheme corresponding to each convolutional neural network inference data flow directed acyclic graph when performing the performance evaluation on each individual; S63, selecting a second preset number of individuals with the best performance as parents based on the performance evaluation value of each individual in the preset macro unit mapping scheme population; S64, after mutating the parent generation, generate new individuals as offspring and evaluate their performance; S65 , combining the parent generation and the child generation to form a new macro unit mapping scheme population, and repeating steps S62 - S64 according to the preset number of iterations of the genetic method to search for the macro unit mapping scheme with the best performance.

6. The method according to claim 5, characterized in that The step S62 includes: S62.

1. Each individual in the population improves the directed acyclic graph of each convolutional neural network inference data flow based on a preset macro unit mapping scheme; S62.

2. Design a corresponding hardware resource allocation solution for each improved convolutional neural network inference data flow directed acyclic graph using the preset third method; S62.

3. Evaluate the performance of each individual in the population based on the improved directed acyclic graph of each convolutional neural network inference data flow and its corresponding hardware allocation plan and the hardware parameter lookup table of the in-memory computing accelerator.

7. The method according to claim 6, characterized in that The hardware parameter lookup table of the in-memory computing accelerator further includes: the operating frequency and power of each peripheral functional component, wherein the peripheral functional components in the in-memory computing accelerator include at least: an arithmetic logic unit, an analog-to-digital converter, and a digital-to-analog converter, and in step S62.2, the preset third method is a heuristic method, and the heuristic method is used to design a corresponding hardware resource allocation scheme for each improved convolutional neural network inference data flow directed acyclic graph in the following manner: 1≤i≤L,c∈peripheral functional components in, Freq represents the computational workload that the peripheral functional component c needs to perform when reasoning about a single operation node of the i-th layer in a convolutional neural network. c Indicates the operating frequency of the peripheral functional component c, P c represents the power of the peripheral functional component c, represents the number of peripheral functional components c assigned to the i-th network layer in the convolutional neural network.

8. The method according to claim 6, characterized in that In step S62.3, the performance of each individual in the population is evaluated in the following manner: Topologically sort the refined directed acyclic graph of each convolutional neural network inference data flow to determine the number of computational cycles of convolutional neural network inference; Based on the improved hardware allocation scheme corresponding to each convolutional neural network inference data flow directed acyclic graph, the inference delay of each operation node in each convolutional neural network inference data flow directed acyclic graph is determined, where the inference delay of the largest operation node is used as the delay of a single computing cycle of the convolutional neural network inference; Determine the energy consumption of peripheral functional components in the in-memory computing accelerator based on the improved directed acyclic graph of each convolutional neural network inference data flow and its corresponding hardware allocation scheme, and the power consumption and delay parameters of the peripheral functional components in the hardware parameter lookup table of the in-memory computing accelerator; Determine the energy consumption of the non-volatile memory in the in-memory computing accelerator based on each convolutional neural network weight replication candidate strategy and the power consumption and delay parameters of the non-volatile memory under each combination parameter in the hardware parameter lookup table; Determine the inference latency of the convolutional neural network based on the number of convolutional neural network inference calculation cycles and the latency of a single calculation cycle; The energy efficiency of the in-memory computing accelerator architecture is determined based on the energy consumption of peripheral functional components and non-volatile memory in the in-memory computing accelerator, the inference delay of the convolutional neural network, the capacity and bandwidth of the cache in the in-memory computing accelerator hardware parameter lookup table, and the power consumption and bandwidth of the on-chip network to achieve performance evaluation of each individual in the population.

9. A convolutional neural network reasoning method, characterized in that: The method comprises: Obtaining the optimal macro unit mapping scheme obtained by the method according to any one of claims 1 to 8 and the corresponding convolutional neural network weight replication candidate strategy and in-memory computing accelerator hardware parameter combination; Based on the optimal macro unit mapping scheme and the corresponding convolutional neural network weight replication candidate strategy, the in-memory computing accelerator hardware parameter combination and the hardware resource allocation scheme, the in-memory computing accelerator architecture configuration is determined to realize the reasoning of the convolutional neural network.

10. A convolutional neural network accelerator architecture search system for in-memory computing, characterized in that: The system comprises: a design space search module, configured to store a convolutional neural network inference task, an in-memory computing accelerator peak power consumption constraint, and a hardware parameter lookup table for the in-memory computing accelerator, wherein the hardware parameter lookup table for the in-memory computing accelerator includes: a plurality of design parameters of the non-volatile memory and a plurality of precisions of the digital-to-analog converter, and constructing a plurality of optional parameter combinations based on the plurality of design parameters of the non-volatile memory, and determining the number of non-volatile memory arrays corresponding to each parameter combination in the in-memory computing accelerator based on each parameter combination and the in-memory computing accelerator peak power consumption constraint; A weight replication module is configured to use a simulated annealing method to screen out a first preset number of convolutional neural network weight replication candidate strategies corresponding to each parameter combination based on the number of non-volatile memory arrays corresponding to each parameter combination in the in-memory computing accelerator of the design space search module and the convolutional neural network inference task; A data flow compilation module is used to compile and generate a directed acyclic graph of the convolutional neural network inference data flow corresponding to each convolutional neural network weight replication candidate strategy based on each precision of the digital-to-analog converter and each convolutional neural network weight replication candidate strategy screened by the weight replication module; A macro unit mapping module is used to determine the number of macro units occupied by each network layer in the convolutional neural network and the number of macro units shared between each network layer based on each convolutional neural network inference data flow directed acyclic graph obtained by the data flow compilation module to obtain a macro unit mapping scheme corresponding to each convolutional neural network inference data flow directed acyclic graph; A hardware allocation module is used to perform performance evaluation on each macro unit mapping scheme using a second preset method, and select the macro unit mapping scheme with the best performance based on the performance evaluation results to obtain the convolutional neural network weight replication candidate strategy, in-memory computing accelerator hardware parameter combination and hardware resource allocation scheme corresponding to the scheme, which is used to determine the in-memory accelerator architecture configuration to achieve accelerated inference of the convolutional neural network, wherein the in-memory computing accelerator hardware parameter combination includes: multiple design parameters of the non-volatile memory and the accuracy of the digital-to-analog converter.

11. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1-8 and 9.

12. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the steps of the method according to any one of claims 1-8 and 9.

Images