Software and hardware joint search method, device and equipment for memory-computing integrated architecture

Abstract

This application relates to the field of hardware-software co-design technology, and particularly to a hardware-software co-search method, apparatus, and device for in-memory computing architectures. The method includes: acquiring multiple sets of design parameters for the in-memory computing architecture and all possible values ​​for each set of design parameters, and combining all possible values ​​to generate a hardware-software co-design space; at the start of the search, constructing a population and searching the co-design space using multiple iterations to obtain candidate design parameters that satisfy the optimization objective; updating the population based on the candidate design parameters; pruning the co-design space using the priority of each candidate design parameter in the population; searching for target design parameters that satisfy the optimization objective in the pruned co-design space; and generating a target network model and a target hardware architecture based on the target design parameters. This enables efficient co-design of the network model and hardware architecture, avoiding tedious manual optimization.

Description

Technical Field

[0001] This application relates to the field of software and hardware co-design technology, and in particular to a software and hardware co-search method, apparatus and device for in-memory computing architecture. Background Technology

[0002] To achieve the joint design of neural network algorithm models and in-memory computing hardware architectures, related technologies have proposed neural network architecture search (NAS) frameworks for in-memory computing architectures, with typical methods including NAS4RRAM and NACIM.

[0003] Among them, (1) under the constraint of a limited number of hardware resources, NAS4RRAM uses an evolutionary algorithm to search for the neural network algorithm model and in-memory computing hardware architecture that can meet the deployment resource constraints and has the best accuracy. At the same time, in order to simulate the unavoidable errors in the computation process of the in-memory computing hardware architecture, NAS4RRAM introduces random errors into the weights for simulation. (2) NACIM focuses on design parameters such as network model, quantization method, circuit structure, and device parameters, and uses accuracy, delay, energy consumption, and area as optimization objectives, and uses reinforcement learning (RL) methods for joint design. For the errors of the in-memory computing hardware architecture, NACIM simulates them by mathematically modeling the errors. The subsequent work of NACIM, UAE, can achieve higher accuracy than NACIM through more appropriate mathematical modeling.

[0004] However, for NAS4RRAM, this method only supports single-bit weights, which face significant accuracy loss in practical applications. Furthermore, the joint search for NAS4RRAM only involves searching for the quantization bit width, neglecting other hardware architecture design parameters such as array size and array arrangement. The NACIM and UAE series of works employ reinforcement learning for joint design space search, resulting in long search times. To achieve high accuracy on hardware, multiple sets of models are needed to simulate the errors of the in-memory computing architecture, further exacerbating the search time. Results from UAE show that searching only a 5-layer neural network algorithm model on a GPU takes 255 hours. Summary of the Invention

[0005] This application provides a hardware and software joint search method, device, electronic device and storage medium for in-memory computing architecture, which can realize the joint and efficient design of network model and hardware architecture, avoiding tedious manual optimization.

[0006] The first aspect of this application provides a software and hardware joint search method for in-memory computing architecture, comprising the following steps: obtaining multiple sets of design parameters for the network model and hardware architecture of the in-memory computing architecture; obtaining all possible values ​​of each set of design parameters and combining all possible values ​​of each set of design parameters to generate a software and hardware joint design space; at the start of the search, constructing a population and searching the joint design space in a multi-round iterative manner to obtain candidate design parameters that satisfy the optimization objective; updating the population based on the candidate design parameters; pruning the joint design space using the priority of each candidate design parameter in the population; and searching for target design parameters that satisfy the optimization objective in the pruned joint design space, and generating a target network model and a target hardware architecture based on the target design parameters.

[0007] Further, the step of searching the joint design space using a multi-round iterative approach to obtain candidate design parameters that satisfy the optimization objective, updating the population based on the candidate design parameters, and pruning the joint design space using the priority of each candidate design parameter in the population includes: deconstructing the hardware-software joint design space into a Cartesian product of all design parameter values, and constructing an original search space based on the Cartesian product of all design parameter values; sampling multiple candidate points from the original search space, and selecting multiple parent candidate points that satisfy the target conditions from the multiple candidate points, wherein each candidate point corresponds to a set of design parameters; performing differential accuracy prediction on the parent candidate points to obtain the candidate design parameters; updating the priority of the candidate design parameters in the population according to the value distribution of the candidate design parameters, and selecting candidate design parameters that satisfy the target priority from the updated population and mutating them into child candidate points, and pruning the target search space from the original search space based on the multiple child candidate points.

[0008] Further, the step of performing differential accuracy prediction on the parent candidate points to achieve performance evaluation of the parent candidate points and obtain the candidate design parameters includes: performing word encoding on the design parameters corresponding to all parent candidate points, mapping the values ​​of the design parameters to normalized vectors; performing different levels of feature transformation on features related to the values ​​of the design parameters, and using a recurrent neural network for feature extraction, encoding them according to the sequential order of blocks in the joint design space to correspond to the time dimension in the recurrent neural network; establishing a mapping relationship between features and performance indicators to be evaluated, normalizing the performance indicators to be evaluated according to their range of occurrence based on the mapping relationship, and using a multilayer perceptron for mathematical modeling and fitting to obtain the candidate design parameters.

[0009] Further, updating the priority of the candidate design parameters in the population based on the value distribution of the candidate design parameters includes: calculating the entropy of the candidate design parameters based on the value distribution of the candidate design parameters, wherein the entropy is used to identify the priority; and updating the priority of the candidate design parameters in the population based on the entropy of each candidate design parameter.

[0010] Furthermore, the acquisition of multiple sets of design parameters for the network model and hardware architecture of the in-memory computing architecture includes: acquiring design parameters for the network model based on supernetting technology, wherein the design parameters include network depth hierarchical design parameters based on the number of blocks in each stage of the supernet, quantization bit width hierarchical design parameters obtained by quantizing the weights and activation data of the network model, and quantization bit width hierarchical design parameters obtained by quantizing the weights and activation data of the network model; and acquiring hardware architecture design parameters based on the impact level on the accuracy and performance of the network model.

[0011] Furthermore, after generating the hardware and software co-design space, the method further includes: establishing the correspondence between different candidate points in the generated hardware and software co-design space through supernetting technology, and realizing parameter sharing among all candidate points through weight sharing technology, wherein each candidate point corresponds to a set of design parameters.

[0012] A second aspect of this application provides a hardware-software joint search device for in-memory computing architecture, comprising: an acquisition module for acquiring multiple sets of design parameters for a network model and hardware architecture of the in-memory computing architecture; a generation module for acquiring all possible values ​​of each set of design parameters and combining all possible values ​​of each set of design parameters to generate a hardware-software joint design space; and a search module for constructing a population at the start of the search and searching the joint design space using a multi-round iterative approach to obtain candidate design parameters that satisfy the optimization objective, updating the population based on the candidate design parameters, pruning the joint design space using the priority of each candidate design parameter in the population, searching for target design parameters that satisfy the optimization objective in the pruned joint design space, and generating a target network model and a target hardware architecture based on the target design parameters.

[0013] Further, the search module is used to: deconstruct the hardware-software joint design space into a Cartesian product of all design parameter values, and construct an original search space based on the Cartesian product of all design parameter values; sample multiple candidate points from the original search space, and select multiple parent candidate points that meet the target conditions from the multiple candidate points, wherein each candidate point corresponds to a set of design parameters; perform differential accuracy prediction on the parent candidate points to obtain the candidate design parameters; update the priority of the candidate design parameters in the population according to the value distribution of the candidate design parameters, and select candidate design parameters that meet the target priority from the updated population, and mutate them into child candidate points; and prune the target search space from the original search space based on the multiple child candidate points.

[0014] Furthermore, the search module is further configured to: encode the design parameters corresponding to all parent candidate points using words, mapping the values ​​of the design parameters to normalized vectors; perform feature transformation at different levels on features related to the values ​​of the design parameters, and extract features using a recurrent neural network, encoding them according to the sequential order of blocks in the joint design space to correspond to the time dimension in the recurrent neural network; establish a mapping relationship between features and performance indicators to be evaluated, normalize the performance indicators to be evaluated according to their range of occurrence based on the mapping relationship, and perform mathematical modeling and fitting using a multilayer perceptron to obtain the candidate design parameters.

[0015] Furthermore, the search module is further configured to: calculate the entropy of the candidate design parameters based on the value distribution of the candidate design parameters, wherein the entropy is used to identify the priority; and update the priority of the candidate design parameters in the population based on the entropy of each candidate design parameter.

[0016] Furthermore, the acquisition module is further configured to: acquire the design parameters of the network model based on supernetting technology, wherein the design parameters include network depth hierarchical design parameters based on the number of blocks in each stage of the supernet, quantization bit width hierarchical design parameters obtained by quantizing the weights and activation data of the network model, and quantization bit width hierarchical design parameters obtained by quantizing the weights and activation data of the network model; and acquire the hardware architecture design parameters according to the impact level on the accuracy and performance of the network model.

[0017] Furthermore, it also includes: a processing module, used to establish the correspondence between different candidate points in the generated hardware and software joint design space through supernetting technology after the hardware and software joint design space is generated, and to realize parameter sharing among all candidate points through weight sharing technology, wherein the candidate point corresponds to a set of design parameters.

[0018] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the hardware-software co-search method for in-memory computing architecture as described in the above embodiments.

[0019] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the hardware-software joint search method for in-memory computing architecture as described in the above embodiments.

[0020] Therefore, this application has at least the following beneficial effects:

[0021] In the hardware-software co-design space, even-numbered convolutional kernels and grouped convolutions are introduced, and mixed-precision search is supported. Hardware support for this hardware-software co-design space is also implemented. A design parameter priority evaluation method is introduced, and on this basis, a dynamic pruning method for the co-design space based on design parameter priority is implemented, which reduces the size of the design space and improves search efficiency. At the same time, an efficient performance predictor is adopted to partially replace the performance simulator for performance evaluation, and by introducing accuracy differential prediction, both efficient and accurate performance prediction is achieved.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0024] Figure 1 This is a flowchart of a hardware and software joint search method for in-memory computing architecture provided according to an embodiment of this application;

[0025] Figure 2 This is a flowchart of a hardware-software joint search method for in-memory computing architecture provided according to an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the hardware and software co-design space provided according to an embodiment of this application;

[0027] Figure 4 A flowchart for updating design parameter priority according to embodiments of this application;

[0028] Figure 5 This is a flowchart of the high-performance predictor provided according to embodiments of this application;

[0029] Figure 6 This is a Pareto boundary comparison map of the CIFAR10 dataset provided according to an embodiment of this application;

[0030] Figure 7 This is an example diagram of a hardware and software joint search device for an in-memory computing architecture provided according to an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0032] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0033] In recent years, deep neural networks (DNNs) have made groundbreaking progress in many fields. However, the explosive growth in the number of parameters and computational demands of DNNs has led to high energy consumption and latency, limiting their application in more scenarios. For example, the GPT-3 model in natural language processing has 175 billion parameters, 45TB of training data, and requires at least 355 GPU years to train. To support the application of deep neural networks in more scenarios, researchers have designed a peer-to-peer (PIM) architecture for efficient computation of deep neural networks.

[0034] The in-memory computing architecture, based on novel non-volatile memory (NVM) devices, provides high storage density while enabling computation to be performed within memory. This allows the in-memory computing architecture to provide sufficient storage space while avoiding redundant data transfer, thereby improving computing energy efficiency. For example, a 2Mb memristor in-memory computing processor was implemented by integrating a 1T1R (1 Transistor 1 Resistance) device array and 22nm CMOS process control and readout circuitry, achieving an energy efficiency of 121.38 TOPS / W, which is three orders of magnitude better than the A100 GPU's 48.75 GOPS / W.

[0035] Convolutional neural network (CNN) algorithms are a representative deep neural network algorithm applied in the field of computer vision. Their core computation is convolution, which can be expressed by the following formula:

[0036]

[0037] Where f i (x,y,z) and f o(x, y, z) represent the values ​​at (x, y, z) coordinates on the input and output feature maps, respectively. The dimensions of x, y, and z are generally referred to as the height dimension, width dimension, and channel dimension, respectively. k is the size of the convolutional kernel. z These are the convolution weights corresponding to the z-th output channel. Convolutional neural networks extract features through convolution (an output point is formed by a weighted combination of several adjacent input points) to complete computer vision tasks.

[0038] Taking resistive RAM (RRAM), a typical device in in-memory computing architecture, as an example, multiple RRAM devices are interconnected to form a cross-array, constituting the basic computing unit of the in-memory computing architecture. Each row of the array is called a word line, each column is called a bit line, and the intersection of the array is called a cell. Data storage can be achieved by changing the resistance value of an RRAM cell by applying different voltages. For example, a high resistance state represents 0, and a low resistance state represents 1. The array structure of RRAM allows it to directly perform matrix-vector multiplication (MVM) in the analog domain. The specific calculation process can be represented by the following formula:

[0039]

[0040] Where v in，j It is the voltage applied to the j-th word line, corresponding to the input vector, g. k，j The conductance of the RRAM at the intersection of the k-th bit line and the j-th word line corresponds to the input matrix. The current i corresponding to each bit line can be obtained in the analog domain using Kirchhoff's laws. out，k The calculation form is shown in the formula above. This formula is completely consistent with the formula for matrix-vector multiplication, thus enabling matrix-vector multiplication to be performed in the analog domain using an RRAM array. It should be noted that due to the existence of non-ideal factors in in-memory computing architectures, these factors can cause computational errors. For example, g k，j The high resistance is used to replace 0 and the low resistance is used to replace 1. However, a high resistance does not mean an open circuit. There will be a small current output, which will affect the final calculation result and cause calculation errors.

[0041] Existing work in the field of in-memory computing primarily focuses on hardware architecture design and joint hardware-software optimization for specific algorithm models. These approaches develop computational model optimization and hardware architecture design independently, neglecting the compatibility between neural network algorithm models and in-memory computing hardware architectures. Experiments show that when the neural network algorithm model is fixed, different in-memory computing hardware architectures exhibit deviations of approximately 3% in accuracy and 92% in energy consumption. Conversely, when the in-memory computing hardware architecture is fixed, different neural network algorithm models show differences of approximately 3.5% in accuracy and 30% in energy consumption. Therefore, to achieve optimal accuracy and computational energy efficiency, it is essential to jointly design the neural network algorithm model and the in-memory computing hardware architecture.

[0042] Although existing methods have made some progress in the joint design of network models and hardware architectures, they still have many shortcomings, making it difficult to apply them in real-world scenarios.

[0043] For NAS4RRAM, this method only supports single-bit weights, which suffer from significant accuracy loss in practical applications. Furthermore, the joint search for NAS4RRAM only involves searching for the quantization bit width, neglecting other hardware architecture design parameters such as array size and array arrangement. The NACIM and UAE series of works employ reinforcement learning for joint design space search, resulting in lengthy search times. To achieve higher accuracy on hardware, multiple sets of models are needed to simulate the errors of the in-memory computing architecture, further exacerbating the search time. Results in UAE show that searching only a 5-layer neural network algorithm model on a GPU takes 255 hours.

[0044] Besides the inherent problems of the methods mentioned above, existing methods face two main issues in the joint design of neural network algorithm models and in-memory computing hardware architectures: First, the joint design space is enormous. Because the design parameters of both the network model and the hardware architecture are simultaneously incorporated into the design space, the joint design space is huge; experiments show that the size of the joint design space can reach 10^6. 90 There are two main issues: First, the design scheme is complex. Second, each evaluation is time-consuming. To accurately assess the performance of the network model on the hardware architecture, a memory-based computing simulator is needed to evaluate its performance. However, due to the complexity of the memory-based computing process, the simulator evaluation is time-consuming. For example, with the MNSIM simulator, a single evaluation takes about 10 minutes, while the joint search typically requires 3,000 evaluations, resulting in a total evaluation process taking about 21 days.

[0045] To address this, this application proposes a high-efficiency hardware-software joint search framework for in-memory computing architectures. By introducing the design parameters involved in the network model and hardware architecture into a joint design space, and through an efficient search strategy, a network model and hardware architecture that meet the actual requirements of the task (e.g., accuracy, energy consumption, area requirements) are automatically and efficiently designed. This improves search efficiency by an order of magnitude, increases the accuracy of the algorithm model on hardware by 10 points, and reduces the hardware energy consumption latency product by 80%.

[0046] The following description, with reference to the accompanying drawings, outlines a hardware-software joint search method, apparatus, electronic device, and storage medium for in-memory computing architectures according to embodiments of this application. Specifically, Figure 1 This is a flowchart illustrating a hardware-software joint search method for in-memory computing architecture provided in an embodiment of this application.

[0047] like Figure 1 As shown, this hardware-software joint search method for in-memory computing architecture includes the following steps:

[0048] In step S101, multiple sets of design parameters for the network model and hardware architecture of the in-memory computing architecture are obtained.

[0049] In step S102, all possible values ​​of each set of design parameters are obtained, and all possible values ​​of each set of design parameters are combined to generate a hardware-software co-design space.

[0050] It is understandable that, such as Figure 2 As shown, to support the joint design of network models and hardware architectures, embodiments of this application consider design parameters related to network models and hardware architectures, and combine the possible values ​​of these design parameters to form a very large-scale joint design space (up to 10). 90 (Size), each point in the joint design space corresponds to a specific set of network model and hardware architecture design parameters, and is also called a candidate point.

[0051] In this application embodiment, multiple sets of design parameters for the network model and hardware architecture of the in-memory computing architecture are obtained, including: obtaining design parameters for the network model based on supernetting technology, wherein the design parameters include network depth hierarchical design parameters based on the number of blocks in each stage of the supernet, quantized bit width hierarchical design parameters obtained by quantizing the weights and activation data of the network model, and quantized bit width hierarchical design parameters obtained by quantizing the weights and activation data of the network model; and obtaining hardware architecture design parameters based on the impact level on the accuracy and performance of the network model.

[0052] Specifically, the space for co-design of hardware and software is as follows: Figure 3As shown, the design space consists of the design space of the neural network algorithm model and the design space of the in-memory computing hardware architecture. The two are combined through a Cartesian product to form a complete joint design space.

[0053] The design space of the algorithm model consists of a combination of design parameters at three levels. The first level is the depth of the network. The supernet corresponding to the hardware-software joint design space in this embodiment consists of three stages and a fully connected layer. The feature map size processed by the network in each stage remains constant, and at the end of each stage, a pooling layer is attached to reduce the size of the feature map, thereby reducing the computational complexity of subsequent stages. For example, on the CIFAR10 dataset, this embodiment uses a typical 2x2 pooling layer, which can reduce the size of the feature map by half. In the first stage, the feature map size is 32x32, while in the third stage, the feature map size is reduced to 8x8. The supernet ends with a fully connected layer, which provides the final classification result. For these three stages, each stage consists of several blocks connected sequentially, and the number of blocks is uncertain, with selectable ranges as shown in the table, including 2, 4, 6, 8, and 10. The number of blocks in each stage constitutes the first level of design parameters. When each stage has 10 blocks, a 30-block network is formed, resulting in a complex model and more accurate results. When each stage has 2 blocks, a 6-block network is formed, resulting in a simple model but less accurate results. By appropriately selecting the number of blocks, the model can be simplified while ensuring accuracy, facilitating efficient hardware execution.

[0054] Table 1

[0055] Network Model Search Space

[0056] Number of blocks 2，4，6，8，10 Number of output channels 16，32，48，64，80，96 kernel size 1，2，3 Number of convolutional groups 1，2，4，8，16

[0057] Quantization parameter search space

[0058] Weight / Activation Bit Width 5，7，9

[0059] Hardware architecture search space

[0060] Cross array size 32，64，128，256 Analog-to-digital converter resolution 4，6，8，10 Digital-to-analog converter resolution 1，2 Memory precision 1，2

[0061] The second level is the block structure level. The internal structure of a block includes convolutional layers (Conv) for feature extraction, batch normalization layers (BN) for feature normalization, and activation layers for non-linear transformations of features to improve their expressive power. Considering that the supernet in this embodiment may consist of up to 30 connected blocks, an excessively deep network structure would be difficult to train. To ensure training effectiveness, this embodiment uses a skip-connection to sum the input of each block with the result of the batch normalization layer before passing it through the activation layer. For each convolutional layer in a block, the number of output channels, the size of the convolutional kernel, and the number of convolutional groups constitute the design parameters for the second level. The selectable range of these parameters is shown in Table 1. This application introduces an even number of convolutional kernels. For in-memory computing architectures, all weights need to be deployed on a computing array, and the array size is often a power of 2. If an odd number of convolutional kernels is used, it means that the weights deployed on the array have an odd number in a certain dimension. In this case, deploying odd-numbered convolutional kernels on a power of 2 size will lead to a waste of some array resources. Using an even number of convolutional kernels can improve the utilization rate of in-memory computing hardware architecture resources. This application also introduces the number of convolutional groups as a design parameter. The number of convolutional groups can divide the naive convolution operation into several groups. For example, when the number of convolutional groups is 2, the first half of the input features is generated by convolution to produce the first half of the output. Similarly, the second half of the input features also corresponds to the second half of the output features. Grouped convolution can reduce the computational cost of convolution operations while keeping the number of input and output channels constant. It also reduces the number of inputs required for parallel computing. For example, when the number of convolution groups is 2, the number of input channels required to compute one output channel is halved. This matches the characteristic of the limited number of word lines that can be activated simultaneously in a memory computing array, making it more friendly to memory computing hardware architecture.

[0062] The third layer is the quantization bit width layer for weights and activations. Due to limitations in in-memory computing hardware architectures, it is difficult to store and compute floating-point format weights and activations. Therefore, when deploying an algorithm model to an in-memory computing hardware architecture, it is necessary to quantize the weights and activations of the algorithm model, converting the floating-point format data into a fixed-point format that the hardware can store and compute. Using a bit-split fixed-point format can achieve higher energy efficiency and accuracy. Therefore, this embodiment of the application also uses a bit-split fixed-point format to quantize the weights and activations of the algorithm model. For weights and activations quantized in fixed-point format, the quantization bit width is the most critical design parameter, directly determining the quantization accuracy and affecting the accuracy of the algorithm model. In this embodiment, the quantization bit width for the weights and activations of each convolutional layer is an independent design parameter, and the selectable parameter range is shown in Table 1. This allows the required quantization bit width for weights and activations to be automatically determined during the search process based on the actual "needs" of the convolutional layer.

[0063] The design parameters of in-memory computing hardware architecture mainly include hardware architecture design parameters that have a significant impact on the accuracy and performance of the algorithm model. These include, for example, the crossbar size of the in-memory computing array, which significantly affects hardware area and latency; the resolution of the digital-to-analog converter (DAC) and analog-to-digital converter (ADC), which significantly affects the accuracy and power consumption of the algorithm model; and the memristor precision, which significantly affects the storage precision of weights and the model accuracy. The selectable value ranges for these parameters are shown in Table 1. Different values ​​of these parameters constitute the design space of the in-memory computing hardware architecture.

[0064] In this embodiment of the application, after generating the hardware and software co-design space, the method further includes: establishing the correspondence between different candidate points in the generated hardware and software co-design space through supernetting technology, and realizing parameter sharing among all candidate points through weight sharing technology, wherein each candidate point corresponds to a set of design parameters.

[0065] Understandably, to achieve efficient searching within such a vast design space, this application employs Supernet and parameter sharing techniques. Unlike naive Neural Network Architecture Search (NAS), which maps different "candidate points" to completely independent network models and hardware architecture designs, this technique uses Supernets to encompass all possible candidate points, which can be generated from the Supernet according to defined rules. For example, if the joint design space includes the kernel size, assuming all possible values ​​for the kernel size include 1x1, 3x3, and 5x5, then the Supernet can correspond to a model with a kernel size of 5x5. When a "candidate point" is selected as 3x3, a 3x3 kernel (including structure and parameters) is extracted from the middle of the 5x5 kernel in the Supernet. Similarly, when a candidate point is selected as 1x1, a 1x1 kernel can be extracted from the middle of the 5x5 kernel. Hypernets establish relationships between different candidate points and support parameter sharing among them. In the example above, some parameters of the 5x5 and 3x3 convolutional kernels correspond to the same parameters on the hypernet, which can accelerate convergence (training a 5x5 convolutional kernel based on a 3x3 kernel converges faster than directly training a 5x5 convolutional kernel). Using hypernets, only one complete training iteration is needed to search the design space, compared to the naive NAS method which requires training on thousands of candidate points, effectively improving search efficiency.

[0066] In step S103, at the start of the search, a population is constructed, and the joint design space is searched in a multi-round iterative manner to obtain candidate design parameters that satisfy the optimization objective. The population is updated based on the candidate design parameters, and the joint design space is pruned using the priority of each candidate design parameter in the population. In the pruned joint design space, the target design parameters that satisfy the optimization objective are searched, and the target network model and target hardware architecture are generated based on the target design parameters.

[0067] It is understood that this application embodiment not only uses supernetting and weight sharing to represent the hardware-software joint design space, but also maintains a "population" during the search process. This population stores a set of "candidate points" that have appeared in previous searches and performed well on the target to be optimized. By continuously searching the joint design space and updating the "population," the best-performing "candidate point" on the target to be optimized in the joint design space can be obtained, leading to the final design parameters. Specifically, this application embodiment uses a multi-round iterative approach to search the joint design space and update the "population." This method allows for continuous adjustment of the search direction, gradually approaching the global optimum, and simultaneously training the weights of the supernet to converge to a better weight.

[0068] Specifically, each iteration of this application embodiment includes two phases: a training phase and an update phase. In the training phase, this application embodiment trains and updates the weights of the supernet, ensuring that the trained supernet weights, after being "generated" through various "candidate points," still maintain good performance. For the "candidate points" needed in the training phase, half are obtained from the "population," and the other half are randomly provided. This approach ensures that the supernet maintains stable performance with "candidate points" from the "population," avoiding significant performance loss from "candidate points" that do not meet the "population's" requirements. Furthermore, by introducing random "candidate points," the supernet can adapt to "candidate points" throughout the entire design space as much as possible. This phase mainly involves updating the supernet weights and does not involve components related to search efficiency, such as search direction and "population" updates; these components will be handled in the update phase.

[0069] During the update phase, this embodiment determines the subsequent search direction and updates the "population" to ensure the efficiency of the search process. In this update phase, a multi-round iterative search is used to search the joint design space to obtain candidate design parameters that meet the optimization objective. The population is updated based on the candidate design parameters, and the joint design space is pruned using the priority of each candidate design parameter in the population. This includes: deconstructing the hardware-software joint design space into a Cartesian product of all design parameter values, and constructing an original search space based on the Cartesian product of all design parameter values; sampling multiple candidate points from the original search space, and selecting multiple parent candidate points that meet the target conditions from these candidate points, where each candidate point corresponds to a set of design parameters; performing differential accuracy prediction on the parent candidate points to obtain candidate design parameters; updating the priority of the candidate design parameters in the population according to the value distribution of the candidate design parameters, and selecting candidate design parameters that meet the target priority from the updated population, and mutating them into child candidate points; and pruning the target search space from the original search space based on multiple child candidate points.

[0070] Among them, candidate design parameters can be understood as selecting some (better) "father" "candidate points" by comparing the performance of all "father" and "candidate points" on the indicators to be optimized.

[0071] It is understood that the embodiments of this application can introduce an assessment of the importance of design parameters in the field of joint design of neural network algorithm parameters and in-memory computing hardware architecture, that is... Figure 2The Priority Update component is located in the upper left corner. This application embodiment designs an evolutionary algorithm based on dynamic parameter priority (Evolutionary Search with Adaptive Parameter Priority, ESAPP). This algorithm determines the priority of each design parameter based on its importance, with higher-important parameters having higher priority. Compared to a naive evolutionary algorithm, this significantly improves search efficiency. At the beginning of the update phase, this evolutionary algorithm samples in the hardware-software joint design space to obtain candidate "parent" points. Next, this application embodiment designs an efficient performance predictor to evaluate all candidate "parent" points. By comparing the performance of all candidate "parent" points on the metrics to be optimized, some selected (better) candidate "parent" points are selected. Subsequently, the evolutionary algorithm based on dynamic parameter priority determines which design is more likely to "mutate" according to the priority of each design parameter, and mutates each selected candidate "parent" point according to a given probability to generate "child" candidate points. Subsequently, this embodiment employs a relatively time-consuming PIM simulator, such as MNSIM and NeuroSim, to evaluate these "child" and "candidate points" and provide simulation evaluation results. Finally, the performance evaluation results provided by the simulator and their corresponding hardware and software design parameters are used as new training data to retrain (fine-tune) the high-efficiency performance predictor, updating the predictor's weights to ensure it consistently provides relatively accurate predictions. Simultaneously, the priority of each design parameter is updated based on the selected "parent" and "candidate points'" values ​​in each design parameter branch and their historical distribution, thereby changing the search direction for the next iteration and improving search efficiency.

[0072] The search algorithm for dynamic parameter priority and the efficient performance predictor will be explained in detail below:

[0073] The hardware-software co-design space provided in this application includes both algorithm model design parameters and hardware architecture-related design parameters, resulting in a massive co-design space, potentially reaching 10^6 ... 90 Naive search methods require hundreds of GPU hours to complete a search. How to perform efficient searches in such a massive joint design space is one of the key problems that hardware-software joint design must solve. This application presents an Evolutionary Search App (ESAPP) algorithm that considers dynamic parameter priorities, achieving efficient searching in a massive joint design space, including the following steps:

[0074] First, the hardware-software co-design space is deconstructed. The hardware-software co-design space contains "candidate points" corresponding to different design parameters, and for any given "candidate point," its value on any given design parameter is fixed. Assuming that for each design parameter, a set corresponding to its value range is defined, in this case, any "candidate point" can be considered as selecting a value from the value range set of each design parameter and then "combining" them. Following this approach, the embodiments of this application can deconstruct the hardware-software co-design space into a Cartesian product of all design parameter values ​​from the perspective of design parameter values. For example... Figure 4 As shown in the schematic diagram of the original search space on the left, there are several independent design parameters, each with its own value range. The Cartesian product of these design parameters constitutes the original search space.

[0075] Secondly, the ESAPP algorithm presented in this application sample some "candidate points" from the original search space, and selects some "parent" "candidate points" with better performance through the screening of an efficient performance predictor.

[0076] Then, based on the value distribution of the selected "parent" and "candidate points" on different design parameters, the priorities of different design parameters are updated (the method for updating parameter priorities will be given later). Subsequently, based on the differences in the priorities of different design parameters, it is determined which design parameters will be "mutated" to form "child" and "candidate points" for subsequent stages. Since the "mutation" is based on the priority of design parameters, the ESAPP algorithm given in this application embodiment can intuitively be considered as dynamically "pruning" the original joint design space according to the priority of design parameters, and improving the search efficiency by searching in a smaller, "pruned" design space, thereby achieving efficient searching in a huge hardware and software joint design space.

[0077] It is important to note that since the ESAPP algorithm performs an equivalent "pruning" from the original joint design space in each iteration, it avoids the problem of losing an important parameter that cannot be recovered, which would lead to a deterioration in search results. This "dynamic" "pruning" method ensures that the search is always conducted in a better and smaller design "subspace" during the search process, while guaranteeing both search efficiency and search results.

[0078] Furthermore, parameter priority is a core concept in the ESAPP algorithm. This embodiment uses the convergence degree of design parameters as an evaluation index of their importance. During the entire search process, when the value of a design parameter is mainly concentrated in one value, it is considered that the design parameter has basically converged and does not require further searching; therefore, a lower parameter priority can be assigned to this design parameter. Conversely, for design parameters whose values ​​are not concentrated in one value but are evenly distributed across different values, these design parameters are considered to require priority in the search and are therefore assigned a higher parameter priority. By assigning different parameter priorities to different design parameters during the search phase, the search for "basically converged" design parameters can be avoided, while the search for "evenly distributed across different values" can be focused on. This reduces the number of design parameters that need to be "mutated" during the search phase, thereby achieving equivalent search space pruning and improving search efficiency. Specifically, this embodiment uses the entropy of the design parameter value distribution as the parameter priority, with the following formula:

[0079]

[0080] Where Ω ω The range of values ​​for the design parameter whose entropy is to be determined is given by f, where ω is an element within that range. ω This refers to the frequency of occurrence of the value during the search phase. In this embodiment, since the values ​​of some "candidate points" on a certain design parameter may not be effective—for example, if the number of blocks in the first stage is selected as 4—then the parameter selection (number of convolution output channels, etc.) of the blocks starting from the 5th block will not affect the network's structure and performance (because these blocks will not be used for computation). In this case, the values ​​of such "candidate points" on these design parameters should not affect the frequency f corresponding to that value. ω Therefore, the above formula does not consider the influence of parameter values ​​on the frequency of occurrence in this case. This application's embodiment implements an engineering approach that considers only the influence of "effective" design parameters on the frequency of occurrence, ensuring that the entropy of the design parameters corresponds to the convergence level of each design parameter.

[0081] Regarding the updating of design parameter priorities, this application embodiment designs an algorithm flow to complete it, such as... Figure 4As shown in the figure. At the beginning of the search process, this embodiment constructs an empty "population" and assigns equal priority to each design parameter. In the sampling process corresponding to row 2 in the figure, this embodiment can sort the performance of individuals in the "population" and select the best R (hyperparameter, specified by the user, default value is 100) as "parents" and "candidate points" (if the number of "population" is less than R, it is supplemented by random generation). Then, according to the aforementioned formula, the entropy of each design parameter is calculated based on the values ​​of all "parents" and "candidate points" on the design parameters, and the entropy of each design parameter is stored in the historical entropy record table (HET), so as to obtain the current and historical entropy values ​​and changes of each design parameter. Based on these values ​​and changes, this embodiment gives the current priority of each design parameter, thus completing the update of the design parameter priority, corresponding to rows 3, 4, and 5 in the figure. Since it is necessary to ensure that the sum of the probabilities of each parameter "mutation" is 1 in the subsequent "mutation" process, it is necessary to make appropriate scaling according to the parameter priority, as shown in row 6 in the figure. The process then proceeds as described above, modifying the "parent" and "candidate points" according to the given "mutation" probabilities to generate "child" and "candidate points" and expand the "population," thus completing the update of the "population."

[0082] The high-efficiency performance predictor will be described below. In this embodiment, differential accuracy prediction is performed on the parent candidate points to evaluate their performance and obtain candidate design parameters. This includes: word encoding of the design parameters corresponding to all parent candidate points, mapping the values ​​of the design parameters to normalized vectors; performing feature transformation at different levels on features related to the design parameter values, and using a recurrent neural network for feature extraction, encoding them according to the order of blocks in the joint design space to correspond to the time dimension in the recurrent neural network; establishing a mapping relationship between features and performance indicators to be evaluated, normalizing the performance indicators to be evaluated according to their range of occurrence based on the mapping relationship, and using a multilayer perceptron for mathematical modeling and fitting to obtain candidate design parameters.

[0083] Understandably, performance evaluation is a key component of traditional Neural Architecture Search (NAS), used to assess which "candidate points" perform better on the target metrics, providing guidance for subsequent searches. Existing joint search frameworks using network models and hardware architectures employ in-memory simulators for performance evaluation. However, current performance evaluators require approximately ten minutes to simulate the performance of a single "candidate point," compromising search efficiency. This application proposes a high-efficiency performance predictor to partially replace the simulator's work, reducing the number of evaluations performed by the in-memory simulator, improving search efficiency, and simultaneously training the predictor to provide predictions that are as close as possible to the simulator's results, thus ensuring search effectiveness.

[0084] Specifically, from the structural perspective of the high-performance predictor, the high-performance predictor proposed in this application consists of three key parts: a structural encoder, a feature extractor, and a regressor. The input to the structural encoder is the "candidate points" in the joint design space, which includes the values ​​of the "candidate points" in the network model design parameters and hardware architecture design parameters. However, considering that the value range of the "candidate points" varies greatly under different design parameters, for example, the maximum value of the number of blocks is 10, while the maximum value of the number of output channels of the convolutional layer is 96, a difference of about 10 times, if prediction is directly based on the vastly different design parameter values, it is easy to ignore the design parameters with smaller value ranges, resulting in poor prediction results. Therefore, this application adopts the word embedding method to map the values ​​of all design parameters of the "candidate points" to normalized vectors, ensuring that the value ranges of these vectors are close. For example, a convolutional kernel size of 1x1 is mapped to (0.2, 0.2), and 3x3 is mapped to (0.6, 0.6). The feature extractor is responsible for converting low-level features directly related to design parameter values ​​into high-level features more relevant to performance prediction, facilitating prediction by the subsequent regressor. The joint design space provided in this embodiment consists of several interconnected blocks, with identical design parameters between blocks. Given this repetitive structure, this embodiment employs a recurrent neural network (RNN) for feature extraction, encoding features according to the block order and corresponding to the time dimension in the RNN. The regressor is responsible for establishing the mapping relationship between the features extracted by the feature extractor and the final predicted performance metrics. Since the performance metrics to be predicted, such as area, energy consumption, and latency, are continuous data within a certain range, this embodiment normalizes these performance metrics according to their range of occurrence and models the problem as a numerical regression problem, using a multilayer perceptron (MLP) for mathematical modeling and fitting.

[0085] From the perspective of prediction data, this application's embodiments creatively propose a differential accuracy predictor. The performance predictor provided in this application's embodiments does not directly give the actual accuracy of the network model after deployment on hardware, because the actual accuracy on hardware is affected not only by errors caused by non-ideal hardware factors but also by the weights of the network model itself. In the search process provided in this application's embodiments, the supernet updates its weights during each iteration, causing weight changes. These weight changes also affect the accuracy of the model after deployment on hardware. Predicting accuracy directly without considering weights leads to poor accuracy prediction results, while considering weights results in a massive amount of weight data, making training difficult to converge. This application's embodiments employ differential accuracy prediction, using the original accuracy corresponding to the network model of the "candidate points" obtained during the search process as a basis (corresponding to...). Figure 5 The base accuracy (in the original text) is the difference between the accuracy after deployment on the hardware and the original accuracy (corresponding to...). Figure 5 The difference (PIM arch acc loss) is caused by the non-ideal factors of the hardware architecture itself. It avoids the impact of changing weights during the search process, thus achieving better accuracy prediction results.

[0086] From the perspective of predictive performance evaluation, this application uses Kendall's Tauranking correlation (KD) as an evaluation metric. This is a commonly used metric to assess the similarity of sequence relationships between two different metrics for the same set of samples. As mentioned earlier, the performance predictor evaluates which "candidate points" perform better on the evaluated metric; that is, it focuses more on the ranking position than the error magnitude during the search process. Since most "candidate points" perform similarly on the evaluation metric, the ranking position and error magnitude may not be consistent. For example, three "candidate points" have actual evaluation metrics of (0.9, 0.81, 0.80), and two prediction results are (0.85, 0.81, 0.80) and (0.9, 0.80, 0.81). It can be seen that the latter has a smaller error magnitude, but its ranking position is incorrect, which is detrimental to the search. Therefore, this application uses ranking position as the evaluation metric for the performance predictor. The specific formula is as follows:

[0087]

[0088] Where M represents the number of "candidate points", sgn(.)sgn (·) This is a sign function. In the embodiments of this application, this function is used to ensure that only the sorting position is considered. i and s iThese represent the predictor's and the actual performance metrics, respectively.

[0089] The following will illustrate the hardware-software joint search method for in-memory computing architecture through specific embodiments, as follows:

[0090] Automated search for design parameters of a 30-layer ResNet-like network and hardware architecture was performed on the CIFAR10 dataset. The performance evaluation metrics (accuracy, area, latency, etc.) provided by MNSIM were used as the final evaluation metrics. The experimental results are shown in Table 2 (Gibbon is a pseudonym for the embodiment in this application). As can be seen from the table, Gibbon, optimized for accuracy, achieves the highest accuracy compared to all other methods. Gibbon, optimized for area, achieves the smallest area compared to other methods, still achieving a 60% area reduction compared to the previously smallest NACIM method. Furthermore, the embodiment in this application can complete the joint design of the network model and hardware architecture on a GPU in only about 7 hours, achieving an order of magnitude improvement in search efficiency compared to methods such as NACIM, UAE, and NAS4RRAM. Experimental results demonstrate that the embodiment in this application achieves a high-quality and fast joint design of the network model and hardware architecture.

[0091] Table 2

[0092]

[0093] The embodiments of this application were also compared with the existing typical design method NACIM on the Pareto boundary, and the results are as follows: Figure 6 As shown. From Figure 6 As can be seen from the figure, the embodiments of this application achieve a better Pareto boundary than the typical method NACIM. EASPP is a search method that considers the importance of parameters proposed in the embodiments of this application, and the results in the figure show that better results can be achieved by using EASPP.

[0094] In summary, the embodiments of this application enable efficient joint design of network models and hardware architectures, avoiding tedious manual optimization. Furthermore, the network model and hardware architecture can be adaptively modified according to task requirements, thus adapting to task needs in various scenarios and environments. Compared to other methods, the embodiments of this application effectively reduce the joint design time from 255 hours to approximately 7 hours. Simultaneously, the embodiments of this application also show significant improvements over other methods in terms of accuracy, area, latency, and energy consumption. Specifically:

[0095] (1) This application proposes an evolutionary algorithm based on design parameter priority, which can dynamically "trim" the joint design space, reducing the size of the joint design space from 10... 90 Reduced to 1042 ;

[0096] (2) The embodiments of this application propose a high-efficiency performance predictor based on a recurrent convolutional neural network. The evaluation time is only 1 / 150 of the simulation evaluation time, and the error is only about 2%. The high-efficiency performance predictor can replace about 95% of the simulation evaluation requirements.

[0097] (3) In terms of specific search performance, the embodiments of this application achieve a search efficiency improvement of 8.4 to 41.3 times compared with existing hardware and software joint search algorithms, and can complete the search on the CIFAR-10 dataset in just 7 hours. Furthermore, it achieves a maximum improvement of 10.7 points in deployment accuracy and a reduction of 6.48 times in Energy-Delay-Product (EDP).

[0098] Next, referring to the accompanying drawings, a hardware and software joint search device for an in-memory computing architecture is described according to an embodiment of this application.

[0099] Figure 7 This is a block diagram of a hardware and software joint search device for an in-memory computing architecture according to an embodiment of this application.

[0100] like Figure 7 As shown, the hardware and software joint search device 10 for in-memory computing architecture includes: an acquisition module 100, a generation module 200, and a search module 300.

[0101] The acquisition module 100 is used to acquire multiple sets of design parameters for the network model and hardware architecture of the in-memory computing architecture; the generation module 200 is used to acquire all possible values ​​of each set of design parameters and combine all possible values ​​of each set of design parameters to generate a joint hardware and software design space; the search module 300 is used to construct a population at the beginning of the search and use a multi-round iterative approach to search the joint design space to obtain candidate design parameters that meet the optimization objective, update the population based on the candidate design parameters, and prune the joint design space using the priority of each candidate design parameter in the population. In the pruned joint design space, the target design parameters that meet the optimization objective are searched, and the target network model and target hardware architecture are generated based on the target design parameters.

[0102] In this embodiment, the search module 300 is used to: deconstruct the hardware-software joint design space into a Cartesian product of all design parameter values, and construct an original search space based on the Cartesian product of all design parameter values; sample multiple candidate points from the original search space, and select multiple parent candidate points that meet the target conditions from the multiple candidate points, wherein each candidate point corresponds to a set of design parameters; perform differential accuracy prediction on the parent candidate points to obtain candidate design parameters; update the priority of candidate design parameters in the population according to the value distribution of the candidate design parameters, and select candidate design parameters that meet the target priority from the updated population and mutate them into child candidate points, and prune the target search space from the original search space based on the multiple child candidate points.

[0103] In this embodiment, the search module 300 is further configured to: encode the design parameters corresponding to all parent candidate points using words, mapping the values ​​of the design parameters to normalized vectors; perform feature transformation at different levels on features related to the values ​​of the design parameters, and extract features using a recurrent neural network, encoding them according to the order of blocks in the joint design space to correspond to the time dimension in the recurrent neural network; establish a mapping relationship between features and performance indicators to be evaluated, normalize the performance indicators to be evaluated according to their range of occurrence based on the mapping relationship, and use a multilayer perceptron for mathematical modeling and fitting to obtain candidate design parameters.

[0104] In this embodiment, the search module 300 is further configured to: calculate the entropy of the candidate design parameters according to the value distribution of the candidate design parameters, wherein the entropy is used to identify the priority; and update the priority of the candidate design parameters in the population according to the entropy of each candidate design parameter.

[0105] In this embodiment of the application, the acquisition module 100 is further used to: acquire the design parameters of the network model based on supernetting technology, wherein the design parameters include network depth hierarchical design parameters based on the number of blocks in each stage of the supernet, quantized bit width hierarchical design parameters obtained by quantizing the weights and activation data of the network model, and quantized bit width hierarchical design parameters obtained by quantizing the weights and activation data of the network model; and acquire hardware architecture design parameters according to the impact level on the accuracy and performance of the network model.

[0106] In this embodiment of the application, the apparatus 10 further includes a processing module. The processing module is used to establish a correspondence between different candidate points in the generated hardware-software co-design space using supernetting technology after the hardware-software co-design space is generated, and to achieve parameter sharing among all candidate points using weight sharing technology, wherein each candidate point corresponds to a set of design parameters.

[0107] It should be noted that the foregoing explanation of the embodiment of the hardware and software joint search method for in-memory computing architecture also applies to the hardware and software joint search device for in-memory computing architecture in this embodiment, and will not be repeated here.

[0108] The hardware-software joint search device for in-memory computing architecture proposed in this application introduces even-numbered convolution kernels and grouped convolutions in the hardware-software joint design space, and supports mixed-precision search. Hardware support for this hardware-software joint design space is also implemented. Furthermore, a design parameter priority evaluation method is introduced, and based on this, a dynamic pruning method for the joint design space based on design parameter priority is implemented, reducing the design space size and improving search efficiency. Simultaneously, a high-efficiency performance predictor is used to partially replace the performance simulator for performance evaluation, and by introducing accuracy differential prediction, both efficient and accurate performance prediction is achieved.

[0109] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0110] The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0111] When the processor 802 executes the program, it implements the hardware and software joint search method for in-memory computing architecture provided in the above embodiments.

[0112] Furthermore, electronic devices also include:

[0113] Communication interface 803 is used for communication between memory 801 and processor 802.

[0114] The memory 801 is used to store computer programs that can run on the processor 802.

[0115] The memory 801 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0116] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0117] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0118] The processor 802 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0119] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described hardware and software joint search method for in-memory computing architecture.

[0120] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0121] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0122] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0123] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0124] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

Claims

1. A method for a hardware-software co-searching method for an in-memory computing architecture, the method comprising: The method comprises the following steps: obtaining a plurality of sets of design parameters of a network model and a hardware architecture of a memory-computing integrated architecture; obtaining all possible values of each set of design parameters, and combining all possible values of each set of design parameters to generate a soft and hardware joint design space; and At the beginning of the search, a population is constructed, and the joint design space is searched in a plurality of rounds of iterations to obtain candidate design parameters that meet the to-be-optimized target, the population is updated based on the candidate design parameters, and the joint design space is pruned using the priority of each candidate design parameter in the population, and in the pruned joint design space, target design parameters that meet the to-be-optimized target are searched, and a target network model and a target hardware architecture are generated based on the target design parameters; The joint design space is searched in a plurality of rounds of iterations to obtain candidate design parameters that meet the to-be-optimized target, the population is updated based on the candidate design parameters, and the joint design space is pruned using the priority of each candidate design parameter in the population, which comprises: decomposing the soft and hardware joint design space into a Cartesian product of all design parameter values of the design space of the neural network algorithm model and the design space of the memory-computing integrated hardware architecture, and constructing an original search space based on the Cartesian product of all design parameter values; a plurality of candidate points are sampled from the original search space, and a plurality of parent candidate points that meet the target condition are selected from the plurality of candidate points, wherein the candidate points correspond to a set of design parameters; the parent candidate points are subjected to differential accuracy prediction to obtain the candidate design parameters; the priority of the candidate design parameters in the population is updated according to the value distribution of the candidate design parameters, the candidate design parameters that meet the target priority are selected from the updated population, and are mutated into child candidate points, and the target search space is pruned from the original search space based on the plurality of child candidate points; the performance of the parent candidate points is evaluated by differential accuracy prediction to obtain the candidate design parameters, which comprises: word encoding the design parameters corresponding to all parent candidate points, and mapping the values of the design parameters to normalized vectors; converting features related to design parameter values at different levels, and extracting features using a recurrent neural network, and encoding according to the order of blocks in the joint design space to correspond to the time dimension in the recurrent neural network; a mapping relationship between features and to-be-evaluated performance indicators is established, the to-be-evaluated performance indicators are normalized according to the range of occurrence based on the mapping relationship, and a multilayer perceptron is used for mathematical modeling and fitting to obtain the candidate design parameters.

2. The method of claim 1, wherein, The priority of the candidate design parameters in the population is updated according to the value distribution of the candidate design parameters, which comprises: calculating the entropy of the candidate design parameters according to the value distribution of the candidate design parameters, wherein the entropy is used to identify the priority; updating the priority of the candidate design parameters in the population according to the entropy of each candidate design parameter.

3. The method of claim 1, wherein, The multiple sets of design parameters of the network model and the hardware architecture of the storage-computing integrated architecture include: The design parameters of the network model are obtained based on a super network technology, wherein the design parameters include network depth level design parameters composed of the number of blocks of each stage in the super network, and quantization bit width level design parameters obtained by quantizing weight and activation data of the network model; The hardware architecture design parameters are obtained according to the influence level of the accuracy and performance of the network model.

4. The method according to any one of claims 1 to 3, characterized in that, After the soft and hardware joint design space is generated, the following steps are further included: The corresponding relationship between different candidate points in the generated soft and hardware joint design space is established by the super network technology, and parameter sharing between all candidate points is realized by a weight sharing technology, wherein the candidate points correspond to a set of design parameters.

5. A hardware and software joint search device for a compute-in-memory architecture, the hardware and software joint search device comprising: The following steps are included: A obtaining module is configured to obtain multiple sets of design parameters of a network model and a hardware architecture of a storage-computing integrated architecture; A generating module is configured to obtain all possible values of each set of design parameters, and combine all possible values of each set of design parameters to generate a soft and hardware joint design space; And A searching module is configured to, at the beginning of searching, construct a population, search the joint design space in a multi-round iteration manner to obtain candidate design parameters that meet a to-be-optimized target, update the population based on the candidate design parameters, and use a priority of each candidate design parameter in the population to clip the joint design space, and search target design parameters that meet the to-be-optimized target in the clipped joint design space, and generate a target network model and a target hardware architecture based on the target design parameters. The search of the joint design space in multiple rounds of iterations to obtain candidate design parameters satisfying the optimization target, updating the population based on the candidate design parameters, and pruning the joint design space using the priority of each candidate design parameter in the population, includes: decomposing the software and hardware joint design space into a Cartesian product of all design parameter values of the design space of the neural network algorithm model and the design space of the integrated computing hardware architecture, and constructing an original search space based on the Cartesian product of all design parameter values; sampling a plurality of candidate points from the original search space, and selecting a plurality of parent candidate points satisfying the target condition from the plurality of candidate points, wherein the candidate points correspond to a group of design parameters; performing differential accuracy prediction on the parent candidate points to obtain the candidate design parameters; updating the priority of the candidate design parameters in the population according to the value distribution of the candidate design parameters, selecting candidate design parameters satisfying the target priority from the updated population, and mutating into child candidate points, and pruning the target search space from the original search space based on the plurality of child candidate points; the differential accuracy prediction of the parent candidate points to realize the performance evaluation of the parent candidate points, and obtain the candidate design parameters, including: word encoding the design parameters corresponding to all parent candidate points, and mapping the value of the design parameter to a normalized vector; converting features related to the value of the design parameter at different levels, and extracting features using a recurrent neural network, and encoding according to the order of the blocks in the joint design space to correspond to the time dimension in the recurrent neural network; establishing a mapping relationship between the features and the performance indicators to be evaluated, normalizing the performance indicators to be evaluated according to the range of occurrence based on the mapping relationship, and using a multilayer perceptron for mathematical modeling and fitting to obtain the candidate design parameters.

6. The apparatus of claim 5, wherein, The search module is further used to: Calculate the entropy of the candidate design parameters according to the value distribution of the candidate design parameters, wherein the entropy is used to identify the priority; Update the priority of the candidate design parameters in the population according to the entropy of each candidate design parameter.

7. The apparatus of claim 5, wherein, The acquisition module is further used to: Acquire the design parameters of the network model based on hypernet technology, wherein the design parameters include network depth level design parameters based on the number of blocks at each stage in the hypernet, and quantization bit width level design parameters obtained by quantizing the weights and activation data of the network model; Acquire the hardware architecture design parameters according to the influence level of the accuracy and performance of the network model.

8. The apparatus of any one of claims 5-7, wherein, Further comprising: A processing module for establishing the corresponding relationship between different candidate points in the generated software and hardware joint design space through hypernet technology after generating the software and hardware joint design space, and realizing parameter sharing between all candidate points through weight sharing technology, wherein the candidate points correspond to a group of design parameters.

9. An electronic device, comprising: Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the soft-hard combined search method for the memory-compute integrated architecture according to any one of claims 1-4.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the soft-hard combined search method for the memory-compute integrated architecture according to any one of claims 1-4.

Images