Adaptive accelerator with alternate production / optimization mode

Abstract

The present disclosure relates generally to adaptive accelerators with alternating production / optimization modes. Systems and methods are provided for accelerator systems including a baseline (production) accelerator, an optimization accelerator, and control hardware accelerators, and operations to cause the production / optimization accelerators to alternate between production and optimization. With two production / optimization accelerators, at any given point in time, one accelerator is adapting while the other is processing data. Once the second accelerator starts doing better (e.g., has adapted to data drift), the accelerators change their modes, and the trainable accelerator becomes the "optimized" accelerator. The accelerators process like this uninterruptedly, maintaining redundancy, which provides the expected quality of service (QoS), and adapts to data / concept drift.

Description

Technical Field

[0001] This disclosure generally relates to adaptive accelerators with alternating production / optimization modes. Background Technology

[0002] Many neural networks are overkill in terms of computation, power, and memory. Redundancy has been used to improve the reliability of neural networks by having spare copies of processing elements and switching from a failed component to its spare copy. In some industries, redundancy involving the deployment of multiple accelerators is a necessary feature due to regulatory requirements (such as ensuring a certain level of reliability). This redundancy often leads to extensive optimizations, such as quantization at different levels of precision. There is no systematic optimization approach yet, and this often requires retraining the model. Summary of the Invention

[0003] According to one aspect of this disclosure, a non-transitory machine-readable medium is provided that stores instructions that cause one or more processors of an electronic device to perform operations including: transmitting preprocessed data to a first accelerator; changing at least one operating parameter of the first accelerator; monitoring the impact of each changed operating parameter of the first accelerator on the Quality of Service (QoS); changing the at least one operating parameter of the first accelerator based on the monitored impact of each changed operating parameter of the first accelerator on the QoS; and determining whether the QoS has met a threshold, wherein, once it is determined that the QoS has met the threshold, the following additional operations are performed: routing unprocessed data to the first accelerator; and transmitting additional preprocessed data to a second accelerator; thereby enabling the first accelerator to adapt by switching from an optimized mode to a production mode.

[0004] According to another aspect of this disclosure, a method for optimizing an accelerator is provided, comprising: setting a first accelerator initially operating in a baseline mode and a second accelerator initially operating in an optimized mode; setting a Quality of Service (QoS) target based on parameters received from an adapter communicatively connected to the first accelerator and the second accelerator; measuring the QoS of operating the first accelerator; and determining whether the measured QoS of operating the first accelerator has satisfied the QoS target associated with the first accelerator, wherein if the QoS target associated with the first accelerator is satisfied, the first accelerator is maintained as a baseline accelerator, and wherein if the QoS target associated with the first accelerator is not satisfied, the first accelerator is optimized by switching the first accelerator to an adapted optimized accelerator.

[0005] According to one aspect of this disclosure, an accelerator system is provided, comprising: a first accelerator; a second accelerator; a processor; and a memory configured to store instructions that, in response to execution by the processor, cause the processor to: change at least one operating parameter of the first accelerator; monitor the impact of each changed operating parameter of the first accelerator on the Quality of Service (QoS); change the at least one operating parameter of the first accelerator based on the monitored impact of each changed operating parameter of the first accelerator on the QoS; and determine whether the QoS has met a threshold, wherein, once it is determined that the QoS has met the threshold, the first accelerator is configured to switch from an optimized mode to a production mode, and the second accelerator is configured to switch from the production mode to an optimized mode. Attached Figure Description

[0006] This disclosure is described in detail with reference to the following accompanying drawings, based on one or more different examples of the disclosed technology. The drawings are provided for illustrative purposes only and depict only typical or exemplary aspects of the disclosed technology.

[0007] Figure 1 The illustration shows a memristor crossbar array (i.e., an accelerator) according to one or more examples described herein.

[0008] Figure 2 The illustration shows a memristor dot product engine with a comparator processor, based on one or more examples described herein.

[0009] Figure 3 The illustration shows a virtualization system of a memristor DPE resource with a memristor dot product engine (DPE) according to one or more examples described herein, and a detailed illustration of the tiles in the memristor DPE resource.

[0010] Figure 4 The illustration shows an adaptive memristor cross-switch array (i.e., an adaptive accelerator) according to one or more examples described herein.

[0011] Figure 5 The illustration shows a memristor cross-switch array (i.e., an optimized accelerator) in optimized mode according to one or more examples described herein.

[0012] Figure 6 The illustration depicts an adaptive accelerator system that uses data tags to sort data, based on one or more examples described herein.

[0013] Figure 7This is a flowchart illustrating a method of using an adaptive accelerator system according to one or more examples described herein.

[0014] Figure 8 This is a flowchart illustrating a method for using an adaptive accelerator system, including data labeling capabilities, according to one or more examples described herein.

[0015] Figure 9 It is a computational component that can be used to implement various features of the examples described in this disclosure.

[0016] Figure 10 A block diagram depicts a computer system in which various features of the examples described herein can be implemented.

[0017] The accompanying drawings are not exhaustive and do not limit this disclosure to the precise form disclosed. Detailed Implementation

[0018] It should be understood that the accompanying drawings and descriptions of the disclosed technology may have been simplified to illustrate elements relevant to a clear understanding of the disclosed technology, while other elements present in typical accelerator systems or typical methods of using accelerator systems have been omitted for clarity. Those skilled in the art will recognize that other elements may be desirable and / or necessary for implementing the disclosed technology. However, since such elements are well-known in the art and because they are detrimental to a better understanding of the disclosed technology, a discussion of such elements is not provided herein. It should also be understood that the accompanying drawings included herein provide only a schematic representation of the structure of the disclosed technology, and structures falling within the scope of the disclosed technology may include structures different from those shown in the drawings. Reference will now be made to the accompanying drawings, in which the same structures bear the same reference names.

[0019] Before explaining in detail at least one example of the disclosed technology, it should be understood that the inventive concepts set forth herein are not limited in their application to the construction details or component arrangements set forth in the following description or illustrated in the accompanying drawings. It should also be understood that the wording and terminology used herein are for descriptive purposes only and should not be considered restrictive.

[0020] It should be further understood that any of the described features may be used alone or in combination with other features. Other inventive devices, structures, apparatuses, systems, methods, features, and advantages will be apparent or become apparent to those skilled in the art upon examination of the accompanying drawings and the detailed description herein. All such additional devices, structures, apparatuses, systems, methods, features, and advantages are intended to be protected by the appended claims.

[0021] For the purposes of this disclosure, the phrases “memristor cross-switch array,” “memristor DPE,” or the term “accelerator” (and its corresponding derivatives) are all used interchangeably.

[0022] As used herein, "dot product" broadly refers to the product of two vectors used to form a scalar whose value is the product of the magnitudes of the vectors and the cosine of the angle between them. In the context of a memristor DPE, the dot product can be determined by applying a voltage to the input and multiplying that voltage by the conductivity programmed into the DPE to obtain the current, as referenced below. Figure 3 To describe in more detail.

[0023] It should be noted that the terms "optimized," "optimal," etc., as used herein, can be used to mean achieving or realizing the most efficient or perfect performance possible. However, those skilled in the art who read this document will recognize that perfection is not always achievable. Therefore, these terms can also encompass achieving or realizing the best possible, most efficient, or most practical performance under given conditions, or achieving or realizing performance superior to that achievable using other settings or parameters.

[0024] In traditional chip processing, accelerators are used to perform several complex computations. For example, in convolutional neural networks (CNNs) and other types of artificial neural networks, the network consists of neurons with learnable weights. These weights are initialized with random numbers drawn from a predefined distribution, such as a Gaussian distribution, to allow the neurons to learn to compute useful features during training. Each neuron receives some inputs, performs a dot product between these inputs and a vector of weights, and may adjust these weights as training progresses. In this way, the weights can be adjusted for each specific training task.

[0025] However, with traditional chip processing, these training computations are performed serially and can waste the processing power that would otherwise be available on the chip. This waste is particularly pronounced in deep learning. For example, in convolutional neural networks (CNNs) and some other classes of deep neural networks, processing is performed at each layer. The weights of a layer consist of a set of learnable filters with small receptive fields that extend across the entire depth of the input. During the forward pass, each filter is convolved across the width and height of the input volume, thus computing the dot product between the filter's entries and the input entries, and producing a 2D activation map of that filter. Therefore, the network learns that a filter will activate when it detects a particular type of feature at a certain spatial location in the input.

[0026] The examples of techniques disclosed herein improve convolution computation on cross-switched accelerators (such as DPEs), which are inefficiently used in conventional chip processing, by incorporating both intra-cross-switched and inter-cross-switched computations. In cross-switched computation, convolutional layers can be unfolded into vectors, and these vectors can be grouped into matrices where each row or column is an unfolded filter. Each submatrix of the input corresponding to the position of the convolution window is also unfolded into a vector. Convolution is computed as the dot product of each input vector and the filter matrix. In intra-cross-switched computation, unused space in the cross-switches is used to store copies of the filter matrices, and X... N Unused space in the matrix is ​​used to store more elements of the input. In the computation between cross switches, unused cross switches are used to store copies of the filter matrix, and unused X... N `s` is used to store more elements of the input. Then, the method performs multiple convolution iterations in a single step.

[0027] Examples of the disclosed techniques also reduce the amount of computation required for convolutions on large inputs. For example, during each convolution, the system can extract useful features from the input using different types of filters, the weights of which are learned automatically during training. In some examples of the disclosed techniques, filters can be copied and convolutions can be performed on unused cross switches in the chip. By using the cross-switch filter copying technique, an improved convolution process can be applied even when cross switches are fully utilized but unused cross switches exist in the chip.

[0028] When computing convolutions with small filters (e.g., filters that fill a small percentage of the cross-switch space), intra-cross-switch computation can reduce the resource waste associated with large cross-switches. The filter size may not change, but the system can implement both inter-cross-switch and intra-cross-switch computations to perform multiple convolution iterations in a single step. Additionally, combining intra-cross-switch and inter-cross-switch computations allows the system to use larger cross-switches while ensuring efficiency regardless of the filter's receptive field size. This can help improve at least the initial processing of CNNs, where traditional systems utilize small weights and large input sizes.

[0029] Figure 1 Example electronic devices are provided according to examples of this disclosure. In some examples, electronic devices (e.g., accelerators, etc.) are organized in a hierarchy of cross switches and local processors that coordinate computations and perform operations not handled by the cross switches. The cross switches can perform matrix-vector multiplication (MVM) on fixed-size blocks of matrices. Figure 1The example electronic device illustrated in the figure includes a memristor cross switch array 100, a row driver circuit 120, and a plurality of column output circuits 130, each column output circuit corresponding to one of the column lines CL of the memristor cross switch array 100.

[0030] The memristor crossbar switch array 100 may include N row lines RL (where N ≥ 1), M column lines CL (where M ≥ 1), and multiple memristors 101, each memristor connected between one of the row lines RL and one of the column lines CL (see dashed box 110). In some examples, each intersection of a row line RL and a column line CL may have a corresponding memristor 101. Row lines RL n Each of these can correspond to a digital input vector. = (x1, x2, …, x N ) numeric input value x n Specifically, row line RL1 corresponds to the numeric input value x1, row line RL2 corresponds to the numeric input value x2, and so on.

[0031] The row driver circuit 120 serves as the input stage of the crossbar switch array 100 and can operate across multiple time periods based on the digital input vector. Analog voltage is applied to the row line RL according to a distributed digital-to-analog conversion architecture. Specifically, in some examples, the row driver circuit 120 can apply voltage to the row line RL over multiple time periods t1, t2, ..., t... P During this period, multiple analog voltages V1, V2, ..., V were respectively... P Apply to the nodes, and for each in the row line RL, in multiple time intervals t1, t2, ..., t... P One of the periods based on digital input vectors The corresponding numeric input value x n Come and line RL n Selectively connect to nodes. For example, a numeric input value x n Any line RL equal to "1" n You can connect to the node during the first time period t1, and input the numerical value x. n Any line RL equal to "2" n You can connect to the node during the first time period t2, and so on.

[0032] Each of the column output circuits 130_M (where M ≥ 1) can include an integrating capacitor C. int (Not shown) and a switch S1 (not shown) controlled by the integral control signal Ctr_Int (not shown). In some examples, each of the column output circuits 130_M may include similar features in a similar configuration (except for being connected to different column lines CL). M ).

[0033] The computational workload associated with creating a deep neural network typically includes a training phase and an inference phase. During training, the neural network attempts to learn from a set of input training data. Each layer of the untrained model is assigned some random weights, and the model runs a forward pass through that input training data. This forward pass computes the model output, for example, predicting the class label for a classification model by determining the class score using the model weights. The class score is compared to the actual label, and the error is calculated using a loss function. Then, in the back pass, the error is propagated back through the neural network, and the weights are updated. For example, the weights can be updated via an algorithm such as gradient descent. The gradient descent algorithm is an optimization algorithm that minimizes a cost function to find the values ​​of the parameters (i.e., coefficients) of the function.

[0034] An epoch refers to a complete forward and backward pass of all samples input into the training data. Typically, using a traditional CMOS-based CPU, these computations are very expensive and are performed with a single weight update after processing each sample. In this case, the training data is divided into batches, and the weights are updated after each batch.

[0035] The common computational architecture used in deep learning workloads (matrix dot product) has led to the adoption of GPUs to provide performance speedups compared to traditional CPUs. GPUs are better at handling vector and matrix computations than CPUs. Single or parallel graphics processing units (GPUs) can be used to accelerate processing. However, GPUs built for graphics applications are not fully optimized for the requirements of deep learning workloads and suffer from performance and energy inefficiencies.

[0036] Using a memristor cross-switch array further reduces processing time on the GPU, and offers high computational performance with low power requirements. The memristor cross-switch array can be used for matrix dot product calculations during the training phase. During backpropagation in the training phase, model weights can be changed from the last layer to the first layer of the model. The memory memristor cross-switch array can be iteratively updated with the model weights, and matrix dot product calculations can be performed to determine the optimal model.

[0037] A comparator circuit electrically coupled to the vector output register of a memristor cross-switch array can compare data already available in the memristor cross-switch array with new model weight data written into the cross-switch array. Two example general methods for updating the weights (represented as trainable parameters of a neural network as floating-point numbers) stored in the cross-switch array are provided below. In the first example method, mismatched values ​​in the memristor cross-switch array are updated, while matched values ​​remain intact. In the second example method, the absolute change in the weight values ​​of the new model weights is evaluated against a threshold to determine whether the cross-switch array should be updated. In either example method, the weights are programmed by changing the conductivity of the memristors in the cross-switch array. In the first example method, if the weights encoded in the cross-switch cell using a specific conductivity perfectly match the new weights to be programmed in the same cell, the weights encoded in that cross-switch cell are not updated. In the second example method, the weights (i.e., the current weight value and the new weight value to be programmed) must not perfectly match to avoid reprogramming a particular cell. Instead, a threshold is introduced that determines the tolerance for the difference between the current and new values ​​of the weights. When training models using traditional hardware, weights are typically described. However, when the model is deployed on hardware powered by a cross-switch array, the conductivity of the memristors is used to represent the weights. Therefore, the threshold here is appropriate as a kind of difference between the current conductivity and the new conductivity programmed into a given cell.

[0038] The following are example steps for determining a threshold in one approach. First, determine the current conductivity C_current. Next, given new values ​​for the weights, estimate the new conductivity expected to be used to program C_new. This estimate may be associated with some uncertainty that can be calculated due to the known noise level in the memristor. While not necessary, the uncertainty of the calculation described below can be considered. Next, define a distance function between the two conductivity values. This distance function defines the similarity between the two values, and typically, multiple such distance functions can be used. An example of such a function is an absolute value: distance(C_current, C_new) = |C_current-C_new|. Another example of such a function could be a relative difference: distance(C_current, C_new) = (C_new – C_current) / C_current. Given a choice of distance function, choose a value referred to above as the threshold. If the selected distance between the current conductivity and the new conductivity to be programmed is below this threshold, the cross-connector is not reprogrammed. This threshold can be estimated empirically or chosen manually. For example, the threshold can fall within a certain conductivity range, which can be absolute or a fraction of the existing weights, such as 2% or 5% of the relative difference distance function mentioned above.

[0039] An example memristor cross-switch array used in neural network deep learning applications is now described. While a specific example of a memristor cross-switch array has been described, other configurations of memristor cross-switch arrays can be used. Figure 2 A memristor dot product engine 200 with a single voltage vector is illustrated. The dot product engine 200 includes a crossbar switch array 210 comprising N row electrodes 204 and M column electrodes 206. Each crossbar switch junction through the crossbar switch array 210 includes a memristor element 201, thereby forming a memory element of the dot product engine. Each memory element may include a memristor and a transistor connected in series with each other. The dot product engine 200 includes a vector input register or vector input 211 for applying a voltage to the row electrodes 204, and a vector output register or vector output 214 for receiving an output voltage derived from the current in the column electrodes 206. The vector input may be coupled to a digital-to-analog converter 261 to convert the digital value into an analog value for writing into the crossbar switch array 210. The vector output 214 may include an analog-to-digital converter 215 to convert the analog value into a digital value. The dot product engine 200 also includes sensing circuitry 216 for converting the current in the column electrodes 206 into a voltage. In the example, sensing circuit 216 includes an operational amplifier 218 and a resistor 221 that can be arranged to represent a virtual ground for a read operation.

[0040] The dot product engine 200 may also include other peripheral circuitry associated with the crossbar switch array 210, which is used as a storage device. For example, the vector input 211 may include a driver connected to the row electrode 204. An address decoder may be used to select the row electrode 204 and activate the driver corresponding to the selected row electrode 204. The driver for the selected row electrode 204 may drive the corresponding row electrode 204 with different voltages, corresponding to vector matrix multiplication or the process of setting the resistance value within the memristor element 201 of the crossbar switch array 210. Similar driver and decoder circuitry may be included for the column electrode 206. Control circuitry may also be used to control the voltage application at the input of the dot product engine 200 and the voltage reading at the output. Digital-to-analog converter circuitry and analog-to-digital converter circuitry may be used at the vector input 211 and the vector output 214. The input signals to the row electrode 204 and the column electrode 206 may be analog or digital. The peripheral circuitry described above may be fabricated using semiconductor processing techniques in the same integrated structure or semiconductor die as the crossbar switch array 210 in the above example. As described in more detail below, two main operations occur during the operation of the dot product engine. The first operation is programming the memristors in the cross-switch array to map mathematical values ​​from the N × M matrix to the array. In one example, one memristor is programmed at a time during the programming operation. The second operation is a dot product or matrix multiplication operation. In this operation, an input voltage is applied and an output voltage is obtained, which corresponds to the result of multiplying the N × M matrix by an N × 1 vector. The input voltage is below a threshold of the programming voltage, so the resistance values ​​of the memristors in array 210 are not changed during the matrix multiplication operation.

[0041] The dot product engine 200 may include an analog-to-digital converter 215 to convert the analog signal of the vector output register 214 into a digital value. The dot product engine 200 may include a digital-to-analog converter 261 to convert the digital value into an analog value for transmission to the vector input register 211.

[0042] The dot product engine 200 may be electrically coupled to the comparator processor 260. The comparator processor 260 may be integrally coupled to and formed as part of the dot product engine 200. The comparator processor 260 may be a separate component (such as an integrated circuit) or a separate processor. The comparator processor 260 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for fetching and executing instructions stored in a machine-readable storage medium. The comparator processor may fetch, decode, and execute instructions to control the process of comparing and / or evaluating data received from the dot product engine 215 (as indicated by arrow 215) with the input data source (as indicated by arrow 250 leading to the comparator processor). As an alternative or supplement to fetching and executing instructions, the comparator processor 260 may include one or more electronic circuits including electronic components for performing the functions of one or more instructions, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The comparator processor may include memory for storing executable instructions and / or be coupled to a separate storage medium 262.

[0043] Machine-readable storage media (such as 262) may include both volatile and non-volatile, removable and non-removable media, and may be any electronic, magnetic, optical or other physical storage device (containing or storing executable instructions, data structures, program modules or other data accessible to the comparator processor 260), such as firmware, erasable programmable read-only memory (EPROM), random access memory (RAM), non-volatile random access memory (NVRAM), optical discs, solid-state drives (SSDs), flash memory chips, etc. Machine-readable storage media may be non-transitory storage media, wherein the term "non-transitory" does not cover transient propagation signals.

[0044] Comparator processor 260 can perform methods for comparing and / or evaluating input vector data (as indicated by arrow 250) with output vector data (as indicated by arrow 255), which has been loaded into the memristor memory of dot product engine 200. Comparator processor 260 can provide a modified data value (as indicated by arrow 265) to another circuit or processor 270 to write the modified data value back into the memristor memory of dot product engine 200.

[0045] In deep learning processing, the comparator circuit described above reduces the number of write operations to the crossbar switch array, thereby accelerating overall computational efficiency and reducing the overall processing time during the training phase of the neural network. However, while the comparator circuit is advantageous for these reasons, further improvements and advantages (e.g., reductions in computational efficiency and overall processing time) can be achieved, as illustrated below.

[0046] Now for reference Figure 3 The illustration shows an example system for virtualizing a memristor DPE. Example system 350 includes a memristor DPE 300. As used herein, the term "memristor" can refer to a passive two-terminal circuit element that maintains a functional relationship between the time integral of current and the time integral of voltage. Further, the term "memristor dot product engine" broadly refers to a dot product engine that includes an array of memristor cross switches.

[0047] Example memristor DPE 300 includes at least one DPE resource 323. In various examples, DPE resource 323 includes at least one memristor crossbar switch array, which includes a plurality of row lines and a plurality of column lines intersecting these row lines. See below for reference. Figure 2 and Figure 3 The example DPE resource 323 is described in more detail.

[0048] The example memristor DPE 300 further includes an interface 363. In various examples, interface 363 is an input / output interface that allows the DPE 300 to be communicatively coupled to external components. For example, a processor of a computing system can be coupled to the DPE 300 through interface 363 and can gain access to DPE resource 323.

[0049] In example system 350, example memristor DPE 300 is further provided with a controller 371. In various examples, controller 371 can control various aspects of example memristor DPE 300. For example, controller 371 can determine which resources are available or allocated to various tasks or users. Controller 371 in example system 350 includes a scheduler 377 and a resource manager 379. For example, controller 371 or scheduler 377 of controller 371 is communicatively coupled to external interface 363 to facilitate access to DPE resource 323 by external components. Resource manager 379 can monitor the availability of various portions of DPE resource 323 and can selectively allocate resource 323 to entities such as external components described above.

[0050] In various examples, the memristor DPE 300 is virtualized by replicating its external interface 363. For example... Figure 3 As illustrated in the figure, example system 350 includes a replication interface 382.a Up to 382 n In various examples, any actual number of replication interfaces 382 can be made. a Up to 382 n Available. In some examples, system 350 can be configured with a maximum number of replication interfaces 382 to allow for sufficient resource availability.

[0051] Replication provides a mechanism by which a single physical memristor DPE 300 can appear as multiple separate physical devices to various users. In this regard, the physical memristor DPE 300 provides each user with a unique memory space, work queue, interrupt, and command processing via each replication interface. In one example, the physical memristor DPE 300 is a Peripheral Component Interconnect High Speed ​​(PCIe) based DPE, which can be configured to appear as multiple functions in the PCI configuration space. Each replication interface provides its own configuration space. Therefore, the physical memristor DPE 300 appears as multiple separate PCIe devices.

[0052] The physical memristor DPE 300 can be virtualized by providing a separate flow layer for each replication interface 382. Therefore, a user associated with a specific replication interface 382 can access the resources of the physical memristor DPE 300 (e.g., DPE resource 323) as a virtual DPE 385 through the corresponding replication interface 382. In this respect, similar to the coupling of the controller 371 to the external interface 363 of the physical memristor DPE 300, each replication interface 382... a Up to 382 n The communication ground is coupled to the controller 371.

[0053] For example, in Figure 3 In the example, the user can use the first copy interface 382 a Access the DPE resource 323 of the physical memristor DPE 300. In this regard, the first replication interface 382... a It can be used with the corresponding virtual DPE 385 a Related. Corresponding virtual DPE 385 a Different flow layers are accommodated by the controller 371 of the physical memristor DPE 300. In various examples, the scheduler 377 can allocate time slots or time slices to various flow layers or virtual DPEs 385. a Up to 385 n。

[0054] In various examples, scheduler 377 dynamically selects a streaming layer or virtual DPE 385 for execution via the appropriate replication interface 382 on each time slot. For example, scheduler 371 can support any of a variety of scheduling algorithms, such as round robin and weighted round.

[0055] In example system 350, the resource manager 379 of controller 371 can selectively allocate at least one resource to a virtual DPE during each time slot. For example, resource manager 379 can dynamically select a scheduling algorithm and manage the types of actions allowed to be performed by the user through appropriate virtual DPEs 385. For example, it can prevent specific flow layers or virtual DPEs from reconfiguring physical memristor DPEs.

[0056] In various examples, the scheduler 377 and resource manager 379 can facilitate the removal of layers from the scheduling algorithm. For example, in some cases, the entire physical memristor DPE 300 can be allocated to a specific flow layer for an extended number of time slots. In this regard, the specific flow layer can be a virtual DPE 385 or an external component coupled to the physical memristor DPE 300 via an external interface 363. In this regard, time slots or time slicing can also be disabled for extended periods.

[0057] The illustration also shows an example DPE resource 323 with a memristor DPE 300. Figure 3 Example system 350. DPE resource 323 comprises multiple tiles 321. In various examples, each tile 321 is a subset of DPE resource 323. For example, multiple tiles 321 can be assigned to a specific task or flow layer. In this respect, DPE resource 323 can be partitioned to accommodate and support any number of tasks simultaneously. Furthermore, tiles 321 can be arranged in any of a variety of ways. For example, in Figure 3 In the example, tiles 321 are arranged in an array.

[0058] The following detailed illustration is provided with reference to a tile 321 in the DPE resource 323 of the memristor DPE 300. Figure 3Example system. Example tile 301 is an example architecture using a memristor crossbar switch array. In this respect, example tile 301 includes a memory buffer 310 (e.g., an enhanced dynamic random access memory (or eDRAM) buffer) and various other components such as an output register 312 and a shift-add unit 314. The memory buffer 310 may be, for example, a DRAM buffer or another type, and may be used to store input values ​​provided to the crossbar switch array 322 described below. Further, the output register 312 may be used to aggregate the results from the shift-add unit 314, also as described below. Each tile of DPE resource 323 (such as tile 301) is provided with a control unit 316, which may be communicatively coupled to the controller 371 of the memristor DPE 300. In this respect, each tile 321 is coupled to the controller 371 via a bus, thereby providing connectivity between the controller 371 and the various components of each tile 321, such as the control unit 316. Example tile 301 is provided with a set of components referred to as in-situ multiply accumulate (IMA) unit 320. Figure 3 In the example, tile 301 is provided with eight IMA units 320.

[0059] exist Figure 3 The diagram illustrates a detailed example architecture of the IMA unit 320. In the illustrated example, the IMA unit 320 is configured with multiple memristor crossbar switch units 322. Figure 3 In the example, the IMA unit 320 is provided with four memristor cross-switch units 322. Each cross-switch unit 322 is coupled to a digital-to-analog converter (DAC) 324 and a sample-and-hold unit 326. Signals from the DAC 324 can be combined (e.g., via input register 328) and provided to a set of analog-to-digital converters (ADCs) 330. Figure 3 The IMA unit 320 further includes a control unit 332, which can be communicatively coupled to the controller 371 of the memristor DPE 300, for example, via the control unit 316 of the tile 301. The IMA unit 320 further includes various other components, such as an output register 334 and a shift-add unit 336. In various examples, the output register 334 of the IMA unit 320 may be part of or coupled to the output register 312 of the tile 301. Similarly, the shift-add unit 336 of the IMA unit 320 may be part of or coupled to the shift-add unit 314 of the tile 301.

[0060] In various examples, each memristor crossbar array 322 includes a plurality of row lines 344 and a plurality of column lines 342 intersecting the row lines 344. Memristor memory elements 346 are located at each intersection of the row lines 344 and column lines 342. Each memristor element 346 receives a programming vector signal representing a value within a matrix, a reference vector signal, and an operation vector signal representing the vector value to be multiplied by the matrix. In this regard, the row lines 344 of the crossbar array 322 are coupled to a DAC 324 to provide an input voltage 348 to the row lines 344 according to an input value stored in a memory buffer 310, thereby forming an input vector signal. Further, each column line 342 is coupled to a sample-and-hold unit 326 to receive current from the column line 342. The sample-and-hold unit 326 is coupled to an ADC 330 and a shift-and-add unit 336 to convert the current in the column line 342 into a voltage signal. In various examples of DPE, vector and matrix multiplication is performed by applying input voltage 348 to row line 344 and collecting current through column line 342 and measuring output voltage.

[0061] Therefore, a dot product operation can be performed on the crossbar switch array 322, and the result can be sent to the ADC 330 and aggregated in the output register 334 or the output register 312. For example, the aggregated result can be sent via the sigmoid operator 313 and stored in the memory buffer 310 for further processing.

[0062] The enhancements to machine learning (ML) algorithms have been continuously improved, thereby expanding the applicability of low-precision arithmetic for implementing computations in ML models. Currently, low-precision processing is limited to certain use cases (ML models) and types of input datasets, making the use of general-purpose accelerators in computational fields inefficient in terms of energy efficiency.

[0063] Moreover, many neural networks are "overkill" in terms of computation, power, and memory. Redundancy has traditionally been used to improve reliability by having spare copies of processing elements and switching from a failed component to its spare copy. A common hardware solution is to use tightly coupled processor cores running synchronously to compare outputs to resolve spoofing errors. A common software solution is to perform redundant computation in distributed systems to speed up ML model inference by selecting the fastest computation path. This redundancy often leads to a lot of optimization, such as quantization at different levels of precision. There is no systematic way to optimize this, and it usually requires retraining the model. Furthermore, energy efficiency is desirable for some applications, especially at the edge of power scarcity, just like performance in some real-time applications.

[0064] The solution to the aforementioned problems is an implementation using a baseline (production) accelerator, an optimization accelerator, and a control hardware accelerator, along with an operation that allows the production / optimization accelerator to alternate between production and optimization. The switching in this implementation will be seamless without any output glitches, meaning the baseline can be flushed while the optimization accelerator is effectively activated. This implementation can generate autonomous adjustments to Quality of Service (QoS) using dynamic policies. Customers can benefit from reconfigurable accelerators, resulting in cost savings and reduced hardware compared to purchasing multiple / redundant solutions for different use cases. Furthermore, adaptability facilitates infrastructure management, thus allowing for less manageability.

[0065] refer to Figures 4 to 6 The solution to the above problem is an implementation method that uses a baseline (production) accelerator, an optimization accelerator and a control hardware accelerator, and an operation that allows the production / optimization accelerator to switch alternately between production and optimization.

[0066] This document discloses an example hardware / software implementation that processes input data on the fly, routes future data with similar characteristics to appropriate ML hardware (with optimal arithmetic accuracy) supporting QoS and service level agreements (SLAs), and provides energy efficiency. This implementation relies on: a) preprocessing new data for specific time intervals; b) obtaining feedback from accelerators with different arithmetic accuracies; and c) recommending optimal arithmetic accuracy and corresponding accelerators for this data. Using this as a reference, any data matching similar characteristics will be routed to the optimal hardware platform in the future. This is implemented using a pair of accelerators, where one accelerator performs continuous evaluation while the other is in production mode.

[0067] This implementation does not keep the accelerator static; this disclosure describes an accelerator that adapts to changing environmental conditions and data and concept drift. Using two optimization / production accelerators, at any given point in time, one accelerator adapts while the other processes the data. Once the second accelerator has adapted to data drift, the accelerator changes its mode, and the trainable accelerator becomes the "optimized" accelerator. The accelerators do this continuously, thereby maintaining redundancy, providing the expected QoS, and adapting to data / concept drift.

[0068] Through this configuration (according to, for example) Figures 4 to 6 (Configuration), the accelerator can be configured to process preprocessed data when in optimized mode. Furthermore, through this configuration, the accelerator can be controlled to: change (e.g., multiple) operating parameters of the first accelerator when in optimized mode. Figure 5The implementation references, for example, weighted quantization, partial sum quantization, and / or activation quantization, operational parameters, and these operational parameters can also be at a higher level, i.e., coarser granularity, such as at the core level or array level, or by rerouting via, for example, short-circuiting certain layers, to determine the impact of each changed operational parameter on QoS (note that parameters can affect QoS by making the adapter perform better or worse than the baseline in terms of QoS objectives. Or similar actions can be taken. Switching will only be considered in the first / better case, not in the second / worse case and the third / similar case); after determining the optimal operational parameters of the first accelerator(s), the first accelerator is switched from optimized mode to production mode; the second accelerator is switched from production mode to optimized mode; and while in optimized mode, the operational parameters of the second accelerator(s) are changed to determine the impact of each changed operational parameter on QoS. Switching from one accelerator running a model in production mode and continuing to optimize that other accelerator can help improve accuracy and energy efficiency. This implementation also implements a broader set of adjustments, such as accuracy, power consumption, data rerouting, etc.

[0069] Now for reference Figure 4 The illustration depicts an example accelerator A (i.e., a memristor cross-switch array A) 410. Accelerator A 410 is generally similar to... Figure 3 The accelerator (memristor cross-switch array 322) shown is included, but with adaptations. Each accelerator 410 includes a plurality of row lines 444 and a plurality of column lines 442 intersecting the row lines 444. Memristor memory elements 446 are located at each intersection of the row lines 444 and column lines 442. Each memristor element 446 receives a programming vector signal for representing a value within a matrix, a reference vector signal, and an operation vector signal for representing the vector value to be multiplied by the matrix. In this respect, the row lines 444 are coupled to a DAC 424 to adjust the values ​​stored in a memory buffer (see [link to DAC 424]). Figure 3 The input value in the memory buffer 310 provides an input voltage 448 to the row line 444, thereby forming an input vector signal. Further, each column line 442 is coupled to an interface (INT) 427 to receive a current I from the column line 442. N The output signal from INT 427 is input to multiplexer 429, which combines the INT output signal. The output from multiplexer 429 is coupled to ADC 430 and shift-adder unit 436 to convert the current I in column line 442. N It is converted into a voltage signal. During operation, the input voltage 448 is applied to the row line 444 and the current I passing through the column line 442 is collected. N It measures the output voltage to perform vector and matrix multiplication. Figure 4The adapter / controller 480 coupled to accelerator A 410 is also illustrated. This implementation allows registers to measure power, performance, etc., at a fine granularity. This implementation also allows for the implementation of a pair of accelerators, where the active accelerator is in production mode and the adaptive accelerator is in optimization mode. Control of these accelerator modes can be implemented via physical or logical implementations. A physical implementation can be employed via a crossbar switch array (see [reference needed]). Figure 4 The columns indicated by dashed lines in the diagram); crossbar switches (see by...) Figure 4 The small dashed line in the text represents the XBar); or IMA unit, tile, chip, board (made of... Figure 4 (Represented by the large dashed lines in the figures). The logical implementation method is not shown in the figures, but can be seen from... g xy This allows for logical implementation via logical horizontal layers or neural networks.

[0070] Figure 5 The illustration shows an example accelerator (i.e., a memristor cross-switch array) 510 in optimized mode. As described above, Figure 5 The operating parameters of the accelerator can be modified to optimize its operation. Therefore, accelerator 510 operates (at least temporarily) as an optimized accelerator. Accelerator 510 includes... Figure 4 The components of accelerator 410 are shown in the figure. Figure 5 Weighted quantization 584 is described for multiple memristor elements 546. Weighted quantization 584 implements partial sum quantization 586 at the output of ADC 530, which enables activation quantization 588 at the output of shift-add unit 536.

[0071] Figure 6 The illustration shows an example adaptive accelerator system 600, depicting the sorting of data using data labels 690. Figure 6 The process of data flowing from both the baseline and optimized accelerators to the adapter / controller is described, which then decides whether and when to switch roles. The adaptive accelerator system 600 includes a baseline accelerator 610a, an optimized accelerator 610b, an adapter / controller 680, and a data tagging component 690. Data tagging allows inline tags to be added to the data stream for seamless switching. This is an example method for achieving proper sequencing without failures. Failures can be avoided by observing the data tags on both accelerators and synchronizing them based on those tags. Additionally, tags can be used for other purposes, such as classifying data to enable a faster optimization process, or identifying similar types of data that can be used to route to the optimal accelerator path (e.g., when multiple baseline accelerators are available).

[0072] The above reference Figure 4The data preprocessing mentioned can be performed by at least one of the accelerators, or, depending on the type of preprocessing, by other processing units in the pipeline preceding the accelerators. Several design patterns exist for feature transformation (data preprocessing). One option would be to embed the transformation step into the model graph (or deploy it along with the model). In this case, the service framework is responsible for placing the transformation and ML model components onto the appropriate components (e.g., an accelerator or CPU only for preprocessing and an accelerator for the ML model).

[0073] Another common design pattern is to use dedicated services to implement versioned data transformations of the ML model (feature store). In the latter case, the ML model accepts the input data as it is given and cannot be changed. Theoretically, in the former case, it is also possible to adapt the data transformations (precision) to various environmental conditions. Therefore, if the inference framework provides this capability, the preprocessing components can actually be deployed on a custom accelerator. In this case, the data precision of these data transformation components can be tuned.

[0074] At least one of the deployed accelerators can measure the overall device's QoS (power, latency, bandwidth) (represented by the baseline accelerator at each time point) and ML performance (e.g., accuracy, precision, recall, etc.) at different granular levels (from cross switches to layers, to chips, to end-to-end neural networks). This information is processed and used for algorithm execution in the adapter / controller, depending on the type of granularity of the desired precision information. This implementation is independent of which type of precision is used; that is, we can use different precision at the physical (in one configuration, for memristors and bit count; or for a Programmable Ultra-efficient Memristor-based Accelerator (PUMA) implementation, for columns, cross switches, chips, boards, etc.) or logical level (layers, neural networks). Therefore, accelerators themselves can function at the chip level, but they can alternatively (or additionally) break down the logical level in different ways. For example, a synchronous closed loop can be implemented at the cross switch level. Other implementation examples are also possible. Furthermore, inputs / weights / activations can be processed jointly or separately.

[0075] Devices (implemented in (and / or) software, hardware, firmware, etc.) can dynamically adapt to the ML models (e.g., neural networks) deployed in the accelerator to achieve the desired QoS within permissible accuracy constraints.

[0076] The controller / adaptor measures QoS increments to determine when to switch the accelerator from production mode to optimized mode or vice versa. Data tagging can be used to sort the data.

[0077] Some examples of implementing the switch include: when the accelerator optimizes to achieve a QoS superior to the baseline using input QoS specified parameters; when the optimization is superior to the baseline solely based on the specified QoS; when the default QoS policy of the device being tested can be optimized for reliability, power, and / or testing (these can be done periodically); and / or when the baseline begins to show errors.

[0078] Figure 7 This is a flowchart illustrating method 700 using an adaptive accelerator system. Method 700 includes setting a baseline and optimizing the accelerator (box 702). According to some examples, setting a baseline and optimizing the accelerator may refer to determining which accelerator initially operates in baseline mode and which accelerator initially operates in optimized mode.

[0079] Method 700 further includes setting a QoS target based on received operational parameters (box 704). In some examples, setting the QoS target includes determining a desired QoS target (e.g., accuracy, latency, bandwidth, power, and / or responsiveness) within the accuracy constraints allowed by the optimized accelerator. The result of setting the QoS target based on the received operational parameters is the identification of a QoS threshold.

[0080] Method 700 further includes initiating QoS measurement based on the use of the baseline accelerator (box 706). In some examples, initiating QoS measurement includes measuring QoS parameters derived from the use of the baseline accelerator, such as accuracy, latency, bandwidth, power, and / or responsiveness.

[0081] Method 700 further includes determining whether QoS criteria are met on the baseline accelerator (box 708). According to some examples, determining whether QoS criteria are met on the baseline accelerator may refer to determining whether the measured QoS is better than a QoS threshold. If the QoS criteria are met on the baseline accelerator, the method can proceed to box 704. In some examples, this loop can be repeated to keep the baseline accelerator in baseline mode (because the QoS criteria continue to be met). Once the QoS criteria are no longer (or are no longer met) on the baseline accelerator, the method can proceed to box 710. In this example, when the QoS criteria are no longer (or are no longer met) on the baseline accelerator, optimization of the baseline accelerator is expected to improve the QoS using the baseline accelerator. This improvement in QoS using the baseline accelerator will be achieved by switching the baseline accelerator to an optimized accelerator.

[0082] Method 700 further includes initiating the optimization process through a series of steps (box 710). According to some examples, the steps for initiating the optimization process may refer to steps implemented and controlled by the adapter / controller. The result of initiating the optimization process is the eventual switch from the baseline accelerator to the optimized accelerator.

[0083] Method 700 further includes determining whether QoS criteria are met on the optimized accelerator (box 712). According to some examples, determining whether QoS criteria are met on the optimized accelerator may mean determining whether the measured QoS is better than a QoS threshold. If QoS criteria are not (or no longer) met on the optimized accelerator, the method can proceed to box 710. This means that optimization is still expected, and therefore the optimized accelerator is maintained as the optimized accelerator. Once QoS criteria are met on the optimized accelerator, the method can proceed to box 714. This means that optimization of the optimized accelerator has been completed, thus determining that a switch from the optimized accelerator to the baseline accelerator should occur.

[0084] Method 700 further includes initiating the synchronization of the baseline and the optimized accelerator (box 714). According to some examples, the synchronization of the baseline and the optimized accelerator ensures that the two accelerators are properly identified by their respective baseline / optimization modes for use in an adaptive accelerator system. Synchronization of the baseline and the optimized accelerator will prevent situations where two accelerators are identified as belonging to the same baseline or optimization mode.

[0085] Method 700 further includes determining whether the baseline and the optimized accelerator are synchronized (box 716). In some examples, if the baseline and the optimized accelerator are not synchronized, the process can proceed to box 714. This loop is repeated until the two accelerators are synchronized. In some examples, if the baseline and the optimized accelerator are synchronized, the process can proceed to box 718.

[0086] Method 700 further includes performing an accelerator switch (box 718). In some examples, this causes a switch from an accelerator initially operating in baseline mode to an adaptive optimized accelerator, and a switch from an accelerator initially operating in optimized mode to an adaptive baseline accelerator.

[0087] Figure 8 This is a flowchart illustrating a method 800 using an adaptive accelerator system that includes data labeling capabilities. Method 800 includes setting a baseline and optimizing the accelerator (box 802). According to some examples, setting a baseline may refer to determining which accelerator initially operates in baseline mode and which accelerator initially operates in optimization mode.

[0088] Method 800 further includes setting a QoS target from the received operating parameters (box 804). In some examples, setting the QoS target includes determining a desired QoS target within the accuracy constraints allowed by the optimized accelerator. The result of setting the QoS target from the received operating parameters is the identification of a QoS threshold.

[0089] Method 800 further includes adding input data with data labels (box 806). According to some examples, adding input data labels could be adding input data labels to an input data stream for measuring QoS on a baseline accelerator and selecting a baseline accelerator. In some examples, using data labels when measuring QoS on a baseline accelerator and selecting a baseline accelerator can enable proper sorting of data without errors, data classification, etc.

[0090] Method 800 further includes initiating QoS measurement using the baseline accelerator (box 808). In some examples, initiating QoS measurement includes measuring QoS parameters derived from the use of the baseline accelerator, such as accuracy, latency, bandwidth, power, and / or responsiveness.

[0091] Method 800 further includes determining whether QoS criteria are met on the baseline accelerator (box 810). According to some examples, determining whether QoS criteria are met on the baseline accelerator may mean determining whether the measured QoS is better than a QoS threshold. If QoS criteria are met on the baseline accelerator, tag information is saved to select the baseline accelerator during primary data processing (box 812). Saving this tag information ensures that the baseline accelerator is used for primary data processing (compared to performance optimization reserved for the optimized accelerator). In some examples, once the tag information is saved to select the baseline accelerator during primary data processing, the method can proceed to box 804. In some examples, this loop can be repeated to keep the baseline accelerator in baseline mode (because QoS criteria continue to be met). Once QoS criteria are no longer (or are no longer) met on the baseline accelerator, the method can proceed to box 814. In this example, when QoS criteria are no longer (or are no longer) met on the baseline accelerator, optimization of the baseline accelerator is expected to improve the QoS of using the baseline accelerator. This improvement in QoS of using the baseline accelerator will be achieved by switching the baseline accelerator to the optimized accelerator.

[0092] Method 800 further includes initiating an optimization process through a series of steps (box 814). According to some examples, the steps for initiating the optimization process may refer to steps implemented and controlled by the adapter / controller. The result of initiating the optimization process is the final analysis of QoS standards on the optimized accelerator and the selection of an optimized accelerator.

[0093] Method 800 further includes adding input data labels (box 816). According to some examples, adding input data labels may refer to adding input data labels to the input data stream for analysis of QoS standards on the accelerator and selection of the accelerator. In some examples, using data labels when analyzing QoS standards on the accelerator and selecting the accelerator can enable proper sorting of data without errors, data classification, etc.

[0094] Method 800 further includes restarting the optimization process by going through the steps again (box 818). According to some examples, the step of restarting the optimization process can refer to the steps implemented and controlled by the adapter / controller. The result of restarting the optimization process (this time with data labeling) is the final analysis of QoS criteria on the optimization accelerator and selection of the optimization accelerator, while using data labeling to benefit from properly sorting the data without failure, classifying the data, etc.

[0095] Method 800 further includes determining whether QoS criteria are met on the optimized accelerator (box 820). According to some examples, determining whether QoS criteria are met on the optimized accelerator may mean determining whether the measured QoS is better than a QoS threshold. If QoS criteria are not (or no longer) met on the optimized accelerator, data tag information is saved to select the optimized accelerator during primary data processing (box 812). Saving this data tag information ensures that the optimized accelerator is used for optimization (compared to primary data processing reserved for the baseline accelerator). In some examples, once the tag information is saved to select the optimized accelerator, the method can proceed to box 814. This means that optimization is still expected, and therefore the optimized accelerator is maintained as the optimized accelerator along with the saved data tag. Once QoS criteria are met on the optimized accelerator, the method can proceed to box 824. This means that optimization of the optimized accelerator has been completed, thus determining that a switch from the optimized accelerator to the baseline accelerator should occur.

[0096] Method 800 further includes sharing data label information with the input stream to select an optimized accelerator (box 824). This means that the optimization of the optimized accelerator has been completed using the data label information shared via the data input stream, making it possible to identify when a switch from the optimized accelerator to the baseline accelerator should occur.

[0097] Method 800 further includes initiating the synchronization of the baseline and optimized accelerator using data tags (box 826). According to some examples, the synchronization of the baseline and optimized accelerator ensures that the two accelerators are properly identified by their respective baseline / optimization modes for use in an adaptive accelerator system. Synchronization of the baseline and optimized accelerator will prevent situations where two accelerators are identified as belonging to the same baseline or optimization mode. Furthermore, since data tags are observed on both accelerators, synchronizing these accelerators using data tags enables proper data sorting without errors, data classification, etc.

[0098] Method 800 further includes determining whether the baseline and the optimization accelerator are synchronized (box 828). In some examples, if the baseline and the optimization accelerator are not synchronized, the process can proceed to box 826. This loop is repeated until the two accelerators are synchronized (when using data labels). In some examples, if the baseline and the optimization accelerator are synchronized, the process can proceed to box 830.

[0099] Method 800 further includes performing an accelerator switch (box 830). In some examples, this causes a switch from an accelerator initially operating in baseline mode to an adaptive optimized accelerator, and a switch from an accelerator initially operating in optimized mode to an adaptive baseline accelerator.

[0100] Adaptive accelerator

[0101] Example implementations of adaptive accelerators use a baseline (production) accelerator (e.g., Figure 6The system comprises an accelerator 610a, an optimization accelerator (e.g., accelerator 610b), and a control (hardware) accelerator (e.g., adapter / controller 680), as well as operations for switching between production and optimization modes. Both main accelerators (i.e., production accelerator 610a and optimization accelerator 610b) have identical capabilities. One of the main accelerators 610a will be in production mode, while the other main accelerator 610b will be in optimization mode. Optimization is performed on the optimization accelerator 610b, such as fine-tuning weights, partial and quantization, and bypassing tiles and layers, to achieve better QoS by targeting QoS goals such as accuracy performance (e.g., latency, bandwidth), power, and / or responsiveness. The control accelerator (or adapter / controller 680) observes the execution of the optimization accelerator 610b and attempts to learn the behavior of both main accelerators 610a and 610b to achieve the requested / identified QoS goals (see the "Accelerator Switching" section below). The control accelerator 680 also instructs (controls) how the optimization accelerator 610b should adapt to achieve the QoS goals. Once the QoS target is achieved, the control accelerator 680 triggers a mode switch between the main accelerators 610a and 610b, whereby the earlier production accelerator 610a becomes the optimized accelerator 610b, and the earlier optimized accelerator 610b becomes the production accelerator 610a. This switching can be achieved by gating the output of one or the other of the main accelerators 610a and 610b. However, ideally, the switching will be performed at the appropriate time to avoid failures in the streaming data.

[0102] The QoS (Quality of Service) of the main accelerators 610a and 610b can be expressed in terms of accuracy, power, bandwidth, latency, responsiveness, and accuracy-speed, while employing feedback-based training techniques using memory about how previous optimizations worked to achieve fine-tuning of optimizations as instructed by the adapter / controller 680. Example target specifications would be: accuracy >80%, power <20 mW, bandwidth >10 MB / s, latency <10 ms, etc.; and the speed at which targets (such as accuracy, power, bandwidth, latency, and / or responsiveness) are achieved. The speed at which these targets are achieved can be expressed as the time to reach each of the listed targets. QoS can be measured in different ways. Accuracy can be measured by comparing its trend and absolute value to its own accuracy or to a baseline threshold. The power of the devices used with the main accelerators 610a and 610b can be measured or calculated; latency and bandwidth can be measured by the corresponding difference between the input and output. Feedback on optimization trends (i.e., optimization progress) is used to compare the trends of QoS objectives over different time intervals (short-term, medium-term, and long-term) to achieve an appropriate response to optimization. Algorithms can be expressed regarding how these strategies are achieved (see the "Switching Algorithms" section below).

[0103] Autonomous adjustment of QoS can leverage dynamic strategies. In other words, the QoS defined by performance metrics does not need to be constant. It can be changed by human operators or can dynamically evolve / adapt as internal or external conditions change. QoS can be defined or represented in two ways: QoS can be represented as an aggregate criterion that combines individual performance metrics using a linear combination of individual performance metrics; or QoS can use an optimization metric that simultaneously depends on other optimization metrics that serve as satisfaction metrics. The model should satisfy the satisfaction metrics, and then the optimization accelerator 610b can be further optimized based on the optimization metrics.

[0104] Regardless of whether Option 1 or Option 2 is used, a single QoS performance metric can evolve dynamically. The following example factors may affect QoS changes: available power (for edge devices with limited battery power, the trade-off between model accuracy and power consumption can change depending on battery state); degradation of external sensors or actuators (for models with high replacement costs, model accuracy constraints can be relaxed when external sensors or actuators degrade); and / or specific geographic areas / locations of interest (this can increase performance constraints while relaxing power constraints when the model finds itself in a location of high interest).

[0105] accelerator switching The adapter / controller 680 measures the QoS increment to determine when to switch. Once the QoS target is achieved for the first time, the modes of the primary accelerators 610a and 610b can be switched, but this does not mean they will switch immediately; that is, to initiate the switching process, the optimized accelerator 610b can be marked as... switchable candidate In this example, sufficient time can be expected to ensure stable QoS targets are achieved, preventing handovers from triggering reverse handovers. Once stability is guaranteed, the optimized accelerator 610b can be labeled as... switchable stable This approach can be extended to multiple optimization accelerators (one of which is, for example, optimization accelerator 610b), each of which can... optimization state( switchable candidate and switchable stable In any of the following states.

[0106] To achieve a seamless switchover without any output failures, data from the baseline accelerator 610a needs to be flushed while the optimized accelerator 610b is effectively activated. From an observer outside the box, there should be no significant difference during the switchover process; that is, the switchover should be seamless. One way to achieve a seamless switchover is to redirect data from the output of one master accelerator to the output of the other. Since the two master accelerators 610a and 610b can be out of sync, the faster master accelerator may have to wait for the slower one to catch up, allowing the adapter / controller 680 to implement flow control and / or buffer the slower master accelerator to increase its speed to at least the speed of the faster master accelerator.

[0107] One way to achieve proper data sorting without errors is by using data tagging (see...). Figure 6 and Figure 8 Output failures can be avoided by observing the data tags on the two main accelerators 610a and 610b during the baseline / optimization accelerator selection process and synchronizing the main accelerators 610a and 610b using the data tags. Additionally, the tags can be used for other purposes, such as: classifying data to enable a faster optimization process; and identifying similar types of data and routing that similar type of data to the optimal baseline accelerator path (if multiple baseline accelerators (one of which is, for example, baseline accelerator 610a) are available). In this way, the baseline accelerator determined to be most suitable for processing a specific type of data is selected as the destination baseline accelerator, thus making data throughput more efficient by using the optimal baseline accelerator.

[0108] Switching Algorithm When optimizing accelerator 610b results in better QoS compared to the input QoS using baseline accelerator 610a, an algorithmic approach may be needed to switch accelerators. In this scenario, the desired switch is from optimized accelerator 610b to production accelerator 610a. A QoS target is specified as a parameter sent to adapter / controller 680. These parameters can be sent during runtime (in-band) or during management time (out-of-band). Adapter / controller 680 measures the QoS using optimized accelerator 610b and determines when to switch the optimized / production mode of the primary accelerators 610a and 610b. The strategy used in making this decision can be complex, with specific criteria including timing. For example, criteria might depend on whether power is within the envelope, whether bandwidth / latency criteria are met, etc. The QoS using baseline accelerator 610a may also drift and deteriorate, causing optimization to perform better over time, thus leading to a switch between optimized and production modes.

[0109] The optimized accelerator 610b significantly outperforms the baseline accelerator 610a, regardless of the specified QoS target / threshold. When the optimized accelerator 610b achieves its QoS and outperforms the baseline accelerator 610a's QoS, the primary accelerators 610a and 610b switch modes. In some cases, it makes sense to immediately switch the modes of the primary accelerators 610a and 610b when one primary accelerator significantly outperforms the other. One use case that fits this scenario is when a primary accelerator recovers from a previous low-power state, and then the relaxed QoS of the current active primary accelerator can be used. Instead of waiting for the optimized accelerator 610b to achieve the new target QoS, a temporary QoS can be set to "slightly better than the active primary accelerator." The primary accelerators 610a and 610b can thus quickly change modes to maintain a better QoS level.

[0110] The default QoS policy of the adapter / controller 680 can be periodically executed to optimize reliability, power, testing, etc. In addition to optimization policies, the main accelerators 610a and 610b can also (or alternatively) perform certain testing policies, such as comparing the outputs of the two main accelerators 610a and 610b against a predefined dataset or data spectrum. Power consumption can be calibrated against predefined limits, and the reliability of each of the main accelerators 610a and 610b can be evaluated.

[0111] Baseline accelerator 610a may begin to produce errors or its performance may dynamically degrade over a short period of time. This may occur due to a fault in the hardware, system software, or firmware. When this happens, the main accelerators 610a and 610b can switch modes even if the optimized accelerator 610b has not yet met its QoS constraints.

[0112] Some examples of possible hardware-specific variations (non-software) that can be used to perform optimizations include: coarse (core-level) quantization versus array-level quantization; rerouting (short-circuiting) certain layers of a neural network; offline (during non-operational periods of the accelerator system) versus real-time optimization (during operational periods of the accelerator system); training quantization-friendly master accelerators (e.g., master accelerators 610a, 610b) and extending training to counteract other accelerator (hardware) specific losses, such as: loss function plus model (half-precision or single-precision) and simulating inference on hardware; and real-time computation of loss to make training hardware-friendly from the start.

[0113] Some examples of adaptive levels include: static (non-pattern-adaptive) master accelerators, where labeled data is then rerouted to an appropriate dynamic (pattern-adaptive) master accelerator (e.g., where multiple master accelerators with the same pattern are used (one of which is, for example, one of master accelerators 610a and 610b)); dynamically adapting (reconfiguring) master accelerators (one of which is, for example, one of master accelerators 610a and 610b) for inference purposes (e.g., for performing inference on hardware); dynamically adapting (reconfiguring) master accelerators (one of which is, for example, one of master accelerators 610a and 610b) for training purposes; and analyzing / learning data streams to reconfigure master accelerators (one of which is, for example, one of master accelerators 610a and 610b).

[0114] Figure 9 The illustrations depict example computing components that can be used to implement dynamically modular and customizable computing systems according to various aspects of this disclosure. Reference is now made to... Figure 9 The computing component 900 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 9 In an example implementation, computing unit 900 includes a hardware processor 902 and a machine-readable storage medium 904. In some examples, computing unit 900 may be... Figure 1 The memristor cross switch array 100 corresponds to the system aspect.

[0115] Hardware processor 902 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for fetching and executing instructions stored in machine-readable storage medium 904. Hardware processor 902 may fetch, decode, and execute instructions (such as instructions 906 to 930) to control processes or operations for implementing a dynamically modular and customizable computing system. As an alternative or supplement to fetching and executing instructions, hardware processor 902 may include one or more electronic circuits, including electronic components for the function of executing one or more instructions, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other electronic circuits.

[0116] Machine-readable storage media (such as machine-readable storage media 904) can be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Therefore, machine-readable storage media 904 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), storage devices, optical discs, etc. In some examples, machine-readable storage media 904 can be a non-transitory storage medium, wherein the term "non-transitory" does not cover transient propagation signals. As described in detail below, machine-readable storage media 904 can be encoded with executable instructions, such as instructions 906 to 930.

[0117] Hardware processor 902 can execute instruction 906 to transfer preprocessed data to the first accelerator. This means that the first accelerator initially and continuously functions to process data when in optimized mode (i.e., as an optimized accelerator, for example, Figure 6 (Accelerator 610b in the middle).

[0118] The hardware processor 902 can execute instructions 908 to change at least one operating parameter of the first accelerator. This occurs during the operation of the first accelerator.

[0119] The hardware processor 902 can execute instruction 910 to monitor the impact of each changed operating parameter of the first accelerator on the Quality of Service (QoS). This means monitoring the QoS of using the first accelerator with the changed operating parameters.

[0120] The hardware processor 902 can execute instruction 912 to change at least one operating parameter of the first accelerator based on the monitored impact of each changed operating parameter of the first accelerator on QoS. This means making an attempt to improve QoS by further changing the operating parameters of the first accelerator.

[0121] Hardware processor 902 can execute instruction 914 to determine whether a threshold has been met for QoS. An example of a threshold could be a percentage of QoS (e.g., latency, bandwidth, or a specific metric such as accuracy) (e.g., 95%). Alternatively, a threshold can be understood as how much imperfection the accelerator can tolerate. For example, a machine learning expert might specify a desired level of model accuracy, such as 91%. It should be noted that hardware experts can anticipate a certain level of uncertainty associated with the programmable conductivity of the memristor cells. For example, 91% accuracy could translate to a range of, for example, 89% to 93%. This means that with a QoS target of 91%, actual accuracy above 89% is acceptable. In another possible scenario, a degradation in hardware level can be observed that determines the actual programmable accuracy of the cross-connect unit. In this case, assuming the degradation and its impact on QoS metrics can be measured or estimated, the threshold can be increased when a larger difference between, for example, desired and actual performance is tolerable. For example, in the example above, the expected threshold range difference from the target could be equal to 2 percentage points (where 91% is the target, and actual accuracy above 89% is acceptable). Then, as the ability to precisely reprogram the cross switch array decreases, moving to a target of 91% may be acceptable with a wider threshold range difference equal to, for example, 5 percentage points. Typically, hardware accuracy above 86% is acceptable.

[0122] Once it is determined that the QoS threshold has been met (meaning that the first accelerator has been fully optimized and no further optimization of the first accelerator is needed), the hardware processor 902 can execute instruction 916 to route unprocessed data to the first accelerator (which will now be used as the baseline accelerator). Figure 6 The accelerator 610a is configured to execute instruction 918 to transfer additional preprocessed data to a second accelerator (which is configured to optimize the accelerator, e.g., accelerator 610b). This configures the second accelerator for analyzing the impact on QoS from final changes to the parameters of the second accelerator.

[0123] The hardware processor 902 can execute instruction 920 to change at least one operating parameter of the second accelerator. This occurs during the operation of the second accelerator.

[0124] Hardware processor 902 can execute instruction 922 to monitor the impact of each changed operating parameter of the second accelerator on QoS. This means monitoring the QoS of using the second accelerator with the changed operating parameters.

[0125] The hardware processor 902 can execute instructions 924 to change at least one operating parameter of the second accelerator based on the monitored impact of each changed operating parameter of the second accelerator on QoS. This means making an attempt to improve QoS by further changing the operating parameters of the second accelerator.

[0126] Hardware processor 902 can execute instruction 926 to determine whether another threshold has been met for QoS. Once it is determined that the other threshold has been met (meaning that the second accelerator has been fully optimized and no further optimization of the second accelerator is required), hardware processor 902 can execute instruction 928 to route unprocessed data to the second accelerator (which will now be used as the baseline accelerator (e.g., accelerator 610a)) and execute instruction 930 to transfer additional preprocessed data to the first accelerator (which is now set to optimize the accelerator (e.g., accelerator 610b)).

[0127] In some examples, QoS includes at least one of the following: (i) accuracy; (ii) latency; (iii) bandwidth; (iv) power; and (v) responsiveness.

[0128] In some examples, the two modification steps include at least one of the following: (i) fine-tuning the weights; (ii) partial and quantization; (iii) bypassing the tiles; and (iv) bypassing the layers.

[0129] In some examples, during both the preprocessing data transmission step, the modification step, and the monitoring step, the first accelerator is in optimization mode and the second accelerator is in production mode; and during the routing step and the transmission of additional preprocessing data step, the first accelerator is in production mode and the second accelerator is in optimization mode.

[0130] In some examples, when it is determined that the QoS threshold has been met, the first accelerator switches from optimized mode to production mode, and the second accelerator switches from production mode to optimized mode.

[0131] In some examples, when it is determined that another threshold has been met for QoS, the second accelerator switches from optimized mode to production mode, and the first accelerator switches from production mode to optimized mode.

[0132] In some examples, the first accelerator and the second accelerator have different arithmetic precision.

[0133] Figure 10 A block diagram depicts an example computer system 1000 in which various aspects of the aspects described herein may be implemented. The computer system 1000 can be used to execute machine-readable instructions to perform the processes described herein. In some examples, the computer system 1000 may correspond to... Figure 1 The modular switching system 100 or the currently configured switching system 140 may be used, but should not be limited to the components disclosed herein.

[0134] Computer system 1000 includes a bus 1002 or other communication mechanism for transmitting information, and one or more hardware processors 1004 coupled to the bus 1002 for processing information. The hardware processors 1004 may be, for example, one or more general-purpose microprocessors.

[0135] Computer system 1000 also includes main memory 1006, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 1002, for storing information and instructions to be executed by processor 1004. Main memory 1006 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 1004. When such instructions are stored in storage media accessible to processor 1004, computer system 1000 becomes a dedicated machine customized to perform the operations specified in the instructions.

[0136] The computer system 1000 further includes a read-only memory (ROM) 1008 or other static storage device coupled to the bus 1002 for storing static information and instructions for the processor 1004. A storage device 1010, such as a disk, optical disk, or USB thumb drive (flash memory drive), is provided and coupled to the bus 1002 for storing information and instructions.

[0137] Computer system 1000 can be coupled to display 1012, such as a liquid crystal display (LCD) (or touchscreen), via bus 1002 for displaying information to a computer user. Input device 1014, including alphanumeric keys and other keys, is coupled to bus 1002 for transmitting information and command selections to processor 1004. Another type of user input device is cursor control 1016, such as a mouse, trackball, or arrow keys, for transmitting directional information and command selections to processor 1004 and for controlling cursor movement on display 1012. In some examples, the same directional information and command selections as with cursor control can be implemented via receiving touch on the touchscreen without a cursor.

[0138] The computing system 1000 may include a user interface module for implementing a GUI, which may be stored as executable software code executed by (multiple) computing devices in a mass storage device. By way of example, this module and other modules may include components such as software components, object-oriented software components, class components, and task components; processes; functions; attributes; programs; subroutines; program code segments; drivers; firmware; microcode; circuits; data; databases; data structures; tables; arrays; and variables.

[0139] Generally, terms such as “component,” “engine,” “system,” “database,” and “data storage area” as used herein can refer to logic implemented in hardware or firmware, or to a set of software instructions written in programming languages ​​such as Java, C, or C++, which may have entry and exit points. Software components can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. It should be understood that software components can be invoked from other components or from themselves, and / or can be invoked in response to detected events or interrupts. Software components configured to execute on a computing device can be provided on computer-readable media such as optical discs, digital video discs, flash memory drives, disks, or any other tangible media, or can be provided as digital downloads (and can be stored originally in a compressed or installable format that needs to be installed, decompressed, or decrypted before execution). Such software code can be stored, in part or in whole, on a memory device executing the computing device for execution by the computing device. Software instructions can be embedded in firmware such as EPROM. It will be further understood that hardware components may include connection logic units such as gates and flip-flops, and / or may include programmable units such as programmable gate arrays or processors.

[0140] Computer system 1000 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic. This custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, combined with the computer system, enable computer system 1000 to be a special-purpose machine or to be programmed as such. According to one example, the techniques described herein are executed by computer system 1000 in response to processor(s) 1004 executing one or more sequences of one or more instructions contained in main memory 1006. Such instructions may be read into main memory 1006 from another storage medium, such as storage device 1010. Execution of the instruction sequence contained in main memory 1006 causes processor(s) 1004 to perform the process steps described herein. In alternative examples, hardwired circuitry may be used in place of or in combination with software instructions.

[0141] As used herein, the term "non-transitory media" and similar terms refer to any medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such non-transitory media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 1010. Volatile media include dynamic memory, such as main memory 1006. Common forms of non-transitory media include, for example, floppy disks, floppy disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, flash memory EPROMs, NVRAMs, any other memory chips or cartridges, and networking versions of these media.

[0142] Non-transient media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transmission of information between non-transient media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing bus 1002. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0143] Computer system 1000 also includes a communication interface 1018 coupled to bus 1002. Communication interface 1018 provides bidirectional data communication coupled to one or more network links connected to one or more local networks. For example, communication interface 1018 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for providing data communication connectivity to a corresponding type of telephone line. As another example, communication interface 1018 may be a local area network (LAN) card for providing data communication connectivity to a LAN-compatible network (or a WAN component communicating with a WAN). Wireless links may also be implemented. In any such implementation, communication interface 1018 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0144] Network links typically provide data communication to other data devices via one or more networks. For example, a network link can provide a connection from a local network to a host computer or to a data device operated by an Internet Service Provider (ISP). The ISP, in turn, provides data communication services through a global packet data communication network now commonly referred to as the "Internet." Both local networks and the Internet use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks, as well as signals on network links and signals through communication interface 1018 (which carries digital data to and from computer system 1000), are example forms of transmission media.

[0145] Computer system 1000 can send messages and receive data including program code through (multiple) networks, network links, and communication interface 1018. In the Internet example, the server can transmit application request code through the Internet, ISP, local network, and communication interface 1018.

[0146] The received code can be executed by processor 1004 and / or stored in storage device 1010 or other non-volatile storage device for later execution upon receipt.

[0147] Each of the processes, methods, and algorithms described in the preceding sections can be implemented in and automated, wholly or partially, by code components executed by one or more computer systems or computer processors, including computer hardware. The one or more computer systems or computer processors can also operate to support the execution of related operations in a “cloud computing” environment or as “software as a service” (SaaS). These processes and algorithms can be implemented, partially or wholly, in a dedicated circuit. The various features and processes described above can be used independently of each other or combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of this disclosure, and certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and blocks or states associated with these methods and processes can be executed in a suitable other order, or can be executed in parallel, or can be executed in some other way. Blocks or states can be added to or removed from the disclosed examples. The execution of certain operations or processes can be distributed across computer systems or computer processors, so that they are not confined to a single machine but deployed across multiple machines.

[0148] As used herein, circuits can be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to constitute a circuit. In implementations, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be shared partially or wholly among one or more circuits. Although various features or functional elements can be described or claimed individually as separate circuits, these features and functions can also be shared within one or more common circuits, and such description should not require or imply the need for separate circuits to implement such features or functions. In cases where software is used to implement the circuit wholly or partially, such software can be implemented to operate in conjunction with a computing or processing system (such as computer system 800) capable of performing the functions described with respect to the software.

[0149] As used herein, the term “or” can be interpreted in an inclusive or exclusive sense. Furthermore, descriptions of resources, operations, or structures in the singular form should not be construed as excluding the plural. Unless expressly stated otherwise, or understood otherwise in the context in which they are used, conditional language (among others, such as “can,” “could,” “might,” or “may”) is generally intended to convey that certain features, elements, and / or steps are included in some examples of this disclosure while not included in others.

[0150] Unless otherwise expressly stated, the terms and phrases used in this document, and their variations thereof, should be interpreted as open-ended rather than restrictive. Adjectives and similar terms such as “conventional,” “traditional,” “normal,” “standard,” and “known” should not be interpreted as limiting the described items to items available for a given period of time or prior to a given time, but should be understood to encompass traditional, conventional, normal, or standard techniques that may be available or known at any time now or in the future. In some cases, the presence of broad words and phrases (such as “one or more,” “at least,” “but not limited to,” or other similar phrases) should not be interpreted as indicating an intention or requirement for a narrower scope where such broad phrases may not exist.

Claims

1. A non-transitory machine-readable medium storing instructions that cause one or more processors of an electronic device to perform operations, the operations including: The preprocessed data is transmitted to the first accelerator; Change at least one operating parameter of the first accelerator, wherein changing the at least one operating parameter includes at least one of fine-tuning weights, partial and quantization, bypassing tiles, and bypassing layers; Monitor the impact of each changed operating parameter of the first accelerator on the Quality of Service (QoS), wherein the QoS includes at least one of accuracy, latency, bandwidth, power, and responsiveness; The at least one operating parameter of the first accelerator is changed based on the monitored impact of each changed operating parameter of the first accelerator on the QoS. as well as Determine whether the QoS has met the threshold, wherein, once it is determined that the QoS has met the threshold, the following additional operations are performed: Routing unprocessed data to the first accelerator; and The additional preprocessed data is transferred to the second accelerator; Thus, the first accelerator can adapt by switching from optimization mode to production mode.

2. The non-transitory machine-readable medium as described in claim 1, wherein, The operation further includes: Change at least one operating parameter of the second accelerator; Monitor the impact of each changed operating parameter of the second accelerator on the QoS; The at least one operating parameter of the second accelerator is changed based on the monitored impact of each changed operating parameter of the second accelerator on the QoS; and Determine whether the QoS has met another threshold, wherein, once it is determined that the QoS has met the other threshold, the following additional operations are performed: Routing unprocessed data to the second accelerator; and The additional preprocessed data is transmitted to the first accelerator.

3. The non-transitory machine-readable medium as described in claim 1, wherein: During the transmission of preprocessed data, the two modification steps, and the monitoring step, the first accelerator is in optimization mode and the second accelerator is in production mode. and During the routing step and the step of transmitting additional preprocessed data, the first accelerator is in production mode and the second accelerator is in optimization mode.

4. The non-transitory machine-readable medium as described in claim 1, wherein, Once it is determined that the QoS has met the threshold, the first accelerator switches from optimization mode to production mode, and the second accelerator switches from production mode to optimization mode.

5. The non-transitory machine-readable medium as described in claim 2, wherein, Once it is determined that the QoS has met the other threshold, the second accelerator switches from optimization mode to production mode, and the first accelerator switches from production mode to optimization mode.

6. The non-transitory machine-readable medium as claimed in claim 1, wherein, The first accelerator and the second accelerator have different arithmetic accuracies.

7. A method for optimizing an accelerator, comprising: Set up a first accelerator that initially operates in baseline mode and a second accelerator that initially operates in optimization mode; The Quality of Service (QoS) target is set based on parameters received from an adapter communicatively connected to the first and second accelerators, wherein the QoS includes at least one of accuracy, latency, bandwidth, power, and responsiveness. Measure the QoS of the first accelerator. as well as Determine whether the measured QoS of the first accelerator has met the QoS target associated with the first accelerator, wherein if the QoS target associated with the first accelerator is met, the first accelerator is maintained as a baseline accelerator, and wherein if the QoS target associated with the first accelerator is not met, the first accelerator is optimized by switching the first accelerator to an adapted optimized accelerator.

8. The method of claim 7, wherein, The method further includes: Measuring the QoS of the second accelerator; and Determine whether the measured QoS of the second accelerator has met the QoS target associated with the second accelerator, wherein if the QoS target associated with the second accelerator is not met, the second accelerator is maintained as an optimized accelerator, and wherein if the QoS target associated with the second accelerator is met, the second accelerator is no longer optimized by switching the second accelerator to an adapted baseline accelerator.

9. The method of claim 8, wherein, The method further includes: The first accelerator and the second accelerator are synchronized before the first accelerator is switched to the adapted optimized accelerator and before the second accelerator is switched to the adapted baseline accelerator.

10. The method of claim 7, wherein, The first accelerator and the second accelerator have different arithmetic accuracies.

11. An accelerator system, comprising: First accelerator; Second accelerator; processor; as well as A memory configured to store instructions that, in response to execution by the processor, cause the processor to perform the following operations: Change at least one operating parameter of the first accelerator, wherein changing the at least one operating parameter includes at least one of fine-tuning weights, partial and quantization, bypassing tiles, and bypassing layers; Monitor the impact of each changed operating parameter of the first accelerator on the Quality of Service (QoS), wherein the QoS includes at least one of accuracy, latency, bandwidth, power, and responsiveness; The at least one operating parameter of the first accelerator is changed based on the monitored impact of each changed operating parameter of the first accelerator on the QoS. as well as Determine whether the QoS has met the threshold, wherein once it is determined that the QoS has met the threshold, the first accelerator is configured to switch from optimization mode to production mode, and the second accelerator is configured to switch from production mode to optimization mode.

12. The accelerator system of claim 11, wherein, The instruction further causes the processor to perform the following operations: Change at least one operating parameter of the second accelerator; Monitor the impact of each changed operating parameter of the second accelerator on the QoS; At least one operating parameter of the second accelerator is changed based on the monitored impact of each changed operating parameter of the second accelerator on the QoS; as well as Determine whether the QoS has met another threshold, wherein once it is determined that the QoS has met the other threshold, the second accelerator is configured to switch from optimization mode to production mode, and the first accelerator is configured to switch from production mode to optimization mode.

13. The accelerator system of claim 11, wherein, The first accelerator was initially configured to receive preprocessed data; and Once it is determined that the QoS has met the threshold, the instruction further causes the processor to perform the following operations: Routing unprocessed data to the first accelerator; and The additional preprocessed data is transmitted to the second accelerator.

14. The accelerator system of claim 13, wherein, During the period when the first accelerator receives the preprocessed data, during the two change steps, and during the monitoring step, the first accelerator is configured in optimization mode and the second accelerator is configured in production mode; and During the routing step and the step of transmitting additional preprocessed data, the first accelerator is in production mode and the second accelerator is in optimization mode.

15. The accelerator system of claim 12, wherein, Once it is determined that the QoS has met the other threshold, the instruction further causes the processor to perform the following operations: Routing unprocessed data to the second accelerator; and The additional preprocessed data is transmitted to the first accelerator.

16. The accelerator system of claim 11, wherein, The first accelerator and the second accelerator have different arithmetic accuracies.

Images