Non-private and private inference scheduling method and device based on reconfigurable chip

Abstract

The application provides a non-privacy and privacy reasoning scheduling method based on a reconfigurable chip, which comprises the following steps: determining whether the privacy requirement of a neural network reasoning task is privacy reasoning; if yes, performing a first scheduling step; otherwise, performing a second scheduling step; in the first scheduling step, analyzing an arithmetic logic unit required by the privacy reasoning, combining a reconfigurable multiplier and an adder by a data layout converter to form the arithmetic logic unit, and reconfiguring an interconnection network of a processing unit array into a butterfly network or a SIMD data path to obtain a reconfigurable chip and perform a reasoning step; in the second scheduling step, setting a slice as a tensor mode by the data layout converter, independently and parallelly working all the slices, performing a multiply-accumulate operation by each slice, transmitting data between processing units PEs through horizontal and vertical links to form a pulsating data stream, forming a pulsating array by the processing unit array, obtaining the reconfigurable chip, and performing the reasoning step; and in the reasoning step, performing the neural network reasoning task by the reconfigurable chip to obtain a reasoning result.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit technology, and particularly to a reconfigurable chip and scheduling method, apparatus, electronic device, computer-readable storage medium and computer program product for unifying non-privacy and privacy reasoning. Background Technology

[0002] With the widespread adoption of cloud computing and artificial intelligence, cloud data centers need to provide neural network inference services for users with varying privacy needs. One type of user submits data in plaintext for non-private inference (NPI); the other type is privacy-sensitive users who require data encryption using technologies such as fully homomorphic encryption (FHE) for privacy inference (PI) on the encrypted data to protect data security. Both non-private and privacy inference are inference tasks; the difference lies in whether the user-uploaded data is encrypted. Taking facial recognition as an example, non-private inference directly performs inference on the user-uploaded facial photo, while privacy inference first encrypts the facial data locally on the user's device before uploading it to the cloud server for inference (in this case, even if data leakage occurs, it will not compromise user privacy because the leaked data is encrypted).

[0003] Currently, there are two main solutions to address this mixed demand:

[0004] Option 1: Dedicated Accelerator Solution. This solution designs dedicated hardware accelerators for NPI and PI tasks separately, such as a Tensor Processing Unit (TPU) for NPI and an FHE accelerator (like SHARP) for PI. The advantages of this solution are high performance and low latency. Its main structure involves designing independent computing units, on-chip storage, and control logic for each computational mode (tensor computation and polynomial ring computation). However, the dedicated accelerator solution has disadvantages: 1. High cost: It requires designing, verifying, and manufacturing two separate chips for NPI and PI, doubling the cost of NRE, especially at advanced process nodes, where the cost can reach hundreds of millions of dollars. 2. Poor flexibility and low resource utilization: The accelerator architectures for NPI and PI differ significantly, making it impossible to share hardware resources. When the load on NPI and PI tasks in the data center is unbalanced (e.g., more PI requests at one time, more NPI requests at another), one accelerator may be idle while the other may be under resource strain, resulting in low overall hardware resource utilization and difficulty in guaranteeing Quality of Service (QoS).

[0005] Option 2: General-Purpose Hardware Solution. This solution implements NPI and PI computation tasks on general-purpose programmable hardware such as a General Purpose Graphics Processing Unit (GPGPU) or a Field-Programmable Gate Array (FPGA). The advantages of this solution are high flexibility and low non-recurring engineering (NRE) costs. However, the general-purpose hardware solution has disadvantages: 1. Low performance and energy efficiency: General-purpose hardware such as GPGPUs are not natively designed for the core operations in FHE (such as large integer modular arithmetic and number theory transformations). When performing PI tasks, their performance is far inferior to dedicated ASIC chips, and their power consumption is higher. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as Figure 8 As shown, this invention proposes a non-privacy and privacy-preservation inference scheduling method based on a reconfigurable chip, including:

[0007] The chip fabrication steps involve constructing a reconfigurable chip including a shared cache, a high-bandwidth memory controller, and multiple ring-tensor cores; each ring-tensor core contains a processing unit array, a data layout converter, and a local cache; the processing unit array is a two-dimensional array composed of multiple processing units (PEs), each PE is composed of multiple slices, and each slice contains a multiplier and an adder.

[0008] The decision step is to determine whether the privacy requirement of the neural network inference task is privacy inference. If it is, the first scheduling step is executed; otherwise, the second scheduling step is executed.

[0009] The first scheduling step involves analyzing the arithmetic logic unit required for the privacy inference. The data layout converter combines and reconstructs the multiplier and adder to form the arithmetic logic unit, and reconstructs the interconnection network of the processing unit array into a butterfly network or SIMD data path to obtain the reconstructed chip and execute the inference step.

[0010] In the second scheduling step, the data layout converter sets the slice to tensor mode, all slices work independently and in parallel, each slice performs multiply-accumulate operations, data is transmitted between processing units (PE) through horizontal and vertical links to form a pulsating data stream, the processing unit array forms a pulsating array, a reconstructed chip is obtained, and the inference step is executed.

[0011] The reasoning step involves using the reconstructed chip to perform the neural network reasoning task and obtain the reasoning result.

[0012] The neural network inference task is a face recognition task that includes images of faces, and the inference result is the face recognition result.

[0013] The aforementioned non-privacy and privacy-preserving inference scheduling method based on reconfigurable chips includes a data layout converter consisting of a vectorized exchange network for performing data rearrangement operations; the exchange network is configured as multiple sub-networks for handling tensor transpose; and the exchange network as a whole is used for handling matrix transpose and automorphism operations.

[0014] The aforementioned non-privacy and privacy inference scheduling method based on reconfigurable chips, wherein the inference step includes:

[0015] The in-core parallel step is executed. When the computational cost of the neural network inference task exceeds a preset value, the computational cost of the neural network inference task is evenly distributed to each ring-tensor core, and each ring-tensor core executes the neural network inference task in parallel.

[0016] The inter-core parallel step is performed when the neural network inference task requires the sequential execution of privacy inference task flows of multiple kernels. The multiple kernels to be executed sequentially are mapped to different ring-tensor cores. The mapped ring-tensor cores form a pipeline, and the neural network inference task is executed in parallel between the kernels.

[0017] The aforementioned non-privacy and privacy inference scheduling method based on reconfigurable chips, wherein the inference step includes:

[0018] Before the execution of the neural network inference task, the computation graph of the neural network inference task is obtained, the computation kernels that are executed in a pipelined manner in the computation graph are determined and merged, the fused computation graph is topologically sorted to form a sequence of kernel functions that satisfy data dependencies, and the sequence is divided into multiple kernel function groups;

[0019] Iterate through all possible ring-tensor core allocations, calculate the optimal intra-core / inter-core parallel strategy for each allocation, and generate a configuration entry. This entry records the kernel function to be executed for each logical ring-tensor core, the communication relationship, and the expected execution time.

[0020] The aforementioned non-privacy and privacy inference scheduling method based on reconfigurable chips includes:

[0021] The resource allocation step involves checking the time requirement and remaining workload of each neural network inference task in the task queue to obtain an urgency score for each task. Based on this urgency score, the remaining idle ring-tensor cores are allocated proportionally to each neural network inference task. Similar logic is used to allocate HBM bandwidth and shared cache size.

[0022] The mapping and execution steps involve logically mapping the physical ring-tensor core region assigned to the task according to the configuration table entries, configuring the reconfigurable chip on-chip network routing, and issuing instructions to execute the next kernel function group.

[0023] like Figure 9 As shown, this invention also proposes a non-privacy and privacy-preservation inference scheduling method based on a reconfigurable chip, including:

[0024] The chip building module constructs a reconfigurable chip including a shared cache, a high-bandwidth memory controller, and multiple ring-tensor cores; each ring-tensor core contains a processing unit array, a data layout converter, and a local cache; the processing unit array is a two-dimensional array composed of multiple processing units (PEs), each PE is composed of multiple slices, and each slice contains a multiplier and an adder.

[0025] The judgment module determines whether the privacy requirement of the neural network inference task is privacy inference. If it is, the first scheduling module is executed; otherwise, the second scheduling module is executed.

[0026] The first scheduling module analyzes the arithmetic logic unit required for the privacy inference. The data layout converter combines and reconstructs the multiplier and adder to form the arithmetic logic unit, and reconstructs the interconnection network of the processing unit array into a butterfly network or SIMD data path to obtain the reconstructed chip, which then executes the inference module.

[0027] The second scheduling module, the data layout converter sets the slice to tensor mode, all slices work independently and in parallel, each slice performs multiply-accumulate operations, data is transmitted between processing units (PE) through horizontal and vertical links to form a pulsating data stream, the processing unit array forms a pulsating array, obtains the reconstructed chip, and executes the inference module;

[0028] The inference module uses the reconstructed chip to perform the neural network inference task and obtain the inference result.

[0029] The aforementioned non-privacy and privacy-preserving inference scheduling method based on reconfigurable chips includes a data layout converter consisting of a vectorized exchange network for performing data rearrangement operations; the exchange network is configured as multiple sub-networks for handling tensor transpose; and the exchange network as a whole is used for handling matrix transpose and automorphism operations.

[0030] This reasoning module includes:

[0031] The in-core parallel module executes the neural network inference task. When the computational load of the neural network inference task exceeds a preset value, the computational load of the neural network inference task is evenly distributed to each ring-tensor core, and each ring-tensor core executes the neural network inference task in parallel.

[0032] The inter-core parallel module executes the privacy inference task flow of multiple kernels in sequence when the neural network inference task needs to execute multiple kernels in sequence. The kernels that need to be executed in sequence are mapped to different ring-tensor kernels. The mapped ring-tensor kernels form a pipeline and execute the neural network inference task in parallel between kernels.

[0033] This reasoning module includes:

[0034] Before the execution of the neural network inference task, the computation graph of the neural network inference task is obtained, the computation kernels that are executed in a pipelined manner in the computation graph are determined and merged, the fused computation graph is topologically sorted to form a sequence of kernel functions that satisfy data dependencies, and the sequence is divided into multiple kernel function groups;

[0035] Iterate through all possible ring-tensor core allocations, calculate the optimal intra-core / inter-core parallel strategy for each allocation, and generate a configuration entry. This entry records the kernel function to be executed for each logical ring-tensor core, the communication relationship, and the expected execution time.

[0036] The aforementioned non-privacy and privacy inference scheduling device based on a reconfigurable chip further includes:

[0037] The resource allocation module checks the time requirement and remaining workload of each neural network inference task in the task queue to obtain an urgency score for each neural network inference task; based on this urgency score, it allocates the remaining idle ring-tensor cores to each neural network inference task proportionally. Similar logic is also used to allocate HBM bandwidth and shared cache size;

[0038] The mapping and execution module assigns the physical ring-tensor core region to the task, performs logical mapping based on the configuration table entries, configures the reconfigurable on-chip network routing, and issues instructions to execute the next kernel function group.

[0039] The present invention also proposes an electronic device, including the aforementioned non-privacy and privacy inference scheduling device based on a reconfigurable chip, wherein the electronic device may be connected to an information display device, which is used to display the inference result with user-set display parameters, attributes or through an artificial intelligence model.

[0040] The present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the non-privacy and privacy inference scheduling method based on the reconfigurable chip.

[0041] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the non-privacy and privacy inference scheduling method based on the reconfigurable chip.

[0042] As can be seen from the above solutions, the advantages of the present invention are:

[0043] 1. Significantly reduced costs: Through a unified architecture design, it avoids developing dedicated chips for NPI and PI separately, saving huge NRE costs.

[0044] 2. Improve hardware resource utilization: The reconfigurable design allows hardware resources to be dynamically allocated according to the real-time NPI / PI task load, solving the problem of idle resources in dedicated accelerator solutions when the load is uneven, and improving the overall return on investment of the data center.

[0045] 3. Combining high performance and high flexibility: It provides near-ASIC performance while possessing the flexibility of general-purpose hardware. Experimental results show that when handling mixed workloads, the performance of this invention far exceeds that of a heterogeneous acceleration platform composed of dedicated accelerators TPU and SHARP (average 1.44 times) and GPGPU (average 20.62 times).

[0046] 4. High-efficiency resource scheduling: The proposed scheduling method can effectively manage and utilize the parallel capabilities of the unified architecture. Compared with the traditional scheduling algorithms PREMA and Planaria, the average latency is significantly reduced (by 3.27 times and 1.79 times respectively in the experiment). Attached Figure Description

[0047] Figure 1 This is a diagram of the RTPU hardware architecture.

[0048] Figure 2 This is a schematic diagram of a reconfigurable processing unit array;

[0049] Figure 3 This is a structural diagram of a PE (polyethylene) device.

[0050] Figure 4 This is a diagram illustrating how to configure PE in tensor mode.

[0051] Figure 5 This is a diagram illustrating the implementation of a reconfigurable data layout converter (DLT).

[0052] Figure 6 This is a schematic diagram illustrating the execution of in-kernel parallelism.

[0053] Figure 7 This is a schematic diagram illustrating inter-core parallelism.

[0054] Figure 8 This is a flowchart of the method of the present invention;

[0055] Figure 9 This is a block diagram of the device of the present invention;

[0056] Figure 10 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0057] Figure 11 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0058] Figure 12 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0059] Figure label:

[0060] A - First electronic device;

[0061] B-Non-privacy and privacy-based inference scheduling devices based on reconfigurable chips;

[0062] C-Data acquisition equipment;

[0063] D-Information display device;

[0064] 1000 - Second electronic device;

[0065] Ⅰ-Computational Unit;

[0066] II-ROM;

[0067] III-RAM;

[0068] N-bus;

[0069] V-Interface;

[0070] VI - Input Unit;

[0071] VII - Output Unit;

[0072] VIII - Storage medium;

[0073] IX - Communication Unit. Detailed Implementation

[0074] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0075] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0077] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0078] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0079] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0080] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0081] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0082] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0083] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0084] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0085] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0088] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] While researching how to efficiently handle inference tasks with hybrid privacy requirements in data centers, the inventors discovered that existing technologies could not balance performance, cost, and flexibility. The fundamental technical challenge lies in the fundamental differences between NPI and PI in their underlying algebraic structures. NPI relies on tensor operations (such as matrix multiplication), while PI (based on FHE) relies on polynomial ring algebra operations. The data bit width, computation rules, and data flow patterns of these two operations are completely different, making it extremely difficult to design a unified architecture with efficient reusable hardware. This invention does not limit itself to these superficial differences but delves into the inherent commonalities of the two computational models, achieving the following key analytical results:

[0090] Analysis Result 1 (Data Flow Similarity): Despite the different operations, some key computational kernels and data layout transformation kernels in NPI and PI share similarities at the data flow level. For example, the basis conversion (BConv) operation in PI is essentially a matrix-matrix modular multiplication, which is highly similar to the General Matrix Multiplication (GEMM) in NPI in terms of data flow, exhibiting commonalities. This enables the interconnection network between shared processing units (PEs).

[0091] Analysis Result 2 (Computational Unit Decomposability): The high-bit-width (e.g., 32-bit or 64-bit) modular arithmetic units required by PI can be decomposed into multiple low-bit-width (e.g., 8-bit) integer arithmetic units constructed through specific combinational logic. These low-bit-width integer arithmetic units are precisely what NPI tensor operations require. This reveals the possibility of reconfiguration and reuse at the most basic arithmetic logic unit (ALU) level.

[0092] Analysis Result 3 (Resource Distribution Similarity): Both NPI and PI accelerators require significant on-chip memory to cache weights, feature maps, or intermediate polynomial data. On-chip memory dominates the chip area, providing a foundation for silicon resource reuse across PI and NPI tasks.

[0093] Based on the above research findings, the inventors propose a new technical solution: designing a reconfigurable unified architecture. This architecture is no longer a simple "physical splicing" of two dedicated accelerators, but rather, through refined reconfigurable design at the microarchitecture level (processing units (PEs) and interconnect networks), it enables dynamic switching between "tensor mode" and "polynomial ring mode," thereby maximizing the sharing of computing, interconnect, and on-chip storage resources. Simultaneously, a scheduling algorithm capable of recognizing hybrid task characteristics and hardware reconfigurability is designed to complement this architecture, achieving global optimization of system resources.

[0094] To achieve the above-mentioned technical effects, based on the above analysis results, the present invention proposes the following key technical points:

[0095] Key Point 1: A reconfigurable Ring & Tensor Core (RTC) design, particularly its internal multimodal processing element (PE). This PE consists of an array of multiple basic computational "slices," each containing low-bit-width (e.g., 8-bit) integer multipliers and adders. Through configurable data paths and control logic, these "slices" can be dynamically organized and connected to operate in different modes.

[0096] 1. Tensor Mode (NPI): "Slices" work independently, performing multiple parallel low-bit-width multiply-accumulate (MAC) operations. The entire PE forms an efficient sub-systolic array to accelerate GEMM.

[0097] 2. Polynomial Ring Mode (PI): "Slices" are aggregated into high-bit-width modular arithmetic units (such as 32-bit modular multipliers and modular adders) to support BConv, Number Theoretic Transform (NTT), element-wise (EWE) operations in FHE.

[0098] Technical effects: It achieves native support for tensor operations and polynomial ring operations within a single PE, and achieves high reuse of hardware resources at the computing unit level (experiments show that the silicon reuse rate exceeds 97%), fundamentally solving the problem of low hardware utilization.

[0099] Key Point 2: A multi-core processor architecture with multiple RTCs and a hybrid parallelization mechanism. Multiple RTCs are organized into a multi-core cluster and interconnected via a network-on-chip (NoC). This architecture supports two parallelization methods:

[0100] 1. Intra-Kernel Parallelization: Split a single large computational task (such as a large GEMM or NTT) and execute it in parallel on multiple RTCs.

[0101] 2. Inter-Kernel Parallelization: Maps multiple consecutive computational tasks with data dependencies to different RTCs to form fine-grained pipeline parallelism.

[0102] Technical effects: Significantly improves throughput and resource utilization for handling mixed workloads. In particular, the inter-core parallelism mechanism effectively alleviates hardware idleness caused by insufficient parallelism in a single computing task, further improving system efficiency.

[0103] Key Point 3: A two-stage scheduling method based on kernel groups. 1. Offline Optimization Stage: The input computation graph (DAG) is analyzed, kernel functions are fused, sorted, and divided into fixed-size "kernel groups." Multiple execution configuration versions are pre-calculated and generated for different resource allocations (number of RTCs), and inefficient versions are pruned. 2. Online Scheduling Stage: When a task arrives or completes, the scheduler dynamically reallocates resources such as RTCs, memory access bandwidth, and on-chip storage for all tasks based on task priority (the ratio of relaxation time to remaining computation) and the pre-generated configuration table, and maps these resources to physical RTCs.

[0104] Technical effects: It enables flexible and efficient allocation of reconfigurable hardware resources, makes full use of inter-core parallel capabilities, and ensures high resource utilization and QoS in complex scenarios with multi-tenancy and mixed privacy requirements.

[0105] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0106] 1. RTPU Hardware Architecture

[0107] like Figure 1 As shown, the RTPU architecture is layered and mainly consists of multiple ring-tensor core (RTC) clusters, a shared buffer, and a high-bandwidth memory (HBM) controller.

[0108] 1.1. Internal Structure of the Ring-Tensor Core (RTC)

[0109] like Figure 1 As shown in section (c), each RTC contains a Process Element Array (PEA), a Data Layout Transformer (DLT), a Local Buffer, and a controller, among other things.

[0110] Step 1: Implementation of the Reconfigurable Processing Cell Array (PEA)

[0111] like Figure 2 As shown, PEA is a two-dimensional array, and its core is the design of multimodal PE.

[0112] Composition of PE: such as Figure 3 As shown, each PE consists of 4x12 basic "slices". Figure 3 As shown in Figure ①, each slice contains an 8-bit multiplier and two 16-bit adders. That is, PEA includes multiple PEs, and each PE includes multiple slices.

[0113] Mode switching:

[0114] The scheduler switches modes based on the privacy requirements of the task. When switching to tensor mode (for NPI): [Example follows] Figure 4 As shown in ①, the control logic configures PE in tensor mode. In this mode, the 48 slices work independently and in parallel, with each slice performing an 8-bit multiply-accumulate operation. The entire PEA forms a (4×n_row)×(12×n_col) systolic array, efficiently performing GEMM operations, such as... Figure 2 As indicated, n_row and n_col represent the number of rows and columns of PEs in the PEA, respectively. Data is transmitted between PEs via horizontal and vertical links, forming a pulsating data stream.

[0115] Switch to polynomial ring mode (for PI): e.g. Figure 4 As shown, the control logic configures the PE in loop mode. Figure 4 As shown in ②~⑥, the internal "slices" are recombined. For example, by combining 16 8-bit multipliers and 24 16-bit adders, a 32-bit integer multiplier is formed. Then, using the Barrett reduction algorithm, a 32-bit modular multiplier is constructed. Similarly, modular adders, modular subtractors, and other arithmetic logic units are constructed. Finally, the entire PE can be configured to perform NTT butterfly operations (mode 2). Figure 4 ③ / ④), Modular multiplication and accumulation of BConv ( Figure 4 Mode ②) or EWE's modular multiplication / modal addition ( Figure 4 Mode ⑤ / ⑥). At this time, the interconnection network of PEA is also reconfigured into a butterfly network (for NTT) or a SIMD data path (for EWE).

[0116] The recombination process includes: by setting a multiplexer, the originally scattered "8-bit multipliers and 16-bit adders inside multiple slices" are connected in series into high-order integer adders and multipliers, and further into modular multipliers, modular adders, etc.

[0117] When the neural network inference service is non-privacy inference, no combination is required. Each slice performs 8-bit integer multiplication and 16-bit integer addition operations (i.e., 8-bit multiply-add accumulation operation) separately.

[0118] When the neural network inference service is privacy-preserving, it should be combined into a 32-bit modular multiplier and modular adder. In this case, PE corresponds to Figure 3 ⑥ in the middle.

[0119] Step 2: Implementation of the reconfigurable data layout converter (DLT).

[0120] like Figure 5 As shown, DLT is used to perform data rearrangement operations such as tensor dimension permutation (Perm), matrix transpose (Trans), and automorphism (Auto). It consists of a vectorized exchange network. Figure 5 The basic unit of the switching network is an 8-bit swapper, which can perform the exchange and temporary storage of two input data streams under the control of the control signal. Therefore, by setting the control signal, the data layout can be changed. Specifically, the tensor dimension permutation (Perm) is 8-bit and can be converted into a series of 8-bit matrix transpose operations. PI, on the other hand, requires 32-bit matrix transpose operations, which are achieved by combining four 8-bit subnetworks to obtain a 32-bit switching network for transformation.

[0121] Bit-width adaptation: This network can be configured as four independent 8-bit subnetworks to handle 8-bit tensor transpose in NPI; or it can be combined into a unified 32-bit network to handle 32-bit matrix transpose and automorphism operations in PI. In this way, DLT is also fully reused by both NPI and PI tasks.

[0122] 1.2. Multi-core parallelization

[0123] Step 3: Perform intra-kernel parallel execution

[0124] like Figure 6 As shown, when a computationally intensive task is received, such as a PI computation task with 20 limbs (components of a polynomial in RNS representation), and the system allocates 10 RTCs, the scheduler will distribute these 20 limbs evenly among the 10 RTCs, with each RTC processing 2 limbs, thus enabling parallel execution and reducing latency.

[0125] Step 4: Perform inter-core parallel processing

[0126] like Figure 7As shown, when processing a PI task flow that requires sequentially executing EWE-MM (Elementwise Modular Multiplication) -> Auto (Automorphism) -> NTT (Number Theoretic Transform) -> Trans (Transpose) -> NTT, the scheduler maps these five consecutive kernel functions to five (or more, depending on the in-kernel parallelism requirements) different RTCs. After the first RTC completes the EWE-MM operation for the first limb, it immediately passes the result to the second RTC to start the Auto operation, while the first RTC begins processing the EWE-MM for the second limb. In this way, multiple RTCs form a pipeline of depth 5, greatly improving throughput and resource utilization.

[0127] The computation kernel is a node in the DAG mentioned above. In this invention, both the computation kernel and the kernel function refer to "kernel".

[0128] 2. Two-phase scheduling method

[0129] Step 5: Offline Optimization

[0130] Before a task is executed, the scheduler processes the task's computation graph (DAG). In the DAG, nodes correspond to kernel functions, and edges represent data dependencies between kernel functions.

[0131] Kernel function fusion and sorting: Merge computational kernels that can be piped (such as a kernel computed in PEA and a kernel computed in DLT) into a larger kernel. Then, perform a topological sort on the graph to form a sequence of kernel functions that satisfy data dependencies.

[0132] Grouping: Divide the sequence into multiple kernel function groups of size n_grp (e.g., 6).

[0133] Generate multi-version configurations: Iterate through all possible RTC allocation numbers (e.g., from 1 RTC to 64 RTCs), calculate the optimal intra-core / inter-core parallelism strategy for each allocation number to balance pipeline stages, and generate a configuration entry. This entry records the kernel functions to be executed for each logical RTC, communication relationships, and estimated execution time.

[0134] The process of calculating the optimal intra-kernel / inter-kernel parallel strategy involves: a computation graph is divided into multiple kernel function groups, and multiple kernel functions within each group are executed in a pipelined manner. The optimal approach is to allocate resources on demand, ensuring that the execution times of each kernel function in the pipeline are as close as possible (because the pipeline throughput depends on the longest stage). For example, in the kernel function group "A->B->C->D", if A has the largest computational load, then the most RTC needs to be allocated.

[0135] Step Six: Online Scheduling

[0136] During system operation:

[0137] Trigger rescheduling: When a new task arrives or an old task completes, resource rescheduling is triggered.

[0138] Resource allocation: The scheduler checks the QoS requirements (deadline) and remaining workload of each task in the task queue and calculates an "urgency score." First, it meets the minimum RTC requirements of each task, then allocates the remaining idle RTC proportionally based on the "urgency score." Similar logic is used to allocate HBM bandwidth and shared buffer size.

[0139] Mapping and Execution: The physical RTC region assigned to the task is logically mapped according to the configuration table, NoC routing is configured, and then instructions are issued to execute the next kernel function group. After execution, the status in the task queue (such as remaining time and the ID of the next group) is updated.

[0140] Through the above implementation methods, the present invention can efficiently and flexibly execute hybrid NPI and PI tasks with a unified and reconfigurable hardware architecture, achieving the comprehensive effects of reducing costs, improving resource utilization, and ensuring high performance.

[0141] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0142] like Figure 9 As shown, this invention also proposes a non-privacy and privacy-preservation inference scheduling method based on a reconfigurable chip, including:

[0143] The chip building module constructs a reconfigurable chip including a shared cache, a high-bandwidth memory controller, and multiple ring-tensor cores; each ring-tensor core contains a processing unit array, a data layout converter, and a local cache; the processing unit array is a two-dimensional array composed of multiple processing units (PEs), each PE is composed of multiple slices, and each slice contains a multiplier and an adder.

[0144] The judgment module determines whether the privacy requirement of the neural network inference task is privacy inference. If it is, the first scheduling module is executed; otherwise, the second scheduling module is executed.

[0145] The first scheduling module analyzes the arithmetic logic unit required for the privacy inference. The data layout converter combines and reconstructs the multiplier and adder to form the arithmetic logic unit, and reconstructs the interconnection network of the processing unit array into a butterfly network or SIMD data path to obtain the reconstructed chip, which then executes the inference module.

[0146] The second scheduling module, the data layout converter sets the slice to tensor mode, all slices work independently and in parallel, each slice performs multiply-accumulate operations, data is transmitted between processing units (PE) through horizontal and vertical links to form a pulsating data stream, the processing unit array forms a pulsating array, obtains the reconstructed chip, and executes the inference module;

[0147] The inference module uses the reconstructed chip to perform the neural network inference task and obtain the inference result.

[0148] The aforementioned non-privacy and privacy-preserving inference scheduling method based on reconfigurable chips includes a data layout converter consisting of a vectorized exchange network for performing data rearrangement operations; the exchange network is configured as multiple sub-networks for handling tensor transpose; and the exchange network as a whole is used for handling matrix transpose and automorphism operations.

[0149] This reasoning module includes:

[0150] The in-core parallel module executes the neural network inference task. When the computational load of the neural network inference task exceeds a preset value, the computational load of the neural network inference task is evenly distributed to each ring-tensor core, and each ring-tensor core executes the neural network inference task in parallel.

[0151] The inter-core parallel module executes the privacy inference task flow of multiple kernels in sequence when the neural network inference task needs to execute multiple kernels in sequence. The kernels that need to be executed in sequence are mapped to different ring-tensor kernels. The mapped ring-tensor kernels form a pipeline and execute the neural network inference task in parallel between kernels.

[0152] This reasoning module includes:

[0153] Before the execution of the neural network inference task, the computation graph of the neural network inference task is obtained, the computation kernels that are executed in a pipelined manner in the computation graph are determined and merged, the fused computation graph is topologically sorted to form a sequence of kernel functions that satisfy data dependencies, and the sequence is divided into multiple kernel function groups;

[0154] Iterate through all possible ring-tensor core allocations, calculate the optimal intra-core / inter-core parallel strategy for each allocation, and generate a configuration entry. This entry records the kernel function to be executed for each logical ring-tensor core, the communication relationship, and the expected execution time.

[0155] The aforementioned non-privacy and privacy inference scheduling device based on a reconfigurable chip further includes:

[0156] The resource allocation module checks the time requirement and remaining workload of each neural network inference task in the task queue to obtain an urgency score for each neural network inference task; based on this urgency score, it allocates the remaining idle ring-tensor cores to each neural network inference task proportionally. Similar logic is also used to allocate HBM bandwidth and shared cache size;

[0157] The mapping and execution module assigns the physical ring-tensor core region to the task, performs logical mapping based on the configuration table entries, configures the reconfigurable on-chip network routing, and issues instructions to execute the next kernel function group.

[0158] like Figure 10 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, which includes a non-privacy and privacy inference scheduling device B based on a reconfigurable chip.

[0159] like Figure 11 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through wired or wireless information transmission schemes. The data acquisition device C is used to collect neural network tasks, models, input data, etc., such as the face recognition task, face recognition model, and face image described in the embodiments of the present invention. The information display device D is used to display the reasoning results obtained by the analysis of the present invention.

[0160] Information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with specified key information, such as category results, based on user-defined display parameters and / or attributes. Display parameters could be, for example, the data range to be displayed, and display attributes could be, for example, the font, color, and whether scrolling is enabled. This allows the user to access this information more quickly without having to access secondary pages or scroll through pages, saving user effort. Alternatively, this information display mechanism can be an artificial intelligence (AI) display model, which can learn the user's key information interests based on previous usage habits, such as viewing time, number of clicks, and number of edits, and then automatically present the user with rich and necessary key information.

[0161] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the non-privacy and privacy inference scheduling methods based on reconfigurable chips provided by the above methods.

[0162] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that executes the aforementioned non-privacy and privacy-based inference scheduling method based on a reconfigurable chip. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0163] Figure 12 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0164] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0165] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0166] The computing unit I can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S5. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0167] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A non-privacy and privacy-preservation inference scheduling method based on a reconfigurable chip, characterized in that, include: The chip fabrication steps involve constructing a reconfigurable chip including a shared cache, a high-bandwidth memory controller, and multiple ring-tensor cores; each ring-tensor core contains a processing unit array, a data layout converter, and a local cache; the processing unit array is a two-dimensional array composed of multiple processing units (PEs), each PE is composed of multiple slices, and each slice contains a multiplier and an adder. The decision step is to determine whether the privacy requirement of the neural network inference task is privacy inference. If it is, the first scheduling step is executed; otherwise, the second scheduling step is executed. The first scheduling step involves analyzing the arithmetic logic unit required for the privacy inference. The data layout converter combines and reconstructs the multiplier and adder to form the arithmetic logic unit, and reconstructs the interconnection network of the processing unit array into a butterfly network or SIMD data path to obtain the reconstructed chip and execute the inference step. In the second scheduling step, the data layout converter sets the slice to tensor mode, all slices work independently and in parallel, each slice performs multiply-accumulate operations, data is transmitted between processing units (PE) through horizontal and vertical links to form a pulsating data stream, the processing unit array forms a pulsating array, a reconstructed chip is obtained, and the inference step is executed. The reasoning step involves using the reconstructed chip to perform the neural network reasoning task and obtain the reasoning result.

2. The non-privacy and privacy inference scheduling method based on reconfigurable chips as described in claim 1, characterized in that, The data layout converter consists of a vectorized exchange network for performing data rearrangement operations; the exchange network is configured as multiple subnetworks for handling tensor transpose; and the exchange network as a whole is used to handle matrix transpose and automorphism operations.

3. The non-privacy and privacy inference scheduling method based on reconfigurable chips as described in claim 1, characterized in that, The reasoning steps include: The in-core parallel step is executed. When the computational cost of the neural network inference task exceeds a preset value, the computational cost of the neural network inference task is evenly distributed to each ring-tensor core, and each ring-tensor core executes the neural network inference task in parallel. The inter-core parallel step is performed when the neural network inference task requires the sequential execution of privacy inference task flows of multiple kernels. The multiple kernels to be executed sequentially are mapped to different ring-tensor cores. The mapped ring-tensor cores form a pipeline, and the neural network inference task is executed in parallel between the kernels.

4. The non-privacy and privacy inference scheduling method based on reconfigurable chips as described in claim 3, characterized in that, The reasoning steps include: Before the execution of the neural network inference task, the computation graph of the neural network inference task is obtained, the computation kernels that are executed in a pipelined manner in the computation graph are determined and merged, the fused computation graph is topologically sorted to form a sequence of kernel functions that satisfy data dependencies, and the sequence is divided into multiple kernel function groups; Iterate through all possible ring-tensor core allocations, calculate the optimal intra-core / inter-core parallel strategy for each allocation, and generate a configuration entry. This entry records the kernel function to be executed for each logical ring-tensor core, the communication relationship, and the expected execution time.

5. The non-privacy and privacy inference scheduling method based on a reconfigurable chip as described in claim 3, characterized in that, include: The resource allocation steps involve checking the time requirements and remaining workload of each neural network inference task in the task queue to obtain an urgency score for each neural network inference task; then, based on this urgency score, the remaining idle ring-tensor cores are allocated proportionally to each neural network inference task; similar logic is also used to allocate HBM bandwidth and shared cache size. The mapping and execution steps involve logically mapping the physical ring-tensor core region assigned to the task according to the configuration table entries, configuring the reconfigurable chip on-chip network routing, and issuing instructions to execute the next kernel function group.

6. A non-privacy and privacy inference scheduling device based on a reconfigurable chip, characterized in that, include: The chip building module constructs a reconfigurable chip including a shared cache, a high-bandwidth memory controller, and multiple ring-tensor cores; each ring-tensor core contains a processing unit array, a data layout converter, and a local cache; the processing unit array is a two-dimensional array composed of multiple processing units (PEs), each PE is composed of multiple slices, and each slice contains a multiplier and an adder. The judgment module determines whether the privacy requirement of the neural network inference task is privacy inference. If it is, the first scheduling module is executed; otherwise, the second scheduling module is executed. The first scheduling module analyzes the arithmetic logic unit required for the privacy inference. The data layout converter combines and reconstructs the multiplier and adder to form the arithmetic logic unit, and reconstructs the interconnection network of the processing unit array into a butterfly network or SIMD data path to obtain the reconstructed chip, which then executes the inference module. The second scheduling module, the data layout converter sets the slice to tensor mode, all slices work independently and in parallel, each slice performs multiply-accumulate operations, data is transmitted between processing units (PE) through horizontal and vertical links to form a pulsating data stream, the processing unit array forms a pulsating array, obtains the reconstructed chip, and executes the inference module; The inference module uses the reconstructed chip to perform the neural network inference task and obtain the inference result.

7. The non-privacy and privacy inference scheduling device based on a reconfigurable chip as described in claim 6, characterized in that, The data layout converter consists of a vectorized exchange network for performing data rearrangement operations; the exchange network is configured as multiple subnetworks for handling tensor transpose. This switching network, as a whole, is used to handle matrix transpose and automorphism operations; This reasoning module includes: The in-core parallel module executes the neural network inference task. When the computational load of the neural network inference task exceeds a preset value, the computational load of the neural network inference task is evenly distributed to each ring-tensor core, and each ring-tensor core executes the neural network inference task in parallel. The inter-core parallel module executes the privacy inference task flow of multiple kernels in sequence when the neural network inference task needs to execute multiple kernels in sequence. The kernels that need to be executed in sequence are mapped to different ring-tensor kernels. The mapped ring-tensor kernels form a pipeline and execute the neural network inference task in parallel between kernels. This reasoning module includes: Before the execution of the neural network inference task, the computation graph of the neural network inference task is obtained, the computation kernels that are executed in a pipelined manner in the computation graph are determined and merged, the fused computation graph is topologically sorted to form a sequence of kernel functions that satisfy data dependencies, and the sequence is divided into multiple kernel function groups; Iterate through all possible ring-tensor core allocations, calculate the optimal intra-core / inter-core parallel strategy for each allocation, and generate a configuration entry. This entry records the kernel function to be executed for each logical ring-tensor core, the communication relationship, and the expected execution time.

8. An electronic device, characterized in that, Includes a non-privacy and privacy inference scheduling device based on a reconfigurable chip as described in claim 6 or 7, wherein the electronic device is connected to an information display device, which is used to display the inference result with user-set display parameters, attributes or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the non-privacy and privacy inference scheduling method based on a reconfigurable chip as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the non-privacy and privacy inference scheduling method based on a reconfigurable chip as described in any of claims 1-5.

Images