Multi-core grain computing resource scheduling method and device for neural network

Abstract

This invention provides a method and apparatus for multi-core computing resource scheduling in neural networks, relating to the field of computer technology. The method includes: dividing all neural network layers of the neural network to be inferred into at least one segment; merging adjacent neural network layers within each segment based on cluster merging relationships to obtain at least one cluster corresponding to each segment; iteratively adjusting all cores in the Multi-Core Module (MCM) based on the latency of each cluster to obtain the target number of cores for each cluster under load balancing; determining the partitioning method for each neural network layer based on its depth within the neural network to be inferred; and scheduling computing resources corresponding to the MCM for the neural network to be inferred based on all segments, all clusters corresponding to each segment, the target number of cores for each cluster, and the partitioning method for each neural network layer. This invention can deploy balanced clusters onto the MCM, simplifying hardware resource allocation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for scheduling multi-core computing resources for neural networks. Background Technology

[0002] To meet the ever-increasing demand for neural network computing, neural network accelerators based on the Multi-Chip Module (MCM) architecture have become a mainstream solution. The MCM architecture integrates multiple smaller chiplets onto a package substrate and interconnects them via Network-on-Chip (NoP). Compared to large monolithic chips, MCMs can overcome the limitations of photolithography size to integrate more transistors, while simultaneously reducing design and manufacturing costs by improving yield. However, as the number of chiplets increases, directly deploying neural network layers on the MCM suffers from high NoP communication overhead and insufficient resource utilization.

[0003] In existing technologies, scheduling methods such as fully sequential execution, full pipelined execution, and segmented pipelined execution are generally used to address the above problems. However, while fully sequential execution performs well with a small number of chips, its performance degrades as the chip integration scale increases. Full pipelined execution is suitable for shallow networks, but it is difficult to achieve pipeline matching for deep networks. Segmented pipelined execution divides deep networks into multiple shallower segments, which are executed sequentially. Each layer within a segment runs in a pipelined manner across the entire package. However, in segmented pipelined execution, too many segments can lead to each layer being distributed across too many chips, resulting in NoP overhead and insufficient utilization. Too few segments can lead to an overly deep pipeline, generating a large number of preheating or cooling bubbles, making it difficult to achieve pipeline matching.

[0004] Therefore, how to solve the load imbalance and communication overhead caused by deploying each layer separately in the existing technology is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a method and apparatus for scheduling multi-core computing resources in neural networks, which solves the problems of uneven load and high communication overhead caused by deploying each layer separately in the prior art.

[0006] This invention provides a multi-core computing resource scheduling method for neural networks, comprising the following steps.

[0007] Divide all neural network layers of the neural network to be inferred into at least one segment, each segment comprising multiple consecutive neural network layers.

[0008] Based on the cluster merging association, adjacent neural network layers within each segment are merged to obtain at least one cluster corresponding to each segment; the cluster merging association is used to characterize the preset layer merging strategy corresponding to different numbers of clusters.

[0009] Based on the latency of each cluster, all chips in the multi-chip module (MCM) are iteratively adjusted to obtain the target number of chips for each cluster under load balancing conditions.

[0010] Based on the depth of each neural network layer in the neural network to be inferred, the partitioning method corresponding to each neural network layer is determined; the partitioning method is used to characterize the storage method of weight parameters and input features.

[0011] Based on all segments, all clusters corresponding to each segment, the number of target cores corresponding to each cluster, and the partitioning method corresponding to each neural network layer, the computing resources corresponding to the MCM are scheduled for the neural network to be inferred.

[0012] According to the multi-core computing resource scheduling method for neural networks provided by the present invention, the cluster merging association relationship is determined based on the following steps: Based on the input and output dimensions of each neural network layer, the parallelizable dimension ratio of each neural network layer is determined; the parallelizable dimension ratio is used to characterize the parallelism of the input and output dimensions of each neural network layer. Starting with each neural network layer as an independent cluster, adjacent clusters are iteratively merged based on the ratio of parallelizable dimensions corresponding to each cluster until all clusters are merged into one cluster, thus obtaining a preset layer merging strategy corresponding to different numbers of clusters.

[0013] According to the multi-core computing resource scheduling method for neural networks provided by the present invention, the iterative merging of adjacent clusters based on the ratio of parallelizable dimensions corresponding to each cluster includes: S11. Based on the ratio of parallelizable dimensions of each cluster and its neighboring clusters, determine the parallel similarity between each cluster and its neighboring clusters; S12. Merge the two clusters with the lowest parallel similarity among all parallel similarities into a new cluster; S13. When the number of the new cluster and the number of other unmerged clusters is equal to 1, stop the iteration and obtain the preset layer merging strategy corresponding to different numbers of clusters. S14. If the number of the new cluster and the other unmerged clusters is greater than 1, the new cluster and the other unmerged clusters are determined as the new clusters. Steps S11 to S12 are repeated until the iteration stops when the number of the new cluster and the other unmerged clusters is equal to 1.

[0014] According to the multi-chip computation resource scheduling method for neural networks provided by the present invention, the step of iteratively adjusting all chips in a multi-chip module (MCM) based on the latency duration corresponding to each cluster to obtain the target number of chips corresponding to each cluster in a load-balanced state includes: S21. Based on the computational cost corresponding to each cluster, all cores in the MCM are initially allocated to each cluster; S22. Based on the delay duration of each cluster, determine the first cluster with the longest delay duration, the second cluster with the shortest delay duration, and the initial overall delay duration. S23. Distribute one core particle corresponding to the second cluster to the first cluster; S24. Based on the adjusted delay duration of each cluster, determine the current overall delay duration of all clusters. S25. If the current overall delay duration is greater than or equal to the initial overall delay duration, stop the iteration and obtain the target number of core particles corresponding to each cluster in the load-balanced state. S26. If the current overall delay duration is less than the initial overall delay duration, the adjusted delay duration corresponding to each cluster is determined as the new delay duration corresponding to each cluster, and the current overall delay duration is determined as the new initial overall delay duration. Steps S22 to S24 are repeated until the current overall delay duration is greater than or equal to the initial overall delay duration, at which point the iteration stops.

[0015] According to the multi-core computational resource scheduling method for neural networks provided by the present invention, the initial allocation of all cores in the MCM to each of the clusters based on the computational load corresponding to each cluster includes: Based on the computational amount of each cluster, determine the computational load ratio of each cluster. Based on the calculated load ratio, all cores in the MCM are proportionally allocated to each of the clusters.

[0016] According to the multi-core computing resource scheduling method for neural networks provided by the present invention, the step of determining the current overall latency of all clusters based on the adjusted latency of each cluster includes: Based on the adjusted delay duration corresponding to each cluster, the adjusted maximum delay duration is determined. Based on the total number of samples, the adjusted longest delay duration, and the number of clusters corresponding to all clusters, determine the current overall delay duration for all clusters.

[0017] According to the multi-core computing resource scheduling method for neural networks provided by the present invention, determining the adjusted delay duration corresponding to each of the clusters includes: The preparation time, computation time, and communication time of each neural network layer in each cluster are determined respectively for the accurate stage, the computation stage, and the communication stage. The layer execution time is determined based on the computation time and communication time corresponding to each neural network layer. Based on the preparation time and layer execution time corresponding to each neural network layer, the adjusted layer delay time corresponding to each neural network layer is determined. Based on the layer delay duration corresponding to each neural network layer in each cluster, the adjusted delay duration corresponding to each cluster is determined.

[0018] According to the multi-core computing resource scheduling method for neural networks provided by the present invention, the partitioning method includes an input-shared partitioning method or a weight-shared partitioning method, wherein: The input sharing partitioning method is used to characterize that each core corresponding to the neural network layer stores all input features and local weight parameters; each core obtains all weight parameters based on on-chip network communication interaction. The weight-sharing partitioning method is used to characterize that each core corresponding to the neural network layer stores all weight parameters and local input features; adjacent cores obtain the input features of the boundary overlapping region based on inter-layer communication interaction.

[0019] According to the multi-core computing resource scheduling method for neural networks provided by the present invention, when the partitioning method includes a weight-sharing partitioning method, each core stores local weight parameters during the non-computation phase; and during the preparation phase, each core interacts with other cores to obtain all weight parameters.

[0020] The present invention also provides a multi-core computing resource scheduling device for neural networks, comprising the following modules.

[0021] A segmentation module is used to divide all neural network layers of the neural network to be inferred into at least one segment, each segment comprising multiple consecutive neural network layers. The merging module is used to merge adjacent neural network layers within each segment based on the cluster merging association relationship to obtain at least one cluster corresponding to each segment; the cluster merging association relationship is used to characterize the preset layer merging strategy corresponding to different numbers of clusters; The iterative adjustment module is used to iteratively adjust all the chips in the multi-chip module (MCM) based on the delay duration of each cluster, so as to obtain the target number of chips for each cluster under the load balancing state. A partitioning module is used to determine the partitioning method corresponding to each neural network layer based on the depth of each neural network layer in the neural network to be inferred; the partitioning method is used to characterize the storage method of weight parameters and input features; The scheduling module is used to schedule the computing resources corresponding to the MCM for the neural network to be inferred based on all segments, all clusters corresponding to each segment, the number of target cores corresponding to each cluster, and the partitioning method corresponding to each neural network layer.

[0022] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-core computing resource scheduling method for neural networks as described above.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-core computing resource scheduling method for neural networks as described above.

[0024] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-core computing resource scheduling method for neural networks as described above.

[0025] The present invention provides a multi-chip computational resource scheduling method and apparatus for neural networks. This method divides all neural network layers of the neural network to be inferred into multiple segments, each segment comprising multiple consecutive neural network layers. Based on cluster merging relationships representing preset layer merging strategies corresponding to different cluster numbers, adjacent neural network layers within each segment are merged, dividing each segment into at least one cluster. Based on the latency of each cluster, all chips in the multi-chip module are iteratively adjusted until a load-balanced state is achieved, obtaining the target chip number for each cluster. Based on the depth of each neural network layer within the neural network to be inferred, the partitioning method for each neural network layer is determined to adjust the storage method of weight parameters and input features for each neural network layer's chips. Based on all segments, all clusters corresponding to each segment, the target chip number for each cluster, and the partitioning method for each neural network layer, the multi-chip module is configured, thereby deploying the neural network to be inferred onto the configured multi-chip module to perform inference tasks. In this invention, clusters are obtained through layer merging, and multi-layer deployment is performed jointly based on the cluster dimension, significantly improving hardware resource utilization compared to deploying each layer individually. Iterative adjustments to the core particles based on the cluster dimension create a network structure composed of load-balanced clusters, which can effectively achieve pipeline-level computing load balancing, reduce pipeline preheating or cooling bubbles, thereby improving system throughput. By adaptively selecting partitioning methods, it alleviates NoP bandwidth pressure and reduces communication overhead. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the multi-core computing resource scheduling method for neural networks provided in an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the structure of a multi-core computing resource scheduling device for neural networks provided in an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] As the number of cores increases, there are two problems with directly deploying neural network layers on the MCM.

[0032] (1) NoP communication overhead is large: Compared with on-chip interconnect, NoP links have lower bandwidth and energy efficiency due to the need for longer transmission distance and larger spacing. Studies have shown that when the number of chips reaches 32, the NoP communication delay exceeds the computation delay, which seriously affects the system performance.

[0033] (2) Insufficient resource utilization: Distributing neural network layers with fixed parallelizable dimensions to a large number of cores results in insufficient workload for each core. With 64 cores, the typical resource utilization is less than 40%.

[0034] To address the aforementioned issues, existing technologies have proposed scheduling methods such as fully sequential execution, fully pipelined methods, and segmented pipelined methods. However, these methods all deploy each neural network layer separately. Since each neural network layer has different shapes and computational requirements, perfectly matching hardware resources to different neural network layers is extremely difficult. This underutilized dimension leads to compromises between computation and communication, making it difficult to achieve optimal utilization, especially as hardware or software scales up.

[0035] Therefore, to address the problems of uneven load and high communication overhead caused by deploying each layer separately in the existing technology, this invention provides a multi-core computing resource scheduling method for neural networks. Figure 1 This is a flowchart illustrating the multi-core computing resource scheduling method for neural networks provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes steps 110 to 150.

[0036] Step 110: Divide all neural network layers of the neural network to be inferred into at least one segment, each segment comprising multiple consecutive neural network layers.

[0037] Specifically, the neural network to be inferred is a neural network with a known network structure. After determining the neural network to be inferred, all neural network layers in the neural network are divided into multiple consecutive segments. Adjacent segments can be executed sequentially. Each segment includes multiple consecutive neural network layers.

[0038] Optionally, the number of neural network layers included in each segment may be equal or unequal, and the embodiments of the present invention do not limit this.

[0039] Optionally, the neural network to be inferred can be ResNet-152, VGG16, AlexNet, or DarkNet19, etc. The embodiments of the present invention do not limit the type of neural network to be inferred.

[0040] Step 120: Based on the cluster merging association relationship, merge adjacent neural network layers within each segment to obtain at least one cluster corresponding to each segment; the cluster merging association relationship is used to characterize the preset layer merging strategy corresponding to different numbers of clusters.

[0041] Specifically, before executing step 120, the neural network layers within a segment can be iteratively merged using a dynamic programming algorithm to obtain preset layer merging strategies corresponding to different cluster numbers, i.e., cluster merging relationships. These cluster merging relationships can include determining which neural network layers to merge based on the preset layer merging strategy when the target number of clusters is determined. For example, if the neural network to be inferred includes 10 neural network layers, and the target number of clusters is 5, the corresponding preset layer merging strategy could be to merge layer 1 and layer 2 into the first cluster, layer 3 and layer 4 into the second cluster, layer 5 into the third cluster, layers 6, 7, and 8 into the fourth cluster, and layers 9 and 10 into the fifth cluster.

[0042] After establishing the cluster merging relationship, the corresponding preset layer merging strategy can be found from the cluster merging relationship by looking up the table according to the number of target clusters. Then, the layer merging scheme corresponding to the preset layer merging strategy is executed on the neural network layer in the segment, so that the neural network layer in the segment is at least one cluster.

[0043] Step 130: Based on the delay duration of each cluster, iteratively adjust all the chips in the multi-chip module (MCM) to obtain the target number of chips for each cluster under load balancing.

[0044] Specifically, after merging all clusters, based on the dimension of the clusters, the latency corresponding to the multi-layer neural network layers is jointly considered, and the hardware resource chips in the multi-chip module (MCM) are iteratively allocated until the overall latency of the segment no longer decreases, thus reaching a load-balanced state. At this point, the target number of chips allocated to each cluster under the load-balanced state can be determined.

[0045] Step 140: Based on the depth of each neural network layer in the neural network to be inferred, determine the partitioning method corresponding to each neural network layer; the partitioning method is used to characterize the storage method of weight parameters and input features.

[0046] Specifically, after allocating the optimal number of target cores to each cluster, the partitioning method for the neural network layers in each cluster is adaptively determined based on the depth of each neural network layer in the neural network to be inferred. Under different partitioning methods, the cores corresponding to each neural network layer store weight parameters and input features differently. Different partitioning combinations produce different communication volumes. In this embodiment of the invention, the communication volume corresponding to each cluster is optimized by determining the partitioning method corresponding to different neural network layers in each cluster.

[0047] It should be noted that the partitioning method includes either input shared partitioning or weighted shared partitioning, wherein: The input-shared partitioning method is used to characterize that each core corresponding to the neural network layer stores all input features and local weight parameters; each core obtains all weight parameters through on-chip network communication interaction. Specifically, if the partitioning method of the neural network layer is the input-shared partitioning method, when the neural network layer is calculated, all the input features required by the neural network layer are copied to all cores participating in the calculation. All weight parameters corresponding to the neural network layer are divided proportionally, and each core only stores local weight parameters. During the calculation phase, each core can communicate with other cores through the on-chip network (NoP) to interactively obtain the local weight parameters stored by other cores, thereby obtaining all weight parameters.

[0048] The weight-sharing partitioning method is used to characterize that each core of the neural network layer stores all weight parameters and local input features; adjacent cores obtain input features of the boundary overlapping region based on inter-layer communication. Specifically, if the partitioning method of the neural network layer is weight-sharing partitioning, when the neural network layer is calculated, all weight parameters corresponding to the neural network layer are copied to all cores participating in the calculation. The input features required for the calculation of the neural network layer are divided, and each core only stores local input features. Before the calculation, each core can communicate with adjacent cores through on-chip network to obtain local input features of the boundary overlapping region. Compared with the prior art of transmitting all input features, the amount of data transmitted and stored in the embodiments of the present invention is significantly reduced.

[0049] It should be noted that the neural network to be inferred includes shallow and deep layers. Shallow layers refer to neural network layers close to the input layer, while deep layers refer to neural network layers close to the output layer or fully connected layers. Since shallow layers have larger activation sizes and deep layers have larger weight sizes, if the neural network layer is in a shallow layer of the neural network to be inferred, a weight-sharing partitioning method can be used to reduce activation transmission. If the neural network layer is in a deep layer of the neural network to be inferred, an input-sharing partitioning method can be used to reduce weight storage.

[0050] Furthermore, when the partitioning method includes a weight-sharing partitioning method, each chip stores local weight parameters during the non-computation phase; during the preparation phase, each chip interacts with other chips to obtain all weight parameters. Specifically, if the partitioning method of the neural network layer is a weight-sharing partitioning method, distributed weight storage is used for all weight parameters to avoid frequent DRAM accesses. That is, all weight parameters are divided into multiple weight blocks, each weight block including local weight parameters. During the non-computation phase, each chip in the neural network layer stores only the local weight parameters from one weight block, rather than all weight parameters, greatly reducing the weight storage pressure cached on each chip. During the preparation phase, all chips communicate with each other through the on-chip network to obtain all weight blocks, thereby obtaining all weight parameters.

[0051] Step 150: Based on all segments, all clusters corresponding to each segment, the number of target cores corresponding to each cluster, and the partitioning method corresponding to each neural network layer, schedule the computing resources corresponding to the MCM for the neural network to be inferred.

[0052] Specifically, after determining the segmented segments, all clusters corresponding to each segment, the number of target chips allocated to each cluster, and the partitioning method corresponding to each neural network layer in each cluster, the multi-chip module is configured according to these parameters, thereby deploying the neural network to be inferred onto the configured multi-chip module, and executing the corresponding inference task through the configured multi-chip module.

[0053] The multi-chip computational resource scheduling method for neural networks provided in this invention divides all neural network layers of the neural network to be inferred into multiple segments, each segment including multiple consecutive neural network layers. Based on the cluster merging association relationship representing a preset layer merging strategy corresponding to different cluster numbers, adjacent neural network layers within each segment are merged, dividing each segment into at least one cluster. Based on the latency of each cluster, all chips in the multi-chip module are iteratively adjusted until a load-balanced state is achieved, obtaining the target chip number for each cluster. Based on the depth of each neural network layer in the neural network to be inferred, the partitioning method corresponding to each neural network layer is determined to adjust the storage method of weight parameters and input features for the chips corresponding to each neural network layer. Based on all segments, all clusters corresponding to each segment, the target chip number for each cluster, and the partitioning method corresponding to each neural network layer, the multi-chip module is configured, thereby deploying the neural network to be inferred onto the configured multi-chip module to perform inference tasks. In this invention, clusters are obtained through layer merging, and multi-layer deployment is performed jointly based on the cluster dimension, significantly improving the utilization rate of hardware resources compared to deploying each layer separately. Iterative adjustments to the core particles based on the cluster dimension create a network structure composed of load-balanced clusters, which can effectively achieve pipeline-level computing load balancing, reduce pipeline preheating or cooling bubbles, thereby improving system throughput. By adaptively selecting partitioning methods, it alleviates NoP bandwidth pressure and reduces communication overhead.

[0054] In one embodiment, the cluster merging association is determined based on the following steps: Based on the input and output dimensions of each neural network layer, the parallelizable dimension ratio of each neural network layer is determined; the parallelizable dimension ratio is used to characterize the parallelism of the input and output dimensions of each neural network layer. Starting with each neural network layer as an independent cluster, adjacent clusters are iteratively merged based on the ratio of parallelizable dimensions corresponding to each cluster until all clusters are merged into one cluster, thus obtaining a preset layer merging strategy corresponding to different numbers of clusters.

[0055] Specifically, in this embodiment of the invention, the inherent parallelism of neural network layers is utilized to reduce the search space. That is, firstly, the parallelizable dimension ratio of each neural network layer is determined based on the parallelism capabilities of its input and output dimensions. Specifically, the input dimension is the number of channels of the input feature corresponding to that neural network layer, and the output dimension is the number of channels of the output feature corresponding to that neural network layer. The ratio of the input dimension to the output dimension corresponding to that neural network layer is calculated; this ratio is the parallelizable dimension ratio for that neural network layer. Then, during initialization, each neural network layer is treated as an independent cluster, and all clusters are iteratively merged. In each iteration, the two adjacent clusters with the closest parallelism capabilities are merged. Iteration stops when all clusters are merged into one cluster. At this point, a preset layer merging strategy corresponding to different numbers of clusters can be obtained.

[0056] In one embodiment, the iterative merging of adjacent clusters based on the ratio of parallelizable dimensions corresponding to each cluster includes: S11. Based on the ratio of parallelizable dimensions of each cluster and its neighboring clusters, determine the parallel similarity between each cluster and its neighboring clusters; S12. Merge the two clusters with the lowest parallel similarity among all parallel similarities into a new cluster; S13. When the number of the new cluster and the number of other unmerged clusters is equal to 1, stop the iteration and obtain the preset layer merging strategy corresponding to different numbers of clusters. S14. If the number of the new cluster and the other unmerged clusters is greater than 1, the new cluster and the other unmerged clusters are determined as the new clusters. Steps S11 to S12 are repeated until the iteration stops when the number of the new cluster and the other unmerged clusters is equal to 1.

[0057] Specifically, starting with each neural network layer as an independent cluster, after determining the parallelizable dimension ratio for each cluster, the absolute value of the difference between the parallelizable dimension ratios of adjacent clusters is calculated. This absolute value is the parallel similarity between the two adjacent clusters. This parallel similarity reflects the similarity of the parallel capabilities between adjacent clusters. The smaller the parallel similarity, the more similar the parallel capabilities between adjacent clusters; conversely, the larger the parallel similarity, the more different the parallel capabilities between adjacent clusters. In the current iteration, after determining all parallel similarities, all parallel similarities are sorted in ascending order. The two clusters corresponding to the highest parallel similarity (i.e., the minimum parallel similarity) are merged into a new cluster, while the other clusters remain unchanged. Then, the number of the new cluster and the other clusters is summed. If the sum equals 1, it indicates that all clusters have been merged into one cluster. At this point, the iteration can be stopped, and the preset layer merging strategy corresponding to different cluster numbers is determined based on the cluster merging process corresponding to each iteration. If the number obtained is greater than 1, the new cluster and other clusters constitute the initial cluster for the next iteration round. The above parallel similarity calculation and cluster merging steps are repeated until all clusters are merged into one cluster and the iteration stops.

[0058] For example, assuming the partitioned clusters contain 6 neural network layers, and each of the 6 neural network layers is initialized as a separate cluster, with the parallelizability ratios for cluster 1 (1.2), cluster 2 (1.5), cluster 3 (1.3), cluster 4 (1.8), cluster 5 (2.0), and cluster 6 (1.9) being calculated, in the first iteration, the parallel similarity between clusters 1 and 2 is calculated to be 0.3, between clusters 2 and 3 to be 0.2, between clusters 3 and 4 to be 0.5, between clusters 4 and 5 to be 0.2, and between clusters 5 and 6 to be 0.1. Among all parallel similarities, the parallel similarity between clusters 5 and 6 is the smallest at 0.1, meaning that clusters 5 and 6 have the most similar parallel capabilities. Therefore, clusters 5 and 6 are merged into a new cluster 56. Subsequently, since the sum of the numbers of clusters 1 to 4 and the new cluster 56 is greater than 1, in the second iteration, the average of the parallelizable dimension ratios of clusters 5 and 6 can be taken as the parallelizable dimension ratio of the new cluster 56, i.e., the parallelizable dimension ratio of the new cluster 56 is 1.95. Then, the parallel similarity between cluster 4 and the new cluster 56 is calculated to be 0.15. Among all the parallel similarities in the second iteration, the parallel similarity between cluster 4 and the new cluster 56 is the smallest at 0.15, indicating that the parallel capabilities of cluster 4 and the new cluster 56 are most similar. Therefore, cluster 4 and the new cluster 56 are merged into a new cluster 456. Then, since the sum of the numbers of clusters 1 to 3 and the new cluster 456 is greater than 1, the third iteration merging continues. The above steps are repeated until all clusters are merged into one cluster, at which point the iteration stops. The resulting cluster merging relationships are shown in Table 1.

[0059] Table 1 Cluster number Preset layer merging strategy 6 Cluster 1, Cluster 2, Cluster 3, Cluster 4, Cluster 5, Cluster 6 5 Cluster 1, Cluster 2, Cluster 3, Cluster 4, Cluster 56 (Cluster 5 + Cluster 6) 4 Cluster 1, Cluster 2, Cluster 3, Cluster 4, Cluster 5, Cluster 6 (Cluster 4 + Cluster 5 + Cluster 6) 3 Cluster 1, Cluster 23 (Cluster 2 + Cluster 3), Cluster 456 (Cluster 4 + Cluster 5 + Cluster 6) 2 Cluster 123 (cluster 1 + cluster 2 + cluster 3), cluster 456 (cluster 4 + cluster 5 + cluster 6) 1 Cluster 123456 (Cluster 1+Cluster 2+Cluster 3+Cluster 4+Cluster 5+Cluster 6) Optionally, when determining the parallelizable dimension ratio of the new cluster, the mean or weighted sum of the parallelizable dimension ratios of the two clusters can be calculated, and this embodiment of the invention does not limit this.

[0060] In one embodiment, the iterative adjustment of all chips in the multi-chip module (MCM) based on the latency of each cluster to obtain the target number of chips for each cluster in a load-balanced state includes: S21. Based on the computational cost corresponding to each cluster, all cores in the MCM are initially allocated to each cluster; S22. Based on the delay duration of each cluster, determine the first cluster with the longest delay duration, the second cluster with the shortest delay duration, and the initial overall delay duration. S23. Distribute one core particle corresponding to the second cluster to the first cluster; S24. Based on the adjusted delay duration of each cluster, determine the current overall delay duration of all clusters. S25. If the current overall delay duration is greater than or equal to the initial overall delay duration, stop the iteration and obtain the target number of core particles corresponding to each cluster in the load-balanced state. S26. If the current overall delay duration is less than the initial overall delay duration, the adjusted delay duration corresponding to each cluster is determined as the new delay duration corresponding to each cluster, and the current overall delay duration is determined as the new initial overall delay duration. Steps S22 to S24 are repeated until the current overall delay duration is greater than or equal to the initial overall delay duration, at which point the iteration stops.

[0061] Specifically, the overall latency for all clusters is the execution time of the segments corresponding to all clusters. The latency for each cluster is the execution time for that cluster, and the latency for each cluster is the sum of the latency for each neural network layer within that cluster. The latency for each neural network layer is the execution time for that layer, and is determined based on the execution time for each layer in the preparation, computation, and communication phases.

[0062] For all clusters within the current segment, the computational load of each cluster reflects its corresponding load. Based on the computational load of each cluster, all cores belonging to this segment in the MCM are initially allocated, determining the initial number and position of cores for each cluster. Then, in the first iteration, simulation is performed based on the initially allocated cores to determine the latency of each cluster. Based on the latency of each cluster, the initial overall latency of the segment for all clusters is determined. Next, the latency of each cluster is sorted in descending order. The cluster with the longest latency is designated as the first cluster (the slowest cluster), and the cluster with the shortest latency is designated as the second cluster (the fastest cluster). Shorter latency indicates more abundant hardware resources and lower load pressure for the cluster; longer latency indicates insufficient hardware resources and higher load pressure for the cluster. Therefore, to achieve load balancing, one core from the second cluster can be allocated to the first cluster; that is, the number of cores in the second cluster decreases by one, and the number of cores in the first cluster increases by one. After adjusting the number of cores in the first and second clusters, simulations are performed on each cluster with the adjusted core count to determine the adjusted delay duration of each cluster, and the adjusted current overall delay duration corresponding to that segment is determined based on the adjusted delay duration of each cluster. Then, the current overall delay duration is compared with the initial overall delay duration. If the current overall delay duration is less than the initial overall delay duration, the number of cores in subsequent iterations is adjusted based on the adjusted clusters until the current overall delay duration is greater than or equal to the initial overall delay duration, at which point the iteration stops. If the current overall delay duration is greater than or equal to the initial overall delay duration, it indicates that a load-balanced state has been reached. At this point, iteration stops, and the adjusted core count of each cluster is the optimal target core count under load-balanced conditions.

[0063] Optionally, when allocating a core from the second cluster to the first cluster, the core closest to the first cluster can be allocated to the first cluster according to the principle of proximity. This embodiment of the invention does not limit this.

[0064] In one embodiment, the initial allocation of all cores in the MCM to each of the clusters based on the computational cost corresponding to each cluster includes: Based on the computational amount of each cluster, determine the computational load ratio of each cluster. Based on the calculated load ratio, all cores in the MCM are proportionally allocated to each of the clusters.

[0065] Specifically, based on the computational load corresponding to each cluster, the computational load ratio for each cluster is determined. Then, based on this computational load ratio, all cores belonging to that segment in the MCM are proportionally allocated. For example, if the number of cores is 100 and the computational load ratio is 1:3:4:2, then cluster 1 can be initially allocated 10 cores, cluster 2 30 cores, cluster 3 40 cores, and cluster 4 20 cores.

[0066] It should be noted that if the initial number of cores allocated to each cluster according to the calculated load ratio is a decimal, the decimal will be rounded to ensure that the initial number of cores allocated to each cluster is an integer.

[0067] Optionally, the computational complexity of each cluster can be calculated based on the shape parameters such as the input dimension, output dimension, and kernel parameters of each neural network layer in each cluster. This embodiment of the invention does not limit this.

[0068] In one embodiment, determining the current overall latency for all clusters based on the adjusted latency for each cluster includes: Based on the adjusted delay duration corresponding to each cluster, the adjusted maximum delay duration is determined. Based on the total number of samples, the adjusted longest delay duration, and the number of clusters corresponding to all clusters, determine the current overall delay duration for all clusters.

[0069] Specifically, since the clusters within each segment constitute different stages of the pipeline, the delay duration of a segment (i.e., the overall delay duration corresponding to all clusters) depends on the cluster with the longest delay duration within that segment. Therefore, after determining the adjusted delay duration for each cluster, the adjusted delay durations are sorted in descending order, and the oldest adjusted delay duration is the longest adjusted delay duration. Next, the sum of the total number of samples and the number of clusters corresponding to all clusters is calculated. This sum is subtracted by one to obtain the difference. Then, the product of this difference and the longest adjusted delay duration is calculated; this product is the current overall delay duration corresponding to all clusters.

[0070] It should be noted that the calculation process for the initial overall delay duration for all clusters is similar to the calculation process for the current overall delay duration, and will not be repeated here in this embodiment of the invention.

[0071] In one embodiment, determining the adjusted delay duration for each cluster includes: The preparation time, computation time, and communication time of each neural network layer in each cluster are determined respectively for the accurate stage, the computation stage, and the communication stage. The layer execution time is determined based on the computation time and communication time corresponding to each neural network layer. Based on the preparation time and layer execution time corresponding to each neural network layer, the adjusted layer delay time corresponding to each neural network layer is determined. Based on the layer delay duration corresponding to each neural network layer in each cluster, the adjusted delay duration corresponding to each cluster is determined.

[0072] Specifically, each neural network comprises three phases during execution: a preparation phase, a computation phase, and a communication phase. The preparation phase retrieves weights and activations from off-chip memory or other chips. The computation phase performs distributed computations on the chips. The communication phase collects computation results and distributes them to the corresponding regions of the next layer. After determining the preparation, computation, and communication durations for each layer, to minimize the impact of communication, in this embodiment, the communication and computation phases are executed concurrently. The larger of the computation and communication durations is determined as the layer execution duration. Then, the sum of the preparation and execution durations is calculated, which represents the layer latency for that neural network layer. Finally, for each cluster, the sum of the layer latencys of all neural network layers within that cluster is calculated; this sum represents the latency for that cluster.

[0073] This invention, implemented using a 28nm 800MHz process, utilizes Timeloop to evaluate chiplet computation, BookSim2 to evaluate NoP communication, and Ramulator2 to evaluate DRAM access. Experiments deploying AlexNet, VGG16, DarkNet19, and ResNet series networks on MCMs with 16 to 256 chiplets demonstrate that this invention achieves optimal performance across all configurations, with the maximum improvement occurring at the deepest network and the largest number of chiplets. Specifically, in an experiment deploying a ResNet-152 network on an MCM with 256 chiplets, this invention achieves a 1.73x throughput improvement compared to existing optimal segmented pipeline schemes, while maintaining similar power consumption. The search algorithm can perform a search on scales of up to 10... 164 We found the scheduling schemes that ranked in the top 0.05% in terms of performance within the design space.

[0074] The following describes the multi-core computing resource scheduling device for neural networks provided by the present invention. The multi-core computing resource scheduling device for neural networks described below can be referred to in correspondence with the multi-core computing resource scheduling method for neural networks described above.

[0075] This invention also provides a multi-core computing resource scheduling device for neural networks. Figure 2 This is a schematic diagram of the structure of a multi-core computing resource scheduling device for neural networks provided in an embodiment of the present invention, as shown below. Figure 2As shown, the multi-core computing resource scheduling device 200 for neural networks includes: a partitioning module 210, a merging module 220, an iterative adjustment module 230, a partitioning module 240, and a scheduling module 250.

[0076] The segmentation module 210 is used to divide all neural network layers of the neural network to be reasoned into at least one segment, each segment including multiple consecutive neural network layers. The merging module 220 is used to merge adjacent neural network layers within each segment based on the cluster merging association relationship to obtain at least one cluster corresponding to each segment; the cluster merging association relationship is used to characterize the preset layer merging strategy corresponding to different numbers of clusters. The iterative adjustment module 230 is used to iteratively adjust all the chips in the multi-chip module MCM based on the delay duration of each cluster, so as to obtain the target number of chips for each cluster under the load balancing state. The partitioning module 240 is used to determine the partitioning method corresponding to each neural network layer based on the depth of each neural network layer in the neural network to be inferred; the partitioning method is used to characterize the storage method of weight parameters and input features; The scheduling module 250 is used to schedule the computing resources corresponding to the MCM for the neural network to be inferred based on all segments, all clusters corresponding to each segment, the number of target cores corresponding to each cluster, and the partitioning method corresponding to each neural network layer.

[0077] The multi-chip computational resource scheduling device for neural networks provided in this invention divides all neural network layers of the neural network to be inferred into multiple segments, each segment including multiple consecutive neural network layers. Based on the cluster merging association relationship representing a preset layer merging strategy corresponding to different cluster numbers, adjacent neural network layers within each segment are merged, dividing each segment into at least one cluster. Based on the latency of each cluster, all chips in the multi-chip module are iteratively adjusted until a load-balanced state is achieved, obtaining the target chip number for each cluster. Based on the depth of each neural network layer in the neural network to be inferred, the partitioning method corresponding to each neural network layer is determined to adjust the storage method of weight parameters and input features for the chips corresponding to each neural network layer. Based on all segments, all clusters corresponding to each segment, the target chip number for each cluster, and the partitioning method corresponding to each neural network layer, the multi-chip module is configured, thereby deploying the neural network to be inferred onto the configured multi-chip module to perform inference tasks. In this invention, clusters are obtained through layer merging, and multi-layer deployment is performed jointly based on the cluster dimension, significantly improving the utilization rate of hardware resources compared to deploying each layer separately. Iterative adjustments to the core particles based on the cluster dimension create a network structure composed of load-balanced clusters, which can effectively achieve pipeline-level computing load balancing, reduce pipeline preheating or cooling bubbles, thereby improving system throughput. By adaptively selecting partitioning methods, it alleviates NoP bandwidth pressure and reduces communication overhead.

[0078] Optionally, the multi-core computing resource scheduling device 200 for neural networks further includes a determination module, which is specifically used for: Based on the input and output dimensions of each neural network layer, the parallelizable dimension ratio of each neural network layer is determined; the parallelizable dimension ratio is used to characterize the parallelism of the input and output dimensions of each neural network layer. Starting with each neural network layer as an independent cluster, adjacent clusters are iteratively merged based on the ratio of parallelizable dimensions corresponding to each cluster until all clusters are merged into one cluster, thus obtaining a preset layer merging strategy corresponding to different numbers of clusters.

[0079] Optionally, this determining module is specifically used for: S11. Based on the ratio of parallelizable dimensions of each cluster and its neighboring clusters, determine the parallel similarity between each cluster and its neighboring clusters; S12. Merge the two clusters with the lowest parallel similarity among all parallel similarities into a new cluster; S13. When the number of the new cluster and the number of other unmerged clusters is equal to 1, stop the iteration and obtain the preset layer merging strategy corresponding to different numbers of clusters. S14. If the number of the new cluster and the other unmerged clusters is greater than 1, the new cluster and the other unmerged clusters are determined as the new clusters. Steps S11 to S12 are repeated until the iteration stops when the number of the new cluster and the other unmerged clusters is equal to 1.

[0080] Optionally, the iterative adjustment module 230 is specifically used for: S21. Based on the computational cost corresponding to each cluster, all cores in the MCM are initially allocated to each cluster; S22. Based on the delay duration of each cluster, determine the first cluster with the longest delay duration, the second cluster with the shortest delay duration, and the initial overall delay duration. S23. Distribute one core particle corresponding to the second cluster to the first cluster; S24. Based on the adjusted delay duration of each cluster, determine the current overall delay duration of all clusters. S25. If the current overall delay duration is greater than or equal to the initial overall delay duration, stop the iteration and obtain the target number of core particles corresponding to each cluster in the load-balanced state. S26. If the current overall delay duration is less than the initial overall delay duration, the adjusted delay duration corresponding to each cluster is determined as the new delay duration corresponding to each cluster, and the current overall delay duration is determined as the new initial overall delay duration. Steps S22 to S24 are repeated until the current overall delay duration is greater than or equal to the initial overall delay duration, at which point the iteration stops.

[0081] Optionally, the iterative adjustment module 230 is specifically used for: Based on the computational amount of each cluster, determine the computational load ratio of each cluster. Based on the calculated load ratio, all cores in the MCM are proportionally allocated to each of the clusters.

[0082] Optionally, the iterative adjustment module 230 is specifically used for: Based on the adjusted delay duration corresponding to each cluster, the adjusted maximum delay duration is determined. Based on the total number of samples, the adjusted longest delay duration, and the number of clusters corresponding to all clusters, determine the current overall delay duration for all clusters.

[0083] Optionally, the iterative adjustment module 230 is specifically used for: The preparation time, computation time, and communication time of each neural network layer in each cluster are determined respectively for the accurate stage, the computation stage, and the communication stage. The layer execution time is determined based on the computation time and communication time corresponding to each neural network layer. Based on the preparation time and layer execution time corresponding to each neural network layer, the adjusted layer delay time corresponding to each neural network layer is determined. Based on the layer delay duration corresponding to each neural network layer in each cluster, the adjusted delay duration corresponding to each cluster is determined.

[0084] Optionally, the partitioning method includes an input-shared partitioning method or a weighted-shared partitioning method, wherein: The input sharing partitioning method is used to characterize that each core corresponding to the neural network layer stores all input features and local weight parameters; each core obtains all weight parameters based on on-chip network communication interaction. The weight-sharing partitioning method is used to characterize that each core corresponding to the neural network layer stores all weight parameters and local input features; adjacent cores obtain the input features of the boundary overlapping region based on inter-layer communication interaction.

[0085] Optionally, when the partitioning method includes a weight-sharing partitioning method, each core stores local weight parameters during the non-computation phase; during the preparation phase, each core interacts with other cores to obtain all weight parameters.

[0086] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can call logical instructions in the memory 330 to execute a multi-chip computational resource scheduling method for a neural network. This method includes: dividing all neural network layers of the neural network to be inferred into at least one segment, each segment comprising multiple consecutive neural network layers; merging adjacent neural network layers within each segment based on cluster merging relationships to obtain at least one cluster corresponding to each segment; the cluster merging relationships characterizing preset layer merging strategies corresponding to different cluster numbers; iteratively adjusting all chips in the multi-chip module (MCM) based on the latency of each cluster to obtain the target chip number corresponding to each cluster under load balancing conditions; determining the partitioning method corresponding to each neural network layer based on the depth of each neural network layer in the neural network to be inferred; the partitioning method characterizing the storage method for weight parameters and input features; and scheduling computational resources corresponding to the MCM for the neural network to be inferred based on all segments, all clusters corresponding to each segment, the target chip number corresponding to each cluster, and the partitioning method corresponding to each neural network layer.

[0087] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-chip computational resource scheduling method for neural networks provided by the above methods. The method includes: dividing all neural network layers of the neural network to be inferred into at least one segment, each segment including multiple consecutive neural network layers; merging adjacent neural network layers within each segment based on cluster merging associations to obtain at least one cluster corresponding to each segment; the cluster merging associations are used to characterize preset layer merging strategies corresponding to different cluster numbers; iteratively adjusting all chips in the multi-chip module (MCM) based on the latency of each cluster to obtain the target chip number corresponding to each cluster under load balancing; determining the partitioning method corresponding to each neural network layer based on the depth of each neural network layer in the neural network to be inferred; the partitioning method is used to characterize the storage method of weight parameters and input features; and scheduling the computational resources corresponding to the MCM for the neural network to be inferred based on all segments, all clusters corresponding to each segment, the target chip number corresponding to each cluster, and the partitioning method corresponding to each neural network layer.

[0089] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-chip computational resource scheduling method for neural networks provided by the methods described above. The method includes: dividing all neural network layers of the neural network to be inferred into at least one segment, each segment comprising multiple consecutive neural network layers; merging adjacent neural network layers within each segment based on cluster merging associations to obtain at least one cluster corresponding to each segment; the cluster merging associations characterizing preset layer merging strategies corresponding to different cluster numbers; iteratively adjusting all chips in the multi-chip module (MCM) based on the latency of each cluster to obtain a target chip number corresponding to each cluster under load balancing conditions; determining the partitioning method corresponding to each neural network layer based on the depth of each neural network layer in the neural network to be inferred; the partitioning method characterizing the storage method for weight parameters and input features; and scheduling computational resources corresponding to the MCM for the neural network to be inferred based on all segments, all clusters corresponding to each segment, the target chip number corresponding to each cluster, and the partitioning method corresponding to each neural network layer.

[0090] This invention also provides a multi-core computing resource scheduling system for neural networks, comprising: An electronic device is used to execute the multi-core computing resource scheduling method for neural networks described in any of the above embodiments to obtain a computing resource scheduling strategy. A multi-chip module is used to execute the inference task of the neural network to be inferred based on the computing resource scheduling strategy.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for scheduling multi-core computing resources in neural networks, characterized in that, include: Divide all neural network layers of the neural network to be reasoned into at least one segment, each segment comprising multiple consecutive neural network layers; Based on the cluster merging relationship, adjacent neural network layers within each segment are merged to obtain at least one cluster corresponding to each segment; The cluster merging association is used to characterize the preset layer merging strategy corresponding to different numbers of clusters; Based on the delay duration of each cluster, all chips in the multi-chip module (MCM) are iteratively adjusted to obtain the target number of chips for each cluster under load balancing. Based on the depth of each neural network layer in the neural network to be inferred, the partitioning method corresponding to each neural network layer is determined; the partitioning method is used to characterize the storage method of weight parameters and input features; Based on all segments, all clusters corresponding to each segment, the number of target cores corresponding to each cluster, and the partitioning method corresponding to each neural network layer, the computing resources corresponding to the MCM are scheduled for the neural network to be inferred.

2. The multi-core computing resource scheduling method for neural networks according to claim 1, characterized in that, The cluster merging association is determined based on the following steps: Based on the input and output dimensions of each neural network layer, the parallelizable dimension ratio of each neural network layer is determined; the parallelizable dimension ratio is used to characterize the parallelism of the input and output dimensions of each neural network layer. Starting with each neural network layer as an independent cluster, adjacent clusters are iteratively merged based on the ratio of parallelizable dimensions corresponding to each cluster until all clusters are merged into one cluster, thus obtaining a preset layer merging strategy corresponding to different numbers of clusters.

3. The multi-core computing resource scheduling method for neural networks according to claim 2, characterized in that, The iterative merging of adjacent clusters based on the ratio of parallelizable dimensions corresponding to each cluster includes: S11. Based on the ratio of parallelizable dimensions of each cluster and its neighboring clusters, determine the parallel similarity between each cluster and its neighboring clusters; S12. Merge the two clusters with the lowest parallel similarity among all parallel similarities into a new cluster; S13. When the number of the new cluster and the number of other unmerged clusters is equal to 1, stop the iteration and obtain the preset layer merging strategy corresponding to different numbers of clusters; S14. If the number of the new cluster and the other unmerged clusters is greater than 1, the new cluster and the other unmerged clusters are determined as the new all clusters. Steps S11 to S12 are repeated until the iteration stops when the number of the new cluster and the other unmerged clusters is equal to 1.

4. The multi-core computing resource scheduling method for neural networks according to claim 1, characterized in that, The step of iteratively adjusting all chips in the multi-chip module (MCM) based on the latency of each cluster to obtain the target number of chips for each cluster under load balancing conditions includes: S21. Based on the computational cost corresponding to each cluster, all cores in the MCM are initially allocated to each cluster; S22. Based on the delay duration of each cluster, determine the first cluster with the longest delay duration, the second cluster with the shortest delay duration, and the initial overall delay duration. S23. Distribute one core particle corresponding to the second cluster to the first cluster; S24. Based on the adjusted delay duration of each cluster, determine the current overall delay duration of all clusters. S25. If the current overall delay duration is greater than or equal to the initial overall delay duration, stop the iteration and obtain the target number of core particles corresponding to each cluster in the load-balanced state. S26. If the current overall delay duration is less than the initial overall delay duration, the adjusted delay duration corresponding to each cluster is determined as the new delay duration corresponding to each cluster, and the current overall delay duration is determined as the new initial overall delay duration. Steps S22 to S24 are repeated until the current overall delay duration is greater than or equal to the initial overall delay duration, at which point the iteration stops.

5. The multi-core computing resource scheduling method for neural networks according to claim 4, characterized in that, The initial allocation of all cores in the MCM to each cluster based on the computational cost corresponding to each cluster includes: Based on the computational amount of each cluster, determine the computational load ratio of each cluster. Based on the calculated load ratio, all cores in the MCM are proportionally allocated to each of the clusters.

6. The multi-core computing resource scheduling method for neural networks according to claim 4, characterized in that, The step of determining the current overall latency for all clusters based on the adjusted latency for each cluster includes: Based on the adjusted delay duration corresponding to each cluster, the adjusted maximum delay duration is determined. Based on the total number of samples, the adjusted longest delay duration, and the number of clusters corresponding to all clusters, determine the current overall delay duration for all clusters.

7. The multi-core computing resource scheduling method for neural networks according to claim 4, characterized in that, Determine the delay duration corresponding to each cluster after adjustment, including: The preparation time, computation time, and communication time of each neural network layer in each cluster are determined respectively for the accurate stage, the computation stage, and the communication stage. The layer execution time is determined based on the computation time and communication time corresponding to each neural network layer. Based on the preparation time and layer execution time corresponding to each neural network layer, the adjusted layer delay time corresponding to each neural network layer is determined. Based on the layer delay duration corresponding to each neural network layer in each cluster, the adjusted delay duration corresponding to each cluster is determined.

8. The multi-core computing resource scheduling method for neural networks according to any one of claims 1-7, characterized in that, The partitioning method includes either input-shared partitioning or weighted-shared partitioning, wherein: The input sharing partitioning method is used to characterize that each core corresponding to the neural network layer stores all input features and local weight parameters; each core obtains all weight parameters based on on-chip network communication interaction. The weight-sharing partitioning method is used to characterize that each core corresponding to the neural network layer stores all weight parameters and local input features; adjacent cores obtain the input features of the boundary overlapping region based on inter-layer communication interaction.

9. The multi-core computing resource scheduling method for neural networks according to any one of claims 1-7, characterized in that, When the partitioning method includes a weight-sharing partitioning method, each core stores local weight parameters during the non-computation phase; during the preparation phase, each core interacts with other cores to obtain all weight parameters.

10. A multi-core computing resource scheduling device for neural networks, characterized in that, include: A segmentation module is used to divide all neural network layers of the neural network to be inferred into at least one segment, each segment comprising multiple consecutive neural network layers. The merging module is used to merge adjacent neural network layers within each segment based on cluster merging association relationships to obtain at least one cluster corresponding to each segment; The cluster merging association is used to characterize the preset layer merging strategy corresponding to different numbers of clusters; The iterative adjustment module is used to iteratively adjust all the chips in the multi-chip module (MCM) based on the delay duration of each cluster, so as to obtain the target number of chips for each cluster under the load balancing state. A partitioning module is used to determine the partitioning method corresponding to each neural network layer based on the depth of each neural network layer in the neural network to be inferred; the partitioning method is used to characterize the storage method of weight parameters and input features; The scheduling module is used to schedule the computing resources corresponding to the MCM for the neural network to be inferred based on all segments, all clusters corresponding to each segment, the number of target cores corresponding to each cluster, and the partitioning method corresponding to each neural network layer.

Images