Hetero-aware adaptive low-bit neural network deployment method

Abstract

The application provides a heterogeneous-aware adaptive low-bit neural network deployment method, and relates to the fields of artificial intelligence and heterogeneous computing technology. First, hierarchical computing feature information of each layer of a neural network and dynamic characteristic parameter information of heterogeneous hardware are acquired to form a multi-level basic information set. Then, a hierarchical performance correlation model is constructed to describe the correlation between computing accuracy, hardware dynamic characteristics and layer computing performance. During online operation, real-time hardware load state and energy efficiency constraint conditions are tracked to generate dynamic state monitoring results. A multi-objective optimization algorithm is used in combination with the two to generate a hierarchical deployment configuration scheme. Finally, a lightweight runtime scheduler is called to allocate computing tasks, load inter-layer dependent data and execute according to the scheme, thereby realizing dynamic neural network deployment across hardware devices and improving deployment effect and operation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and heterogeneous computing, in particular to a heterogeneous-aware adaptive low-bit neural network deployment method. BACKGROUND

[0002] In today's era of rapid development of artificial intelligence, neural networks have been widely applied in many fields such as image recognition, natural language processing, etc. However, the huge amount of computation of neural networks requires high demand for hardware resources, which makes its deployment in resource-constrained heterogeneous hardware environment face many challenges.

[0003] Traditional neural network deployment methods often use fixed computing precision and hardware allocation strategies. On the one hand, for the selection of computing precision, it is usually determined in the model training stage and remains unchanged throughout the deployment process, without fully considering the characteristics of different hardware devices and the dynamic changes of actual runtime. For example, in some scenarios where the requirement for computing precision is not high, high-precision computation is still used, resulting in unnecessary resource waste; and in scenarios where there is a certain tolerance for precision, low-bit computation is not fully utilized to improve computational efficiency. On the other hand, in terms of hardware allocation, it is usually simply allocated according to the static computing power of hardware devices, without considering the dynamic resource occupation characteristics, load fluctuation trends and energy efficiency constraints of hardware devices. This may lead to some hardware devices being overloaded during operation, while other hardware devices are idle, which cannot achieve efficient utilization of hardware resources, and thus affects the deployment effect and running efficiency of neural networks. SUMMARY

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present application, the present application provides a heterogeneous-aware adaptive low-bit neural network deployment method, which comprises:

[0005] Obtaining hierarchical computing feature information of each layer of the neural network and parameter information reflecting dynamic characteristics of the heterogeneous hardware, forming a multi-level basic information set, the hierarchical computing feature information including basic computing operation types, high-order data interaction modes and inter-layer dependency weights of each layer, and the dynamic hardware parameter information including static computing power, dynamic resource occupation characteristics and load fluctuation trends of each hardware device in the heterogeneous hardware;

[0006] Based on the multi-level basic information set, constructing an offline performance model of each layer of the neural network under different low-bit computing precision and each hardware device reflecting the dynamic characteristics of the heterogeneous hardware, obtaining a hierarchical performance correlation model, the hierarchical performance correlation model being used to describe the time sequence correlation between computing precision, hardware device dynamic characteristics and layer computing performance;

[0007] During online running, real-time load states of each hardware device in the heterogeneous hardware and dynamically adjusted energy efficiency constraint conditions are tracked, dynamic state monitoring results are generated, the real-time load states include device resource occupancy, calculation response delay and load fluctuation rate, and the dynamically adjusted energy efficiency constraint conditions include device-level power consumption upper limit, system-level calculation efficiency lower limit and load fluctuation adaptation threshold;

[0008] In combination with the hierarchical performance correlation model and the dynamic state monitoring results, a multi-objective optimization algorithm is used to dynamically select adaptive low-bit calculation precision and corresponding hardware devices for each layer of the neural network, and a hierarchical deployment configuration scheme is formed, the hierarchical deployment configuration scheme includes precision selection results, device allocation results, inter-layer data transmission weights and dynamic adjustment trigger conditions of each layer;

[0009] A lightweight runtime scheduler with task preloading capability is called, and calculation tasks of each layer of the neural network are allocated to corresponding hardware devices according to the hierarchical deployment configuration scheme, inter-layer dependent data is synchronously loaded, and calculation tasks are executed using selected low-bit calculation precision, dynamic neural network deployment across hardware devices is realized, and hierarchical cross-device deployment execution results are obtained.

[0010] In another aspect, the application also provides a heterogeneous-aware adaptive low-bit neural network deployment system, comprising:

[0011] A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to execute the above-mentioned heterogeneous-aware adaptive low-bit neural network deployment method by executing the machine-executable instructions.

[0012] Based on the above aspects, by obtaining hierarchical calculation feature information of each layer of the neural network and dynamic characteristic parameter information of the heterogeneous hardware, a multi-level basic information set is formed, a hierarchical performance correlation model constructed based on the multi-level basic information set can accurately describe the time sequence correlation relationship between calculation precision, hardware device dynamic characteristics and layer calculation performance, so that it can be flexibly adjusted during deployment. The dynamic state monitoring results tracked during online running can reflect the running state and energy efficiency constraint conditions of the heterogeneous hardware in real time. In combination with the performance correlation model and the monitoring results, a hierarchical deployment configuration scheme generated by using a multi-objective optimization algorithm can comprehensively consider multiple factors, dynamically select adaptive low-bit calculation precision and hardware devices for each layer of the neural network, realize efficient use of resources and maximize calculation performance. Finally, through the lightweight runtime scheduler, task allocation and calculation execution are performed according to the deployment configuration scheme, dynamic neural network deployment across hardware devices is realized, the deployment effect and running efficiency of the neural network in the heterogeneous hardware environment are effectively improved, and the adaptability and flexibility of the system are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is an execution flow schematic diagram of a heterogeneous-aware adaptive low-bit neural network deployment method provided by an embodiment of the present application.

[0014] Figure 2 is a schematic diagram of exemplary hardware and software components of a heterogeneous-aware adaptive low-bit neural network deployment system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0015] The present application will be described in detail below with reference to the accompanying drawings, Figure 1 is a flow schematic diagram of a heterogeneous-aware adaptive low-bit neural network deployment method provided by an embodiment of the present application, which will be described in detail below.

[0016] Step S110: Obtain hierarchical computing feature information of each layer of the neural network and parameter information reflecting dynamic characteristics of the heterogeneous hardware, form a multi-level basic information set, the hierarchical computing feature information includes the basic computing operation type, high-order data interaction mode and inter-layer dependency weight of each layer, and the dynamic hardware parameter information includes the static computing capability, dynamic resource occupation characteristics and load fluctuation trend of each hardware device in the heterogeneous hardware.

[0017] In this embodiment, the deployment of a convolutional neural network model in an image recognition field on a heterogeneous hardware platform including a central processing unit, a graphics processing unit and a neural network processing unit is taken as an example for illustration. The convolutional neural network model includes input layer, multiple convolutional layers, pooling layer, fully connected layer and output layer and different types of network layers. When obtaining the hierarchical computing feature information and dynamic hardware parameter information, comprehensive and detailed information collection and analysis of each layer of the network and the hardware device are required.

[0018] Step S111: Analyze the structured network description file of the neural network to be deployed, extract the inter-layer topological connection relationship, structure attribute of each layer and inter-layer data dependency strength of the neural network, and generate multi-level layer structure description information, the multi-level layer structure description information includes the input and output data dimensions of each layer, layer type identifier, inter-layer data flow direction and dependency weight coefficient.

[0019] First, a structured network description file of a convolutional neural network model to be deployed is parsed. The structured network description file can be a file stored in a protocol buffer format or other standard format, which records the overall architecture information of the network in detail. During the parsing process, the file content is read by a dedicated parsing tool, the identification information of each layer is identified, and the topological connection relationship between layers is determined. For example, it is identified that the output of the input layer is connected to the input of the first convolutional layer, the output of the first convolutional layer is connected to the input of the pooling layer, the output of the pooling layer is connected to the input of the next convolutional layer, and so on, until the output layer, thereby constructing a complete inter-layer topological connection relationship diagram.

[0020] Next, the structural attributes of each layer are extracted. For a convolutional layer, its structural attributes include the number of convolutional kernels, the size of the convolutional kernel, the step size, the padding method, etc.; for a pooling layer, it includes the pooling type, the pooling kernel size, the step size, etc.; for a fully connected layer, it includes the number of neurons, etc. At the same time, the input and output data dimensions of each layer are determined, for example, the input data dimension of the input layer can be a specific image size and channel number, and after the convolutional layer processing, the output data dimension will change accordingly according to the number of convolutional kernels, the step size and the padding method. The layer type identifier is used to clearly distinguish whether each layer is a convolutional layer, a pooling layer or a fully connected layer, etc.

[0021] The inter-layer data flow direction is determined according to the inter-layer topological connection relationship, which clearly shows which layer the data flows from and which layer the data flows to. The inter-layer data dependency strength is quantitatively represented by a dependency weight coefficient, which takes into account factors such as data transmission volume, transmission frequency, and the degree of influence of data on the calculation results of subsequent layers. For example, if the output data of a certain convolutional layer is the main input of the next key convolutional layer, and the data transmission volume is large and the frequency is high, then the dependency weight coefficient between the two layers will be relatively large. Through the above process, multi-level layer structure description information is generated, including the input and output data dimensions of each layer, the layer type identifier, the inter-layer data flow direction and the dependency weight coefficient.

[0022] Step S112: Based on the multi-level layer structure description information, perform hierarchical feature extraction on each layer, first extract the basic calculation operation type and basic data interaction amount of the layer, then analyze the data interaction frequency, data reuse rate and feature dimension of the interaction data between the layer and the adjacent layer, generate the high-order data interaction mode of the layer, and finally determine the inter-layer dependency weight of the layer combined with the inter-layer dependency weight coefficient, and integrate to form the hierarchical calculation feature information of each layer. The basic calculation operation type includes convolution operation, full connection operation and activation function operation, and the basic data interaction amount includes layer input data volume and layer output data volume.

[0023] Based on the multi-level layer structure description information generated in step S111, hierarchical feature extraction is performed on each layer. Taking one of the convolution layers as an example, first, it is determined that the basic calculation operation type thereof is a convolution operation. The basic data interaction amount is calculated according to the input and output data dimensions of the convolution layer, and the input data amount is the product of the parameters of the input data dimensions, and the output data amount is the product of the parameters of the output data dimensions.

[0024] Then, the data interaction frequency of the convolution layer and the adjacent layers is analyzed. By counting the number of data transmissions between the convolution layer and the previous layer (such as the previous pooling layer) and the next layer (such as the next convolution layer or pooling layer) in one complete inference process of the model, the data interaction frequency is obtained. The data reuse rate refers to the proportion of the data of the output data of the convolution layer that is reused by the next layer. For example, if the next layer performs sliding window processing on the output data of the convolution layer, some data will be used multiple times, thereby forming a certain data reuse rate. The feature dimension of the interaction data is the dimension information of the input and output data of the convolution layer, including height, width, and channel number. By comprehensively considering the data interaction frequency, the data reuse rate, and the feature dimension of the interaction data, the high-order data interaction mode of the convolution layer is generated.

[0025] Finally, the inter-layer dependency weight of the layer is determined in combination with the dependency weight coefficient of the convolution layer and the adjacent layers obtained in step S111. The inter-layer dependency weight is obtained by normalizing the dependency weight coefficient, and is used to more accurately represent the strength of the dependency relationship between the layer and the adjacent layers. The basic calculation operation type, the basic data interaction amount, the high-order data interaction mode, and the inter-layer dependency weight are integrated to form the hierarchical calculation feature information of the convolution layer. For other types of network layers, such as fully connected layers and pooling layers, hierarchical feature extraction is also performed according to a similar process, so as to obtain the hierarchical calculation feature information of all layers.

[0026] Step S113: Dynamic parameter acquisition is performed on each hardware device contained in the heterogeneous hardware. First, the static calculation capability parameters of each hardware device are acquired, and then the resource occupation parameters, response delay parameters, and power consumption parameters of the device are acquired under different simulation loads. The variation law of the parameters under different loads is analyzed to generate dynamic resource occupation characteristics. Finally, the load fluctuation trend of the device is fitted based on historical load data, and the dynamic hardware parameter information of each hardware device is integrated.

[0027] For the three hardware devices of central processing unit, graphics processing unit and neural network processing unit in a heterogeneous hardware platform, dynamic parameter collection is performed. First, the static computing capability parameters of each hardware device are collected. For the central processing unit, the number of floating point operations per second, the number of parallel computing threads (such as the number of cores of multi-core processors and the support of hyper-threading technology), and the basic memory capacity (such as the cache capacity integrated with the processor and the accessible main memory capacity) are obtained by querying the hardware specification manual or using hardware detection tools. The static computing capability parameter collection methods of graphics processing unit and neural network processing unit are similar, and the corresponding number of floating point operations per second, number of parallel computing threads (or number of stream processors and similar parameters) and basic memory capacity (such as video memory capacity, on-chip storage capacity, etc.) are obtained.

[0028] Then, the dynamic parameters of the device are collected under different simulation loads. Simulation load is realized by running benchmark test programs with different computing intensities, for example, for the central processing unit, running numerical calculation programs with different thread numbers and computing complexities; for the graphics processing unit, running graphics processing programs with different resolutions and rendering complexities; for the neural network processing unit, running neural network inference tasks of different sizes. Under each simulation load, the resource occupation parameters (such as CPU occupation rate of central processing unit, GPU utilization rate of graphics processing unit, and computing unit utilization rate of neural network processing unit) are collected in real time using special performance monitoring tools, response delay parameters (such as the time interval from task submission to completion), and power consumption parameters (real-time power consumption values of the device during running are obtained through hardware power monitoring instruments or software tools).

[0029] Then, the variation law of these parameters under different loads is analyzed. For resource occupation parameters, the load intensity (such as the amount of computation or data of the task) is taken as the horizontal axis, and the resource occupation rate is taken as the vertical axis, and the curve is drawn to obtain the load occupation rate correlation curve. Similarly, the load delay correlation curve (the relationship curve of load intensity and response delay) and the load power consumption correlation curve (the relationship curve of load intensity and power consumption) are drawn, which together constitute the dynamic resource occupation characteristics of the device.

[0030] Finally, the load fluctuation trend of the device is fitted based on historical load data. The load data of each hardware device in different time periods in the past (such as several days or weeks) is collected, including resource occupation rate, task type and other information at different times. Using time series analysis method, the historical data is processed, and a mathematical model of load change with time is established, so as to fit the load fluctuation trend of the device, for example, to predict the approximate change range and regularity of the device load in different time periods of a day. The static computing capability parameters, dynamic resource occupation characteristics (including three correlation curves) and load fluctuation trend are integrated to form the dynamic hardware parameter information of each hardware device.

[0031] Step S114: The multi-level layer structure description information, the hierarchical computing feature information of each layer, and the dynamic hardware parameter information of each hardware device are hierarchically associated and integrated. The hierarchical computing feature information is mapped to the corresponding layer structure description information according to the layer identifier, and the dynamic hardware parameter information is associated with the device type according to the device identifier, to form an initial multi-level information set. For a layer with missing hierarchical computing feature information, a statistical method or a preset rule is used to complete the high-order data interaction mode and the inter-layer dependency weight according to the feature information of the same type of layer. For a device with missing parameter information reflecting the dynamic characteristics of the hardware, an estimation method is used to complete the load fluctuation trend according to the parameter information of the same type of device. Finally, a multi-level basic information set is generated. Each layer information entry in the multi-level basic information set includes a layer identifier, corresponding hierarchical computing feature information, and layer structure description information. Each device information entry includes a device identifier, corresponding dynamic hardware parameter information, and a device type identifier.

[0032] First, the multi-level layer structure description information, the hierarchical computing feature information of each layer, and the dynamic hardware parameter information of each hardware device are hierarchically associated and integrated. The hierarchical computing feature information is mapped to the corresponding layer structure description information according to the layer identifier (such as the unique number of each layer in the network description file), to ensure that the structure attributes, input and output data dimensions, and other structure description information of each layer correspond to its basic computing operation type, high-order data interaction mode, and other computing feature information. For example, the hierarchical computing feature information of the convolution layer numbered "conv1" is associated with its input and output data dimensions, layer type identifier, and other contents in the multi-level layer structure description information.

[0033] Then, the dynamic hardware parameter information is associated with the device type (central processing unit, graphics processing unit, neural network processing unit) according to the device identifier (such as "CPU0", "GPU0", "NPU0"), to clearly define the type of each device and its corresponding static computing capability, dynamic resource occupation characteristics, and load fluctuation trend.

[0034] Through the above two operations, an initial multi-level information set is formed. In the initial multi-level information set, there can be a situation where the hierarchical calculation characteristic information of some layers is missing, for example, some custom type of network layer or layer with special structure, whose high-order data interaction mode and inter-layer dependency weight are difficult to directly extract. For these layers, statistical methods or preset rules are used for completion. The statistical method refers to analyzing the distribution law of the high-order data interaction mode and inter-layer dependency weight of the layers of the same type (such as other custom layers or standard layers with similar structure), and estimating the corresponding parameter value according to the structural similarity and data interaction characteristics of the layer and the layers of the same type. The preset rule is to set fixed high-order data interaction mode parameters and inter-layer dependency weight ranges for specific types of layers according to domain experience or network design principles.

[0035] Similarly, for some hardware devices, if the load fluctuation trend in the dynamic hardware parameter information is missing (such as newly added hardware devices lacking sufficient historical load data), an estimation method is used for completion. According to the load fluctuation trend obtained by the same type of device, combined with the difference between the static calculation capability parameter and the dynamic resource occupation characteristics of the device and the same type of device, the load fluctuation trend is adjusted and estimated, for example, if the static calculation capability of the device is higher than that of the same type of device, the fluctuation under the same load may be relatively small, and accordingly the load fluctuation trend curve of the same type of device is adjusted, such as scaling or translation, to obtain the load fluctuation trend of the device. After completion, the multi-level basic information set is finally generated, wherein each layer information item and device information item contains complete related information.

[0036] Step S120: based on the multi-level basic information set, constructing an offline performance model of each layer of the neural network under different low-bit calculation precisions and each hardware device in the heterogeneous hardware reflecting the dynamic characteristics of the hardware, obtaining a hierarchical performance correlation model, the hierarchical performance correlation model is used to describe the time sequence correlation between calculation precision, hardware device dynamic characteristics and layer calculation performance.

[0037] After obtaining the multi-level basic information set, an offline performance model is constructed based on the set. Here, still taking the deployment of an image recognition convolutional neural network on a heterogeneous hardware platform containing a central processing unit, a graphics processing unit and a neural network processing unit as an example. The purpose of constructing the offline performance model is to describe the calculation performance of each network layer on different hardware devices under different low-bit calculation precisions, considering the dynamic characteristics of the hardware.

[0038] Step S121: Extract the hierarchical computing feature information of each layer from the multi-level basic information set and the dynamic hardware parameter information of each hardware device, and determine the low-bit computing precision gradient range to be tested, which contains multiple bit number selections in ascending order of bit number, each bit number selection corresponds to a set of computing precision configuration parameters, and the computing precision configuration parameters include numerical representation format, operation bit width and rounding strategy.

[0039] From the multi-level basic information set, the hierarchical computing feature information of each network layer (such as input layer, convolutional layer, pooling layer, fully connected layer, output layer, etc.) is extracted respectively, including the basic computing operation type, high-order data interaction mode, and inter-layer dependent weight of each layer; At the same time, the dynamic hardware parameter information of the central processing unit, the graphics processing unit and the neural network processor is extracted, covering static computing capability, dynamic resource occupation characteristics and load fluctuation trend.

[0040] Then, the low-bit computing precision gradient range to be tested is determined. Low-bit computing precision usually refers to computing precision lower than traditional 32-bit or 64-bit, multiple bit numbers are selected in ascending order of bit number, for example, 4-bit, 8-bit and 16-bit are selected as the low-bit computing precision gradient to be tested. Each bit number selection corresponds to a set of computing precision configuration parameters. In terms of numerical representation format, 4-bit precision may use fixed-point number representation, 8-bit and 16-bit precision can select fixed-point number or floating-point number (such as half-precision floating-point number) representation according to requirements; The operation bit width is consistent with the selected bit number, i.e. 4-bit precision corresponds to 4-bit operation bit width, and so on; The rounding strategy is determined according to different computing requirements and hardware support conditions, such as 4-bit and 8-bit precision can use truncation rounding or rounding, and 16-bit precision can use more complex strategies such as banker rounding if it is a floating-point number.

[0041] Step S122: For each layer, according to each bit number selection in the low-bit computing precision gradient range, a plurality of rounds of performance data collection are sequentially performed: first, the computing task of the layer is packaged as a task unit suitable for the hardware device, then the task unit is distributed to each hardware device in the heterogeneous hardware, and the computing task is executed under different simulated load states of the device, and the computing time, power consumption, data transmission delay and performance stability parameters during each round of task execution are collected, and the multi-round collection results are integrated to generate a multi-dimensional performance data set of the layer under each hardware device at the bit number selection.

[0042] For each layer in the neural network, for example, the first convolutional layer, according to the low-bit computing precision gradient range (4-bit, 8-bit, 16-bit) determined in step S121, a plurality of rounds of performance data collection are sequentially performed for each bit number selection.

[0043] First, the calculation task of the convolution layer is packaged into a task unit that is adapted to each hardware device. For a central processing unit, the convolution calculation task is decomposed into subtasks suitable for multi-thread parallel execution, optimized using an instruction set supported by the central processing unit (such as a SIMD instruction), and packaged into a task unit that can be independently scheduled; for a graphics processing unit, the convolution operation is converted into a task format suitable for execution of a kernel function of the graphics processing unit, and packaged into a task unit that can be recognized by the graphics processing unit; for a neural network processing unit, the convolution layer calculation task is compiled into a corresponding machine code or microinstruction sequence according to the instruction set and the calculation unit architecture of the neural network processing unit, to form a task unit adapted to the neural network processing unit.

[0044] Then, the packaged task units are distributed to each hardware device in the heterogeneous hardware. On each hardware device, different device load states are generated by adjusting the simulated load, for example, for a central processing unit, its CPU occupancy is adjusted to be at low, medium, and high levels by running background calculation tasks; for a graphics processing unit and a neural network processing unit, their load states are similarly adjusted. In each device load state, the calculation task unit of the layer at the current bit number selection is executed.

[0045] During the task execution process, multi-dimensional performance data is collected using performance monitoring tools and power consumption measuring instruments. The calculation time is the time experienced from the start of the execution of the task unit to the completion of the calculation and the output of the result; the power consumption is the average power consumption value of the device during the execution of the task; the data transmission delay is the sum of the time required for the input data of the task unit to be transmitted from the storage location to the calculation unit and the time required for the calculation result to be transmitted from the calculation unit to the storage location; the performance stability parameter is measured by repeatedly executing the same task multiple times, and calculating the standard deviation or coefficient of variation of the calculation time, power consumption, and other indicators; the smaller the standard deviation or coefficient of variation, the higher the performance stability.

[0046] Each round of performance data collection is performed in a specific device load state, and for each bit number selection of each layer, multiple rounds (such as one round each in low, medium, and high load states) of performance data collection are performed on each hardware device. The calculation time, power consumption, data transmission delay, and performance stability parameters collected in multiple rounds are integrated, and after removing outliers, the average value or a certain weight is fused to generate a multi-dimensional performance data set of the layer at the bit number selection on each hardware device.

[0047] Step S123: Based on the multi-dimensional performance data set of all layers, the layer calculation feature hardware dynamic characteristic performance mapping relationship is constructed: first, the feature correlation is performed between the basic calculation operation type in the hierarchical calculation feature information, the high-order data interaction mode and the static calculation ability, the dynamic resource occupation characteristic in the dynamic hardware parameter information, the associated features are obtained, and then the mapping function of the associated features and the multi-dimensional performance data is established to generate an initial offline performance model.

[0048] Based on the multi-dimensional performance data set of all layers (input layer, each convolution layer, pooling layer, fully connected layer, output layer, etc.) under different low-bit calculation precision and different hardware device load states, the layer calculation feature hardware dynamic characteristic performance mapping relationship is constructed.

[0049] First, the feature correlation is performed. The basic calculation operation type (such as convolution operation, full connection operation, etc.) and the high-order data interaction mode (including data interaction frequency, data multiplexing rate, interaction data feature dimension, etc.) of each layer are extracted from the hierarchical calculation feature information; the static calculation ability (floating point operation per second, parallel computing thread number, memory basic capacity, etc.) and the dynamic resource occupation characteristic (load occupancy rate correlation curve, load delay correlation curve, load power consumption correlation curve) of each hardware device are extracted from the dynamic hardware parameter information. The above features are combined, for example, the convolution operation type, the data interaction frequency, and the data multiplexing rate of a certain convolution layer are associated with the floating point operation per second, the parallel computing thread number, and the load occupancy rate correlation curve of the central processing unit, and the associated feature vector describing the combination of the layer calculation feature and the hardware dynamic characteristic is formed.

[0050] Then, the mapping function of the associated features and the multi-dimensional performance data is established. The multi-dimensional performance data includes calculation time consumption, power consumption, data transmission delay, and performance stability. Machine learning methods (such as multiple linear regression, decision tree, neural network, etc.) are used to construct the mapping function. The associated feature vector is taken as the input, and the multi-dimensional performance data is taken as the output. The model is trained through the training sample (i.e., the associated features of each layer and the corresponding multi-dimensional performance data) to determine the parameters of the mapping function. For example, if a neural network is used as the mapping function model, the number of input layer neurons corresponds to the dimension of the associated feature vector, the number of output layer neurons corresponds to the number of indicators of the multi-dimensional performance data (such as 4 output neurons corresponding to 4 indicators), and the network weight and bias are adjusted through the back propagation algorithm, so that the model can accurately predict the multi-dimensional performance data according to the input associated features. The mapping function obtained through the training is the initial offline performance model, and the initial offline performance model can predict the layer calculation performance based on the layer calculation feature and the hardware dynamic characteristic.

[0051] Step S124: performing multi-round iterative optimization on the initial offline performance model: the first round of optimization adjusts the weight coefficients of the mapping function using the gradient descent algorithm to reduce the deviation of the model prediction value from the actual multidimensional performance data; the second round of optimization introduces a time correlation factor to integrate the load fluctuation trend of the hardware device into the model to correct the performance prediction results under different load states; the third round of optimization adjusts the inter-layer performance correlation parameters of the model based on the inter-layer dependency weight to ensure the consistency of the adjacent layer performance prediction, and generates the offline performance model after multi-round optimization.

[0052] The initial offline performance model is subjected to multi-round iterative optimization to improve its prediction accuracy and reliability.

[0053] The first round of optimization uses the gradient descent algorithm. The actual multidimensional performance data is compared with the prediction value of the initial offline performance model, and the deviation (such as mean square error) between the two is calculated. The deviation is taken as the optimization objective function, and the mapping function (such as the weight coefficients in the neural network model, the split threshold in the decision tree model, etc.) is adjusted by the gradient descent algorithm. In each iteration, the partial derivative (gradient) of the objective function with respect to each weight coefficient is calculated, and then the weight coefficient is updated in the opposite direction of the gradient to gradually reduce the deviation of the model prediction value from the actual performance data. After multiple iterations, until the deviation converges to a preset threshold range or reaches the maximum number of iterations, the first round of optimization is completed.

[0054] The second round of optimization introduces a time correlation factor to integrate the load fluctuation trend of the hardware device into the model. The load state of the hardware device changes over time, and the load state at different time points will affect the processing performance of the network layer computing task. The time correlation factor is constructed according to the load fluctuation trend of the hardware device (such as the load curve changing over time extracted from the multi-level basic information set), which is used to represent the correlation between load states at different times. The time correlation factor is added to the offline performance model as an additional input feature or model adjustment parameter, for example, when the model is predicting, not only the current time hardware dynamic characteristics are considered, but also the load state information of the previous time or multiple historical times, the prediction result is corrected through the time correlation factor. For example, if the load fluctuation trend of the hardware device shows that its load will show an upward trend in the future, then when predicting the performance at the next time, the prediction value of data transmission delay and computing time is appropriately increased to reflect the negative impact that the increase in load may bring, so as to correct the performance prediction results under different load states.

[0055] The third round of optimization is based on the inter-layer dependency weight adjustment model of the inter-layer performance correlation parameter. There is a data dependency relationship between each layer of the neural network, and the computing performance of adjacent layers affects each other. For example, if the computing time of the previous layer is longer, the next layer needs to wait for the input data, which affects the overall computing efficiency. The inter-layer dependency weight (obtained from the hierarchical feature information calculation) reflects the strength of the dependency relationship between adjacent layers. In the offline performance model, the inter-layer performance correlation parameter is used to control the mutual influence between the performance prediction results of adjacent layers. According to the size of the inter-layer dependency weight, the values of these parameters are adjusted for adjacent layers with larger inter-layer dependency weights. The performance correlation parameter of the adjacent layer is increased, so that the model considers the performance prediction results of adjacent layers more when predicting the performance of a certain layer, thereby ensuring the consistency of the performance prediction of adjacent layers. For example, if the inter-layer dependency weight between a convolution layer and the next pooling layer is large, the predicted performance of the convolution layer will be an important reference factor when predicting the performance of the pooling layer. If the predicted time consumption of the convolution layer increases, the predicted time consumption of the pooling layer should also be adjusted accordingly to maintain the consistency of the performance change trend of the two layers. After the three rounds of optimization, the multi-round optimized offline performance model is generated.

[0056] Step S125: Based on the multi-round optimized offline performance model, the layers are split and associated according to the layer identifier and the hardware device identifier, and a corresponding device precision performance sub-model is generated for each layer. Then, all the sub-models of the layers are associated through the inter-layer dependency weight to form a hierarchical performance correlation model. Each sub-model entry in the hierarchical performance correlation model includes a layer identifier, a bit selection, a hardware device identifier, a dynamic load state, corresponding multi-dimensional performance data, and an inter-layer correlation weight.

[0057] The multi-round optimized offline performance model is a whole model that can predict the performance of all network layers under different conditions. In order to facilitate fast query and application during subsequent online dynamic deployment, the whole model needs to be split and associated hierarchically.

[0058] The whole offline performance model is split into sub-model parts corresponding to each network layer according to the layer identifier (such as the unique number or name of each layer). At the same time, for each layer sub-model part, it is further split according to the hardware device identifier (the identifier of the central processing unit, graphics processing unit, and neural network processing unit) to obtain the performance model of each layer on different hardware devices. Then, combined with different bit selection and dynamic load state, a corresponding device precision performance sub-model is generated for each layer. Each device precision performance sub-model specifically describes the multi-dimensional performance data (computing time consumption, power consumption, data transmission delay, and performance stability) of a specific layer under a specific bit selection on a specific hardware device under different dynamic load states.

[0059] After generating the device precision performance sub-models of each layer, all the sub-models of the layers are associated and integrated through the inter-layer dependency weights. The inter-layer dependency weights reflect the dependency relationship between the layers, and connect the device precision performance sub-models of adjacent layers through the corresponding inter-layer dependency weights to form an overall hierarchical performance correlation model. In the hierarchical performance correlation model, each sub-model entry contains detailed information: the layer identifier clearly indicates the network layer to which the sub-model corresponds; the bit number selection indicates the low-bit calculation precision used; the hardware device identifier indicates the corresponding hardware device; the dynamic load state describes the load of the device under the performance prediction; the corresponding multi-dimensional performance data is the core result of the model prediction; and the inter-layer correlation weight reflects the correlation strength between the sub-model and the sub-model of the adjacent layer. Through the above method, the hierarchical performance correlation model can describe the time sequence correlation relationship between the calculation precision, the dynamic characteristics of the hardware device, and the layer calculation performance.

[0060] Step S130: In the online running process, the real-time load state of each hardware device in the heterogeneous hardware and the dynamically adjusted energy efficiency constraint condition are tracked to generate a dynamic state monitoring result. The real-time load state includes device resource occupancy, calculation response delay, and load fluctuation rate. The dynamically adjusted energy efficiency constraint condition includes device-level power consumption upper limit, system-level calculation efficiency lower limit, and load fluctuation adaptation threshold.

[0061] When the neural network model is running online, the state of the heterogeneous hardware devices needs to be tracked in real time, and the monitoring result is generated in combination with the energy efficiency constraint condition. Still taking the online inference process of the image recognition convolutional neural network on the heterogeneous hardware platform including the central processing unit, the graphics processing unit, and the neural network processing unit as an example. During the online running process, the load state of the hardware devices changes in real time, and the energy efficiency constraint condition may also be dynamically adjusted according to the overall system running situation, so it is necessary to continuously monitor and generate a dynamic state monitoring result.

[0062] Step S131: A load tracking module is deployed for each hardware device in the heterogeneous hardware. The load tracking module includes a real-time data acquisition unit, a fluctuation trend analysis unit, and a constraint adaptation unit. The real-time data acquisition unit is used to acquire device parameters at a dynamic time interval. The fluctuation trend analysis unit is used to analyze the parameter change law. The constraint adaptation unit is used to adjust the energy efficiency constraint parameter.

[0063] Load tracking modules are deployed on the central processing unit, the graphics processing unit, and the neural network processing unit of the heterogeneous hardware platform. The load tracking module can be a software component running in the device driver layer or the system kernel layer, or a combination of a special monitoring circuit integrated in the hardware device and a matching software.

[0064] The load tracking module includes three main units. The real-time data collection unit is responsible for collecting various parameters of the device at a dynamic time interval, which can be flexibly adjusted according to the current load state of the device to reduce unnecessary performance overhead while ensuring monitoring accuracy; the fluctuation trend analysis unit analyzes the collected parameter data in real time to identify its change law and trend over time; and the constraint adaptation unit dynamically adjusts the energy efficiency constraint parameters of the device according to the load state and fluctuation trend of the device, as well as the overall energy efficiency target of the system.

[0065] Step S132: Set a dynamic collection time interval based on the load fluctuation trend of the hardware device, increase the collection interval when the load fluctuation rate is lower than a set first threshold, and decrease the collection interval when the load fluctuation rate is higher than a set second threshold. Collect the current resource occupancy rate, calculation response delay, and instantaneous power consumption of each hardware device. The resource occupancy rate includes memory occupancy rate, calculation unit occupancy rate, and bandwidth occupancy rate.

[0066] The load fluctuation trend is obtained from the dynamic hardware parameter information in the multi-level basic information set, which reflects the speed of load change of the hardware device in the historical running process. Based on this trend, the dynamic collection time interval of the real-time data collection unit is set. Two thresholds, a first threshold and a second threshold (the second threshold is greater than the first threshold), are set to determine the high and low of the current load fluctuation rate.

[0067] When the load fluctuation rate (the magnitude of load change per unit time) of the hardware device monitored by the fluctuation trend analysis unit is lower than the set first threshold, it indicates that the device load state is relatively stable, and at this time the data collection interval is increased, for example, from the original 10 milliseconds to 50 milliseconds, to reduce the impact of data collection on device performance. When the load fluctuation rate is higher than the set second threshold, it indicates that the device load changes dramatically, and more frequent data collection is needed to accurately capture its state changes, so the collection interval is reduced, for example, from 10 milliseconds to 2 milliseconds.

[0068] In each collection time interval, the real-time data collection unit collects the current resource occupancy rate, calculation response delay, and instantaneous power consumption of the hardware device. The resource occupancy rate specifically includes memory occupancy rate (the percentage of the current memory capacity used by the device to the total memory capacity), calculation unit occupancy rate (such as CPU core occupancy rate of central processing unit, stream processor utilization rate of graphics processing unit, operation unit usage rate of neural network processing unit, etc.), and bandwidth occupancy rate (data transmission bandwidth utilization rate of internal data bus or external interface of the device). The calculation response delay is obtained by recording the time interval from the start of execution to the completion of a plurality of recently submitted calculation tasks and taking the average. The instantaneous power consumption is read in real time by a hardware sensor or a dedicated power consumption monitoring tool.

[0069] Step S133: Through the fluctuation trend analysis unit, the collected resource occupancy, computing response delay and instantaneous power consumption are analyzed in a set time window, the change rate of each parameter in unit time is calculated, and the change direction is counted. The change rate and change direction of each parameter are integrated to evaluate the load fluctuation rate of each hardware device and judge the load fluctuation direction, which includes rising trend, falling trend and stable trend.

[0070] The fluctuation trend analysis unit analyzes the received real-time acquisition data (resource occupancy, computing response delay and instantaneous power consumption). First, a time window is set, such as a time range of the past 5 seconds or 10 seconds, and the parameter data collected in the time window is arranged in time sequence.

[0071] For each parameter (resource occupancy, computing response delay, instantaneous power consumption), the change rate is calculated in the time window. The calculation method of the change rate is to subtract the parameter value at the first sampling time from the parameter value at the last sampling time in the time window, and then divide by the length of the time window (unit time) to obtain the average change amount of the parameter in unit time, that is, the change rate. At the same time, the change direction of each parameter is counted. If the change rate is positive, the change direction is rising; if the change rate is negative, the change direction is falling; if the absolute value of the change rate is less than a very small threshold (which can be considered as approximately zero), the change direction is stable.

[0072] Then, the change rate and change direction of each parameter are integrated to evaluate the load fluctuation rate. According to the importance of each parameter to the device load state, different weights are assigned to resource occupancy, computing response delay and instantaneous power consumption (such as resource occupancy weight 0.5, computing response delay weight 0.3, and instantaneous power consumption weight 0.2). Multiply the change rate of each parameter by its corresponding weight, then add the weighted change rates to get the integrated change rate. The absolute value of the integrated change rate is taken as the load fluctuation rate of the hardware device.

[0073] Finally, the load fluctuation direction is judged. If the integrated change rate is positive, the load fluctuation direction is rising trend, indicating that the load of the device is gradually increasing; if the integrated change rate is negative, it is falling trend, and the load is gradually decreasing; if the absolute value of the integrated change rate is less than the set stable threshold, it is stable trend, and the load state remains basically unchanged.

[0074] Step S134: Obtain initial energy efficiency constraints by the constraint adaptation unit, and dynamically adjust the constraint parameters in combination with the load fluctuation trend and the current load state of each hardware device: increase the adjustment threshold of the device-level power consumption upper limit for devices with an upward load trend, and decrease the adjustment threshold of the system-level computing efficiency lower limit for devices with a downward load trend, to generate dynamically adjusted energy efficiency constraints.

[0075] The constraint adaptation unit first obtains initial energy efficiency constraints, which are usually pre-set by the system administrator according to the specifications of the hardware devices, the requirements of the application scenarios (such as the real-time requirements of image recognition, the endurance requirements of the system, etc.), and the energy efficiency targets, including the device-level power consumption upper limit (the maximum power consumption value allowed for each hardware device during operation), the system-level computing efficiency lower limit (the minimum efficiency requirement for the entire heterogeneous hardware system to perform computing tasks, such as the number of inference tasks completed per second), and the load fluctuation adaptation threshold (the threshold allowing the load of the hardware device to fluctuate within a certain range).

[0076] In combination with the load fluctuation trend (historical trend obtained from the multi-level basic information set) and the current load state (current load fluctuation rate and fluctuation direction obtained by the fluctuation trend analysis unit) of each hardware device, these constraint parameters are dynamically adjusted. For hardware devices with an upward load trend, in order to avoid the system from being unstable or overheating due to the rapid increase of power consumption exceeding the upper limit, the adjustment threshold of the device-level power consumption upper limit is increased, i.e., the device-level power consumption upper limit is allowed to be dynamically adjusted within a larger range, for example, the original adjustment threshold range is ±10% of the basic power consumption upper limit, which is now increased to ±15%, so that the power consumption limit can be appropriately relaxed at the initial stage of load increase to ensure the smooth execution of computing tasks, while strict limitation is imposed when approaching the power consumption upper limit. For devices with a downward load trend, their computing resources are relatively abundant, and the adjustment threshold of the system-level computing efficiency lower limit can be reduced at this time, i.e., the adjustable range of the system-level computing efficiency lower limit is reduced, for example, from the original ±5% of the basic efficiency lower limit to ±3%, to more strictly require devices to maintain high computing efficiency in low load state, avoiding resource waste. Through the above adjustment, dynamically adjusted energy efficiency constraints are generated.

[0077] Step S135: Integrate the resource occupancy rate, computing response delay, instantaneous power consumption, load fluctuation rate, load fluctuation direction of each hardware device, and the dynamically adjusted energy efficiency constraints, and establish an association mapping according to the device identifier to generate a dynamic state monitoring result, wherein each device entry in the dynamic state monitoring result includes a device identifier, a real-time load parameter set, a load fluctuation analysis result, and corresponding dynamic energy efficiency constraint parameters.

[0078] The information obtained by each hardware device through the above steps is integrated. The real-time load parameter set includes resource occupancy (memory occupancy, computing unit occupancy, bandwidth occupancy), computing response delay, and instantaneous power consumption; the load fluctuation analysis result contains load fluctuation rate and load fluctuation direction; the dynamic energy efficiency constraint parameter is the device-level power consumption upper limit, the system-level computing efficiency lower limit, and the load fluctuation adaptation threshold adjusted by the constraint adaptation unit.

[0079] The above information is associated and mapped according to the device identifier (such as "CPU0", "GPU0", "NPU0"), ensuring that each item of information accurately corresponds to the corresponding hardware device. For example, the real-time load parameters, load fluctuation analysis results, and dynamic energy efficiency constraint parameters of the central processor are integrated into the entry identified by "CPU0". After integration, a dynamic state monitoring result is generated, which comprehensively reflects the current state and energy efficiency constraint of each hardware device in the heterogeneous hardware during online operation.

[0080] Step S140: Combining the hierarchical performance correlation model and the dynamic state monitoring result, a multi-objective optimization algorithm is used to dynamically select an adaptive low-bit computing precision and a corresponding hardware device for each layer of the neural network, forming a hierarchical deployment configuration scheme. The hierarchical deployment configuration scheme includes the precision selection result of each layer, the device allocation result, the inter-layer data transmission weight, and the dynamic adjustment trigger condition.

[0081] After obtaining the hierarchical performance correlation model and the dynamic state monitoring result, the two need to be combined, and a multi-objective optimization algorithm is used to select a suitable low-bit computing precision and hardware device for each layer of the neural network. Taking online inference of an image recognition convolutional neural network as an example, the computing characteristics and hardware resource requirements of each network layer (such as convolutional layer, pooling layer, fully connected layer, etc.) are different, and the real-time state of the hardware devices (central processor, graphics processing unit, neural network processing unit) is also different, so dynamic selection is needed to achieve the best overall deployment effect.

[0082] Step S141: Extract the real-time load parameter set, load fluctuation analysis result, and dynamic energy efficiency constraint parameter of each hardware device from the dynamic state monitoring result, filter out the hardware devices whose current load meets the constraint requirement based on the dynamic energy efficiency constraint parameter, and then filter out the devices whose load fluctuation rate is lower than the set threshold based on the load fluctuation trend, to generate a dynamic available hardware device list. The dynamic available hardware device list contains device identifier, real-time load summary, and load stability rating.

[0083] From the dynamic state monitoring results, the real-time load parameter set (resource occupancy rate, calculation response delay, instantaneous power consumption) of the central processor, the graphics processor and the neural network processor, the load fluctuation analysis results (load fluctuation rate, load fluctuation direction) and the dynamic energy efficiency constraint parameters (device-level power consumption upper limit, system-level calculation efficiency lower limit, load fluctuation adaptation threshold) of the three hardware devices are extracted.

[0084] Firstly, the hardware devices are screened based on the dynamic energy efficiency constraint parameters. For each device, it is checked whether the current real-time load parameter meets the requirements of the dynamic energy efficiency constraint parameters. For example, it is judged whether the instantaneous power consumption of the device is lower than the device-level power consumption upper limit, and whether the calculation response delay is within an acceptable range (indirectly reflecting whether the calculation efficiency meets the system-level calculation efficiency lower limit requirement). The hardware devices whose current load meets these constraint requirements are screened out, and those devices that have exceeded the constraint conditions (such as excessive power consumption, excessive response delay) are excluded.

[0085] Then, further screening is combined with the load fluctuation trend (historical trend obtained from the multi-level basic information set and the current load fluctuation rate in the dynamic state monitoring results). A load fluctuation rate threshold is set, which is determined according to the system's requirement for stability. From the preliminarily screened devices, the devices with a load fluctuation rate lower than the set threshold are selected, and these devices have a relatively stable load state.

[0086] After two rounds of screening, the remaining hardware devices constitute a dynamic available hardware device list. In this list, each device entry contains device identification (such as "CPU0", "GPU0", "NPU0"), real-time load summary (a brief summary of the real-time load parameter set, such as "high load", "medium load", "low load", or specific numerical ranges of key parameters), and load stability rating (rated according to the load fluctuation rate and fluctuation direction, such as "excellent", "good", "general", the lower the fluctuation rate and the more stable the direction, the higher the rating).

[0087] Step S142: Extract the layer accuracy performance sub-model corresponding to each device in the dynamic available hardware device list from the hierarchical performance correlation model, split by layer identifier to obtain multi-dimensional performance prediction data of each layer under different bit selection on each available device, integrate to generate layer accuracy dynamic device performance data set, the multi-dimensional performance prediction data includes predicted calculation time consumption, predicted power consumption, predicted data transmission delay and predicted performance stability.

[0088] The hierarchical performance correlation model contains sub-model entries corresponding to each network layer and each hardware device under different bit selection. According to the device identification in the dynamic available hardware device list, all layer precision performance sub-models corresponding to the available devices are extracted from the hierarchical performance correlation model. For example, if the dynamic available hardware device list contains a central processor and a graphics processor, all sub-models corresponding to the central processor and the graphics processor under different bit selection of each network layer are extracted.

[0089] The extracted layer precision performance sub-models are split by layer identification to obtain multi-dimensional performance prediction data of each network layer (such as the input layer, convolution layer 1, pooling layer 1, convolution layer 2, fully connected layer, etc.) under different bit selection (such as 4 bits, 8 bits, 16 bits) on each available hardware device (such as the central processor, the graphics processor). The multi-dimensional performance prediction data is obtained by the hierarchical performance correlation model according to the dynamic characteristics of the current available device (real-time load state information obtained from the dynamic state monitoring result), including predicted computation time (time required for the layer to complete computation under the current device and bit selection), predicted power consumption (predicted power consumption during execution of the computing task), predicted data transmission delay (predicted delay of layer input and output data transmission on the device), and predicted performance stability (stability level or parameter evaluated by the fluctuation of the predicted computation time, power consumption, etc.).

[0090] All multi-dimensional performance prediction data of all layers are classified and sorted by layer identification, bit selection, and hardware device identification, integrated into a unified data structure, and a layer precision dynamic device performance dataset is generated.

[0091] Step S143: For each layer of the neural network, based on the layer precision dynamic device performance dataset and the hierarchical computation feature information of the layer, a multi-objective optimization function is constructed: the first objective function is to reduce the predicted power consumption, the second objective function is to shorten the predicted computation time, the third objective function is to improve the predicted performance stability, and the high-order data interaction mode and the inter-layer dependency weight of the layer are converted into constraint conditions and integrated into the function, the constraint conditions include the transmission delay upper limit determined by the data interaction frequency and the range requirement of the performance deviation with adjacent layers determined by the inter-layer dependency weight.

[0092] For each layer (such as convolution layer 1) in the neural network, based on the multi-dimensional performance prediction data of the layer under different bit selection and available hardware devices in the layer precision dynamic device performance dataset, and the high-order data interaction mode (data interaction frequency, data multiplexing rate, interaction data feature dimension) and the inter-layer dependency weight (dependence relationship strength with adjacent layers) of the layer obtained from the hierarchical computation feature information, a multi-objective optimization function is constructed.

[0093] The multi-objective optimization function includes three objective functions. The first objective function aims to reduce the predicted power consumption, i.e., the selected bit number and hardware device combination should make the predicted power consumption of the layer as small as possible; the second objective function aims to shorten the predicted computation time, i.e., the computation task of the layer should be completed as soon as possible; and the third objective function aims to improve the predicted performance stability, i.e., the computation performance of the layer should be kept stable during the running process and the fluctuation should be reduced.

[0094] Meanwhile, the high-order data interaction mode of the layer and the inter-layer dependency weight are converted into constraint conditions and integrated into the function. The data interaction frequency in the high-order data interaction mode reflects the frequency of data transmission between the layer and the adjacent layer. The higher the data interaction frequency, the more strict the requirement for the data transmission delay. Therefore, the upper limit of the transmission delay is determined by the data interaction frequency as the first constraint condition, and the predicted data transmission delay of the selected combination cannot exceed the upper limit. The inter-layer dependency weight represents the correlation degree of the computation performance between the layer and the adjacent layer. The greater the weight, the greater the influence of the performance of the adjacent layer on the layer. Therefore, the range requirement of the performance deviation from the adjacent layer is determined by the inter-layer dependency weight as the second constraint condition, i.e., the deviation of the predicted computation time, power consumption and other performance indicators of the layer from the corresponding indicators of the adjacent layer should be within the specified range.

[0095] Step S1431: Extract the predicted computation time, predicted power consumption and predicted performance stability data of the current analysis layer at each available hardware device under all bit number selections from the layer precision dynamic device performance data set, and establish a three-dimensional data matrix according to the bit number selection and device identifier.

[0096] For the network layer (such as convolutional layer 1) currently being analyzed, the predicted computation time, predicted power consumption and predicted performance stability data of the layer at each available hardware device (such as a central processing unit, a graphics processing unit) under all bit number selections (such as 4 bits, 8 bits, 16 bits) are extracted from the layer precision dynamic device performance data set.

[0097] A three-dimensional data matrix is established according to the bit number selection and device identifier. The rows of the three-dimensional data matrix represent different bit number selections, the columns represent different available hardware devices, and each element in the three-dimensional data matrix contains the predicted computation time, predicted power consumption and predicted performance stability of the corresponding bit number and device combination. For example, for the combination of bit number selection of 4 bits and hardware device of central processing unit, the predicted computation time value, predicted power consumption value and predicted performance stability value of convolutional layer 1 under the combination are stored in the matrix element. By establishing a three-dimensional data matrix, the performance prediction data of the current analysis layer under different precisions and devices can be organized and presented, which facilitates the construction and solution of the subsequent objective function.

[0098] Step S1432: Extract high-order data interaction mode and inter-layer dependency weight from the hierarchical calculation feature information of the layer, analyze the data interaction frequency, data reuse rate and interaction data feature dimension in the high-order data interaction mode, determine the maximum allowed value of data transmission delay, and analyze the adjacent layer correlation coefficient in the inter-layer dependency weight to determine the maximum allowed range of performance deviation between the layer and the adjacent layer.

[0099] From the hierarchical calculation feature information of the current analysis layer, high-order data interaction mode and inter-layer dependency weight are extracted. The high-order data interaction mode contains information such as data interaction frequency, data reuse rate and interaction data feature dimension. The data interaction frequency refers to the number of data transmissions between the layer and the adjacent layer per unit time; the data reuse rate is the proportion of the output data of the layer that is reused by the next layer; and the interaction data feature dimension includes dimension information such as the height, width and channel number of the data.

[0100] According to the data interaction frequency, the maximum allowed value of data transmission delay is determined. The higher the data interaction frequency, the greater the impact of data transmission on overall performance, so a smaller maximum allowed value of transmission delay needs to be set. For example, if the data interaction frequency is very high (such as transmitting data once every millisecond), the maximum allowed value of transmission delay may be set to a small value (such as a few microseconds); if the data interaction frequency is lower, the transmission delay limit can be appropriately relaxed. Data reuse rate and interaction data feature dimension also affect the requirements for transmission delay. A high data reuse rate means that the data transmitted once can be used multiple times, so the sensitivity to single transmission delay is relatively reduced; a large interaction data feature dimension means a large data volume, so the time required to transmit the same data volume is longer, and therefore more stringent delay limits may be required. Taking these factors into consideration, the maximum allowed value of data transmission delay is determined.

[0101] The adjacent layer correlation coefficient in the inter-layer dependency weight is analyzed. The adjacent layer correlation coefficient reflects the degree of correlation between the current layer and the adjacent layers (the previous layer and the next layer) in the calculation process. The larger the adjacent layer correlation coefficient, the greater the impact of the performance of the adjacent layers on the calculation performance of the current layer, so the range of performance deviation between the layer and the adjacent layers needs to be strictly controlled. For example, if the adjacent layer correlation coefficient between the current layer and the next layer is 0.8 (assuming the correlation coefficient range is 0-1), the predicted calculation time deviation between the layer and the next layer is set to not more than 10%, the predicted power consumption deviation is set to not more than 15%, and so on, thereby determining the maximum allowed range of performance deviation between the layer and the adjacent layers.

[0102] Step S1433: Taking the minimization of predicted power consumption as the first objective function, the predicted power consumption data is taken as the function input variable, the function output value is set to increase with the decrease of power consumption, and according to the dynamic resource occupation characteristics of the hardware device, a higher weight coefficient is given to the power consumption data of the device with a load fluctuation rate lower than the set threshold.

[0103] The first objective function aims to minimize the predicted power consumption. The predicted power consumption data of the current analysis layer obtained from the three-dimensional data matrix under different bit number selections and hardware device combinations are taken as the input variables of the objective function.

[0104] The function output value is set to increase with the decrease of power consumption, that is, the smaller the predicted power consumption value, the larger the function value of the first objective function, so that in the subsequent optimization solution, the solution with larger function value, that is, the combination with smaller power consumption, can be preferentially selected.

[0105] At the same time, according to the dynamic resource occupation characteristics of the hardware device (obtained from the dynamic hardware parameter information in the multi-level basic information set), the power consumption data corresponding to the device with a load fluctuation rate lower than a set threshold is given a higher weight coefficient. The load occupation rate correlation curve and other information in the dynamic resource occupation characteristics of the hardware device can reflect the load fluctuation of the device, and the device with small load fluctuation has more stable performance in the running process, and the credibility and reference value of its power consumption data are higher. Therefore, when calculating the first objective function value, the predicted power consumption data from the device with small load fluctuation is multiplied by a weight coefficient greater than 1, and the power consumption data of the device with large load fluctuation is multiplied by a weight coefficient less than 1, so as to highlight the importance of the load stable device in the power consumption optimization target.

[0106] Step S1434: taking the shortest prediction calculation time as the second objective function, taking the prediction calculation time data as the function input variable, setting the function output value to increase with the shortening of the calculation time, and adjusting the weight in combination with the basic calculation operation type of the layer to give higher weight to the time data of the calculation-intensive layer.

[0107] The second objective function aims to achieve the shortest prediction calculation time. The predicted calculation time data in the three-dimensional data matrix is taken as the input variable of the function.

[0108] The function output value is set to increase with the shortening of the calculation time, that is, the shorter the predicted calculation time, the larger the function value of the second objective function, so as to tend to select the precision and device combination with faster calculation speed in the optimization process.

[0109] The weight is adjusted in combination with the basic computing operation type of the layer (obtained from the hierarchical computing feature information). The basic computing operation type includes convolution operation, full connection operation, activation function operation, etc. Different operation types have different requirements for computing resources and time-consuming characteristics, and the computing time-consuming of computing-intensive operations (such as convolution operation and full connection operation) accounts for a large proportion of the entire network execution time, and has a greater impact on the overall performance. Therefore, for computing-intensive layers (such as convolution layers), a higher weight is given to the predicted computing time-consuming data of the layers, that is, multiplied by a larger weight coefficient; for non-computing-intensive layers (such as some simple activation function layers), a lower weight coefficient is given. Through the above method, the second objective function focuses more on optimizing the computing time-consuming of the layers that have a greater impact on the overall performance.

[0110] Step S1435: Taking the maximum predicted performance stability as the third objective function, taking the predicted performance stability data as the input variable of the function, setting the output value of the function to increase with the increase of the stability coefficient, and associating the inter-layer dependency weight of the layer, the stability data of the layer with high dependency is given a higher weight.

[0111] The third objective function aims to improve the predicted performance stability, that is, to maximize the predicted performance stability. The predicted performance stability data in the three-dimensional data matrix is taken as the input variable of the function. The predicted performance stability data can be a coefficient representing the stability degree (such as calculated based on the fluctuation of the predicted computing time, power consumption, etc. The larger the coefficient, the better the stability).

[0112] The output value of the function is set to increase with the increase of the stability coefficient, that is, the larger the predicted performance stability coefficient, the larger the function value of the third objective function, so that the combination with better stability is preferentially selected during optimization.

[0113] The inter-layer dependency weight of the layer (obtained from the hierarchical computing feature information) is associated, and the stability data of the layer with high dependency is given a higher weight. The larger the inter-layer dependency weight, the closer the dependency relationship between the layer and the adjacent layer, and the greater the influence of the performance stability on the stable operation of the adjacent layer and even the entire network. Therefore, for the layer with a larger inter-layer dependency weight (i.e., the layer with high dependency), a larger weight coefficient is multiplied to the predicted performance stability data of the layer when calculating the third objective function value, so as to emphasize the importance of the stability; for the layer with low dependency, a smaller weight coefficient is given.

[0114] Step S1436: The maximum allowed value of the data transmission delay determined by the high-order data interaction mode is converted into the first constraint condition, requiring that the predicted data transmission delay corresponding to the candidate combination does not exceed the allowed value; and the maximum allowed range of the performance deviation determined by the inter-layer dependency weight is converted into the second constraint condition, requiring that the performance deviation of the candidate combination and the candidate combination of the adjacent layer does not exceed the range.

[0115] The maximum allowed value of the data transmission delay determined in step S1432 is converted into a first constraint condition. In the multi-objective optimization function, any one of the candidate combinations of bit number selection and hardware devices whose corresponding predicted data transmission delay (obtained from the layer-precision dynamic device performance data set) is less than or equal to the maximum allowed value must be considered as a feasible solution, otherwise the candidate combination will be excluded.

[0116] The maximum allowed range of the performance deviation between the layer and the adjacent layer determined in step S1432 is converted into a second constraint condition. For each candidate combination, the deviation between its predicted computation time, predicted power consumption equivalent performance indicators and the corresponding performance indicators of the candidate combinations that can be selected by the adjacent layers in the respective optimization process must be within the maximum allowed range, for example, the difference between the predicted computation time of the current layer and the predicted computation time of the next layer does not exceed a certain percentage (such as 10%) of the predicted computation time of the next layer, so as to ensure the performance coordination between adjacent layers.

[0117] Step S1437: integrating the first target function, the second target function and the third target function with the first constraint condition and the second constraint condition, constructing a multi-objective optimization function, and setting the weight coefficients of each target function based on the priority in the dynamically adjusted energy efficiency constraint condition, to generate the multi-objective optimization function of the layer.

[0118] The first target function, the second target function and the third target function are integrated, and the first constraint condition and the second constraint condition are integrated, to construct the multi-objective optimization function of the current analysis layer.

[0119] The weight coefficients of each target function are set according to the priority in the dynamically adjusted energy efficiency constraint condition. The dynamically adjusted energy efficiency constraint condition may have different priority requirements for reducing power consumption, shortening computation time and improving stability, for example, when running an image recognition model on a battery-powered mobile device, the priority of reducing power consumption can be set as the highest; in a scene with extremely high real-time requirement (such as image recognition in autonomous driving), the priority of shortening computation time can be set as the highest. According to these priorities, the three target functions are assigned corresponding weight coefficients, and the target function with high priority is assigned a large weight coefficient. For example, if the priority of reducing power consumption is the highest, the priority of shortening computation time is the second, and the priority of improving stability is the lowest, the weight coefficient of the first target function is the largest, the weight coefficient of the second target function is the second, and the weight coefficient of the third target function is the smallest.

[0120] The mathematical form of the multi-objective optimization function can be expressed as the weighted sum of the target functions and the corresponding weight coefficients, while being limited by the first constraint condition and the second constraint condition. In the above manner, the complete multi-objective optimization function of the current analysis layer is generated.

[0121] Step S144: solve the multi-objective optimization function by using a non-dominated sorting algorithm to generate a plurality of sets of non-dominated precision device candidate combinations, each candidate combination comprising a bit number selection, a hardware device identifier, and a corresponding calculation result of the first objective function, the second objective function, and the third objective function.

[0122] The non-dominated sorting algorithm is a commonly used multi-objective optimization algorithm, which can find a set of optimal solutions (non-dominated solutions) among multiple conflicting objectives. The multi-objective optimization function of the current analysis layer is input into the non-dominated sorting algorithm for solving.

[0123] Step S1441: all bit number selections of the layer and combinations of available hardware devices are taken as an initial solution set, and each combination corresponds to a set of input parameters of the multi-objective optimization function.

[0124] The initial solution set is composed of all possible combinations of bit number selections and available hardware devices of the layer. For example, if there are three bit number selections of 4 bits, 8 bits, and 16 bits, and two available hardware devices of a central processing unit and a graphics processing unit, the initial solution set contains 3x2=6 combinations, each combination (such as 4 bits-central processing unit, 4 bits-graphics processing unit, 8 bits-central processing unit, etc.) is a set of input parameters of the multi-objective optimization function.

[0125] Step S1442: for each combination in the initial solution set, calculate the function value under the first objective function, the second objective function, and the third objective function to obtain a three-dimensional objective function value vector of each combination.

[0126] For each combination in the initial solution set, the corresponding predicted power consumption, predicted calculation time, and predicted performance stability data are input into the first, second, and third objective functions in the multi-objective optimization function to calculate the function value of each objective function. The three function values form a three-dimensional vector, which is the three-dimensional objective function value vector of the combination. For example, the first objective function value of a certain combination is 0.8 (assuming the range is 0-1, and the larger the value, the lower the power consumption), the second objective function value is 0.7 (the larger the value, the shorter the time), and the third objective function value is 0.6 (the larger the value, the higher the stability), and the three-dimensional objective function value vector is (0.8, 0.7, 0.6).

[0127] Step S1443: perform the first round of non-dominated sorting: iterate through each combination in the initial solution set, compare the three-dimensional objective function value vector of the combination with all other combinations, if there is a combination whose all objective function values are not inferior to the combination, and at least one objective function value is superior to the combination, then the combination is dominated, otherwise the combination is a non-dominated solution, and all non-dominated solutions are classified into the first non-dominated layer.

[0128] The first round of non-dominated sorting finds all non-dominated solutions from the initial solution set. Each combination in the initial solution set is traversed and its three-dimensional objective function value vector is compared with the three-dimensional objective function value vectors of all other combinations.

[0129] If for a combination A, there exists another combination B, the first objective function value of combination B is greater than or equal to the first objective function value of combination A, the second objective function value is greater than or equal to the second objective function value of combination A, the third objective function value is greater than or equal to the third objective function value of combination A, and at least one objective function value is strictly greater than the corresponding objective function value of combination A, then combination A is dominated by combination B, and combination A is not a non-dominated solution. Conversely, if there is no such combination B, then combination A is a non-dominated solution.

[0130] All non-dominated solutions are classified into the first non-dominated layer, which are optimal under the current conditions and cannot be outperformed by other solutions in all objectives.

[0131] Step S1444: Remove the solutions in the first non-dominated layer from the initial solution set, perform the second round of non-dominated sorting on the remaining solutions, repeat the comparison process, and generate the second non-dominated layer; iterate in turn until all solutions are classified into different non-dominated layers.

[0132] After the first round of non-dominated sorting is completed, all solutions in the first non-dominated layer are removed from the initial solution set. The same non-dominated sorting process as the first round is performed on the remaining solutions to find non-dominated solutions among the remaining solutions, which are classified into the second non-dominated layer.

[0133] In the same way, the third, fourth, and subsequent rounds of non-dominated sorting are performed in turn, and each time non-dominated solutions are found from the remaining solutions and classified into the corresponding non-dominated layer until all solutions in the initial solution set are assigned to different non-dominated layers. The smaller the non-dominated layer number, the better the overall performance of the solutions in the layer.

[0134] Step S1445: Calculate the crowding distance of each non-dominated layer: for each solution in each non-dominated layer, sort the solutions in each objective function dimension according to the function value, set the crowding distance of the first and last solutions to infinity, and calculate the sum of the function value differences between adjacent solutions in the objective function dimension for the intermediate solutions as the crowding distance of the solution. The larger the crowding distance, the sparser the distribution of the solution.

[0135] The crowding distance is used to measure the crowding degree between solutions in the same non-dominated layer. The larger the crowding distance, the sparser the distribution of solutions around the solution, indicating that the solution has good diversity in the objective space.

[0136] For all solutions in each non-dominated layer, sort them in ascending or descending order of function value in each objective function dimension (first objective function, second objective function, third objective function) respectively (usually in ascending order). For the solutions at the head and tail of the sorted list, set their crowding distance to infinity to ensure that these boundary solutions will not be eliminated, maintaining the diversity of solutions. For the solutions in the middle, calculate the difference between the function value of the solution and its adjacent solutions (the previous solution and the next solution) in each objective function dimension, and add the differences to obtain the crowding distance of the solution. For example, in the first objective function dimension, the function value of solution C is 0.7, the function value of the previous solution B is 0.6, and the function value of the next solution D is 0.8, then the difference of solution C in this dimension is (0.8-0.6)=0.2; similarly, calculate the differences in the second and third objective function dimensions and add them to obtain the crowding distance of solution C.

[0137] Step S1446: screening candidate combinations based on the hierarchy of non-dominated layers and the crowding distance of solutions in the layers: preferentially selecting non-dominated layers with higher hierarchy, sorting the solutions in the non-dominated layer in descending order of crowding distance, and selecting a plurality of solutions with higher ranking, the number of selected solutions being determined based on the total number of solutions in the layer and the inter-layer dependency complexity, each selected solution corresponding to a set of precision device candidate combinations.

[0138] When screening candidate combinations, first preferentially consider non-dominated layers with higher hierarchy, because the solutions in these layers have better overall performance. For example, preferentially select solutions in the first non-dominated layer, and if the number of solutions in the first non-dominated layer is insufficient or the diversity is insufficient, consider the second non-dominated layer.

[0139] In the selected non-dominated layer, sort the solutions in descending order of crowding distance. Solutions with larger crowding distance have better diversity and can provide more different precision and device combination options. The number of selected solutions is determined based on the total number of solutions in the layer and the inter-layer dependency complexity. When the inter-layer dependency complexity is high (i.e., the dependency relationship between the layer and the adjacent layer is complex), more candidate combinations need to be selected to ensure that the best combination that adapts to the adjacent layer can be found; when the total number of solutions is larger, some solutions can also be selected appropriately. For example, if the first non-dominated layer has 10 solutions and the inter-layer dependency complexity is high, the top 5 solutions are selected.

[0140] Each selected solution corresponds to a set of precision device candidate combinations, i.e., includes bit number selection, hardware device identification, and calculation results of the three objective functions.

[0141] Step S1447: extracting the bit number selection, hardware device identification, and calculation results of the first objective function, the second objective function, and the third objective function corresponding to the selected solution, and generating a plurality of non-dominated precision device candidate combinations, each candidate combination being attached with an objective function value vector and a crowding distance identifier.

[0142] The selected bit number corresponding to the solution selected in step S1446 is selected, the hardware device identifier is extracted, and the calculation results (i.e., the objective function value vector) of the solution under the first objective function, the second objective function, and the third objective function and the crowding distance identifier are recorded. The above information is combined to generate multiple groups of non-dominated precision device candidate combinations. Each group of candidate combinations is accompanied by an objective function value vector and a crowding distance identifier for subsequent inter-layer correlation evaluation.

[0143] Step S145: Based on the inter-layer dependence weight of the layer, the inter-layer correlation of the multiple groups of non-dominated precision device candidate combinations is evaluated, the synergy degree of each group of candidate combinations and the adjacent layer candidate combinations in terms of data transmission delay, power consumption, and computing efficiency is comprehensively evaluated, a comprehensive performance matching degree is obtained, and the candidate combination with the highest performance matching degree is selected as the precision device selection result of the layer. The performance matching degree is a comprehensive measurement value obtained based on the synergy degree evaluation of data transmission delay, power consumption, and computing efficiency.

[0144] The inter-layer dependence weight reflects the correlation closeness between the current layer and the adjacent layer. The greater the weight, the closer the inter-layer correlation, and the higher the requirement for the synergy degree. Based on the inter-layer dependence weight of the layer, the inter-layer correlation of the generated multiple groups of non-dominated precision device candidate combinations is evaluated.

[0145] For each group of candidate combinations of the current layer, the candidate combinations (or the candidate combinations being synchronized) of the adjacent layers (the previous layer and the next layer) are evaluated in terms of synergy degree. The evaluation content includes three aspects of data transmission delay, power consumption, and computing efficiency. The data transmission delay synergy degree evaluates whether the predicted data transmission delay of the candidate combinations of the current layer and the adjacent layers matches, for example, if the current layer selects a high bit number (large data volume) and the data interaction frequency is high, and the predicted data transmission delay of the adjacent layer is large, the synergy degree of the two is low. The power consumption synergy degree evaluates whether the predicted power consumption of the candidate combinations of the current layer and the adjacent layers is within the overall power consumption range acceptable by the system, and whether the power consumption trends of the two are consistent. The computing efficiency synergy degree evaluates whether the predicted computing time of the current layer and the adjacent layers matches to avoid the situation that the computing time of a certain layer is too long and becomes a bottleneck of the overall performance.

[0146] Based on the evaluation results of the synergy degree of the three aspects, a comprehensive performance matching degree is obtained. The performance matching degree is a comprehensive measurement value, and the value range can be 0 to 1. The greater the value, the higher the synergy degree. For example, the evaluation scores of the data transmission delay synergy degree, the power consumption synergy degree, and the computing efficiency synergy degree are 0.8, 0.7, and 0.9, respectively. If the weights of the three are the same, the performance matching degree is (0.8+0.7+0.9) / 3=0.8.

[0147] The candidate combination with the highest performance matching degree is selected as the precision device selection result of the layer. If the candidate combinations of adjacent layers have not been determined (e.g., the network layers are sequentially optimized and solved), the performance matching degrees of multiple candidate combinations of adjacent layers can be calculated respectively, the candidate combination of the current layer with the highest average performance matching degree is selected, or the final evaluation is performed after the selection results of adjacent layers are determined.

[0148] Step S146: Perform inter-layer data dependency mapping on the precision device selection results of all layers, plan data transmission paths between adjacent layers based on the high-order data interaction mode of each layer and the inter-layer dependency weight, determine the bandwidth allocation proportion and data transmission priority of each transmission path, and generate inter-layer transmission configuration information.

[0149] After the precision device selection of all network layers is completed, the selection results of each layer are mapped for inter-layer data dependency. The inter-layer data dependency mapping is based on the inter-layer topological connection relationship of the neural network to determine the data transmission direction and data volume between layers.

[0150] Based on the high-order data interaction mode (including data interaction frequency, data multiplexing rate, and interaction data feature dimension) of each layer and the inter-layer dependency weight, the data transmission paths between adjacent layers are planned. The data transmission paths between layers with high data interaction frequency and large interaction data feature dimension should be planned with sufficient bandwidth and low delay; the transmission paths between adjacent layers with large inter-layer dependency weight should have high reliability and stability.

[0151] The bandwidth allocation proportion of each transmission path is determined. According to the data transmission demand (calculated from the data transmission rate demand of the interaction data feature dimension and the data interaction frequency) of each transmission path and the total available bandwidth resources of the system, a certain proportion of bandwidth is allocated to each transmission path. The path with high data transmission demand is allocated with high bandwidth proportion.

[0152] At the same time, the data transmission priority of each transmission path is determined. The data transmission priority of the transmission path with high data interaction frequency and large inter-layer dependency weight is set to high to ensure that these critical data can be transmitted in priority and reduce the waiting time. For example, the data transmission between convolutional layers and pooling layers usually has high frequency and large dependency, so the transmission priority is set to the highest; while the data transmission priority between some auxiliary layers can be set to be lower.

[0153] The transmission path, bandwidth allocation proportion, and data transmission priority information are integrated to generate inter-layer transmission configuration information.

[0154] Step S147: Integrate the precision device selection results of all layers and the inter-layer transmission configuration information, add dynamic adjustment trigger conditions, and finally form a hierarchical deployment configuration scheme. The dynamic adjustment trigger conditions include hardware device load fluctuation exceeding a threshold, layer calculation performance falling below a threshold, and inter-layer transmission delay rising above a threshold. The hierarchical deployment configuration scheme includes the layer identifier of each layer, the bit selection result, the hardware device identifier, the inter-layer transmission path parameter, the bandwidth allocation ratio, the data transmission priority, and the dynamic adjustment trigger condition.

[0155] Integrate the precision device selection results of all layers (including the layer identifier, the bit selection result, and the hardware device identifier) and the inter-layer transmission configuration information (including the inter-layer transmission path parameter, the bandwidth allocation ratio, and the data transmission priority) to ensure that the selection result of each layer is accurately associated with the corresponding transmission configuration information.

[0156] Add dynamic adjustment trigger conditions, which are used to determine whether the deployment configuration scheme needs to be adjusted during the online running of the neural network. The hardware device load fluctuation exceeding a threshold means that when the load fluctuation rate of a hardware device exceeds a set threshold, the performance of the device may change significantly, triggering an adjustment. The layer calculation performance falling below a threshold means that when the actual calculation performance of a network layer (such as an increase in calculation time consumption or an increase in power consumption) exceeds a certain threshold of the predicted value, an adjustment is triggered. The inter-layer transmission delay rising above a threshold means that when the inter-layer data transmission delay exceeds a certain threshold of the predicted transmission delay, an adjustment is triggered.

[0157] Integrate all these information into a structured document to finally form a hierarchical deployment configuration scheme. The hierarchical deployment configuration scheme describes in detail the deployment details of each network layer and the related configuration of inter-layer data transmission.

[0158] Step S150: Call a lightweight runtime scheduler with task preloading capability to allocate the calculation tasks of each layer of the neural network to the corresponding hardware device according to the hierarchical deployment configuration scheme, synchronously load the inter-layer dependent data, and execute the calculation tasks using the selected low-bit calculation precision, thereby achieving dynamic neural network deployment across hardware devices and obtaining a hierarchical cross-device deployment execution result.

[0159] After obtaining the hierarchical deployment configuration scheme, it needs to be implemented through a runtime scheduler. The lightweight runtime scheduler has task preloading capability and can efficiently manage and schedule calculation tasks across hardware devices. Taking online inference of an image recognition convolutional neural network as an example, the scheduler needs to allocate the calculation tasks of each layer to the corresponding hardware device according to the deployment configuration scheme, coordinate data transmission and calculation execution, and finally complete the dynamic deployment and inference task execution of the entire neural network.

[0160] Step S151: input the hierarchical deployment configuration scheme into a lightweight runtime scheduler with task preloading capability, wherein the lightweight runtime scheduler comprises a hierarchical task management module, a precision dynamic configuration module, a dependent data preloading module, a cross-device transmission coordination module, and a dynamic reallocation module.

[0161] The hierarchical deployment configuration scheme is input into the lightweight runtime scheduler in a specific format (such as JSON, XML, or a custom binary format). The lightweight runtime scheduler is a component running in the system software layer of a heterogeneous hardware platform system, and has the characteristics of being lightweight, occupying less system resources.

[0162] The lightweight runtime scheduler comprises five main modules: the hierarchical task management module is responsible for managing and scheduling the computing tasks of each layer of the neural network; the precision dynamic configuration module is used to configure the low-bit computing precision for each layer computing task according to the deployment configuration scheme; the dependent data preloading module is responsible for preloading inter-layer dependent data to reduce the computing waiting time; the cross-device transmission coordination module coordinates the data transmission between different hardware devices; and the dynamic reallocation module dynamically adjusts the task allocation and precision configuration when the system state changes.

[0163] Step S152: through the hierarchical task management module, parse the layer identifier, hardware device identifier, and dynamic adjustment trigger condition of each layer in the hierarchical deployment configuration scheme, generate a task execution sequence according to the inter-layer dependency weight, encapsulate each layer computing task into a task unit containing layer identifier, computing operation instruction and data requirement list, generate layer device task allocation instruction, and the layer device task allocation instruction is used to indicate sending each task unit to the corresponding hardware device.

[0164] The hierarchical task management module first parses the hierarchical deployment configuration scheme, extracts the layer identifier, hardware device identifier (the hardware device to which the computing task of the layer should be allocated), and dynamic adjustment trigger condition of each layer.

[0165] According to the inter-layer dependency weight of each layer, all layers are sorted to generate a task execution sequence. The inter-layer dependency weight reflects the dependency relationship between layers. The layers that are depended on (preorder layers) should be placed before the dependent layers (postorder layers). For example, the input layer is the preorder layer of all other layers and is placed at the front; the convolution layer is placed before the pooling layer; and the fully connected layer is placed before the output layer. In this way, the order of task execution is ensured to comply with the computing logic of the network.

[0166] The computing task of each layer is encapsulated as a task unit. The task unit contains a layer identifier (used to identify the network layer to which the task unit corresponds), a computing operation instruction (detailed description of the specific computing operation that the layer needs to perform, such as the convolution kernel parameters, step size, padding mode of the convolution operation, weight matrix and bias vector of the fully connected layer, etc.), and a data requirement list (listing the source, data volume, data format, etc. of the input data required by the layer computing task).

[0167] According to the task execution sequence and the hardware device identifier corresponding to each task unit, a layer device task allocation instruction is generated. The layer device task allocation instruction explicitly indicates that each task unit is sent to the corresponding hardware device, for example, the task unit of the convolution layer 1 is sent to the graphics processor, and the task unit of the fully connected layer is sent to the central processor.

[0168] Step S153: Through the precision dynamic configuration module, the layer identifier and bit number selection result of each layer in the hierarchical deployment configuration scheme are parsed, the corresponding computing precision configuration parameter is determined in combination with the basic computing operation type of the layer, the layer precision dynamic configuration instruction is generated, and the layer precision dynamic configuration instruction is used to instruct each hardware device to configure the low-bit computing unit according to the parameter.

[0169] The precision dynamic configuration module parses the hierarchical deployment configuration scheme to obtain the layer identifier and bit number selection result (such as 4 bits, 8 bits, 16 bits) of each layer.

[0170] Step S1531: Extract the layer identifier, bit number selection result and basic computing operation type of all layers from the hierarchical deployment configuration scheme, and establish a layer precision operation association table according to the layer identifier.

[0171] The layer identifier, corresponding bit number selection result and basic computing operation type (such as convolution operation, fully connected operation, activation function operation, etc.) of each layer are extracted from the hierarchical deployment configuration scheme. According to the layer identifier, the above information is associated to establish a layer precision operation association table, which shows the bit number used by each layer and the computing operation type executed.

[0172] Step S1532: For each layer identifier in the layer precision operation association table, determine the numerical representation format based on the bit number selection result.

[0173] For each layer identifier in the layer precision operation association table, the numerical value representation format is determined according to the bit number selection result. The commonly used numerical value representation formats of low-bit calculation precision include fixed-point numbers and floating-point numbers. For example, 4-bit and 8-bit precision usually adopt fixed-point number representation because fixed-point number implementation is simple and the hardware overhead is small, and the decimal point position of the fixed-point number can be pre-set according to the numerical value range and precision requirement of the layer calculation; 16-bit precision can select half-precision floating-point number format to reduce data storage and transmission volume while ensuring a certain precision. The specific numerical value representation format also needs to consider the support situation of the hardware device, for example, some neural network processors may have optimization support for specific fixed-point number format.

[0174] Step S1533: adjusting the operation bit width based on the basic calculation operation type of the layer: adopting an operation bit width matching the bit number for convolution operation, increasing a specified bit width for the full connection operation based on the bit number and the sensitivity of the operation type to error to reduce the cumulative error, and adopting a fixed bit width for the activation function operation to adapt to the nonlinear calculation, to generate an operation bit width parameter.

[0175] The operation bit width refers to the number of bits of the numerical value participating in the calculation. The operation bit width is adjusted based on the basic calculation operation type of the layer.

[0176] For convolution operation, its calculation process is relatively regular, and the requirement for precision can be better met by low-bit number, so an operation bit width matching the bit number selection result is adopted, for example, if the bit number selection is 8 bits, the operation bit width is also 8 bits.

[0177] For full connection operation, since it involves a large number of matrix multiplication and addition operations, errors are easy to accumulate, and the requirement for calculation precision is relatively high. Therefore, on the basis of the bit number selection result, a specified bit width is increased according to the sensitivity of the operation type to error. For example, if the bit number selection is 8 bits, and the full connection layer is sensitive to error, the operation bit width is increased by 2 bits, and 10-bit operation bit width is adopted to reduce the influence of cumulative error on the calculation result. The increased bit width can be realized by low-bit or high-bit extension, and the specific mode is determined according to hardware support and numerical value representation format.

[0178] For activation function operation (such as ReLU, Sigmoid, Tanh, etc.), its nonlinear characteristics make it easy to produce large precision loss at low-bit precision. In order to adapt to nonlinear calculation, a fixed bit width (such as 16 bits) is adopted, which is usually higher than the bit number selection of convolution operation and full connection operation, to ensure the accuracy of the activation function calculation. By generating the above adjusted operation bit width parameter, appropriate operation bit width is configured for each layer calculation task.

[0179] Step S1534: Determine the rounding strategy by combining the bit number selection result and the basic calculation operation type: adopt the rounding rounding strategy for the calculation-intensive operation, adopt the truncation rounding strategy for the data interaction-intensive operation, and adopt the banker rounding strategy for the precision-sensitive operation, and generate the rounding strategy parameter.

[0180] The rounding strategy is used to process the decimal part of the calculation result to adapt to the limited bit number representation. The appropriate rounding strategy is determined by combining the bit number selection result and the basic calculation operation type.

[0181] The calculation-intensive operation (such as the calculation of the convolution layer and the fully connected layer) involves a large number of numerical operations, and the rounding rounding strategy can reduce the accumulation of rounding errors to a certain extent, so the rounding rounding strategy is adopted for such operations, that is, when the decimal part is greater than or equal to 0.5, it is rounded up, and when it is less than 0.5, it is rounded down.

[0182] The data interaction-intensive operation (such as data transmission and storage operation between layers) has a higher requirement for data processing speed, the truncation rounding strategy (directly truncating the decimal part) is simple and fast, although it will introduce a certain error, but for the data interaction-intensive operation, the speed improvement is more important, so the truncation rounding strategy is adopted.

[0183] The precision-sensitive operation (such as some activation function operations or network output layer calculations with extremely high output precision requirements), adopts the banker rounding strategy (rounding to the nearest even number), which can offset the rounding errors in a statistical sense, further improving the numerical precision after rounding. By determining the rounding strategy, the rounding strategy parameter is generated.

[0184] Step S1535: Integrate the numerical representation format, operation bit width parameter and rounding strategy parameter to form the calculation precision configuration parameter of the layer, associate the layer identifier with the calculation precision configuration parameter, and generate the layer precision configuration sub-instruction of the layer.

[0185] Integrate the numerical representation format, operation bit width parameter and rounding strategy parameter to form the complete calculation precision configuration parameter of the current layer. Associate the layer identifier with the calculation precision configuration parameter to determine the network layer corresponding to the configuration parameter, and generate the layer precision configuration sub-instruction of the layer. The layer precision configuration sub-instruction is a precision configuration command for a single layer.

[0186] Step S1536: Time sequence sort the layer precision configuration sub-instructions of all layers, determine the execution order of the instructions based on the inter-layer dependency weight, execute the configuration instructions of the front-dependent layers first, and then execute the configuration instructions of the subsequent layers, and add the execution time sequence identifier to each sub-instruction.

[0187] According to the inter-layer dependency weight of each layer of the neural network, the layer precision configuration sub-instructions of all layers are time-sequentially ordered. The preposed dependency layer with high inter-layer dependency weight needs to complete the precision configuration in priority, because the calculation of the subsequent layer may depend on the calculation result of the preposed layer, and the calculation precision of the preposed layer will affect the format and precision of the output data of the preposed layer. Therefore, the layer precision configuration sub-instruction of the preposed dependency layer is executed first, and then the configuration sub-instruction of the subsequent layer is executed.

[0188] An execution time sequence identifier is added to each layer precision configuration sub-instruction, which indicates the execution order of the sub-instruction in the overall configuration sequence, for example, an increasing serial number is used to represent the execution order.

[0189] Step S1537: For each hardware device, all layer precision configuration sub-instructions of all layers that need to be executed on the hardware device are integrated, ordered according to the execution time sequence identifier, added with a device identifier and an instruction validity condition, encapsulated as a layer precision dynamic configuration instruction, and each sub-instruction in the layer precision dynamic configuration instruction includes a layer identifier, a device identifier, a calculation precision configuration parameter, an execution time sequence identifier and a validity condition.

[0190] For each hardware device (central processing unit, graphics processing unit, neural network processing unit), all layer precision configuration sub-instructions of all network layers that need to be executed on the device are collected. The above-mentioned sub-instructions are ordered according to the execution time sequence identifier to ensure that they are executed in the correct order.

[0191] A device identifier (indicating which hardware device the sub-instruction should be sent to) and an instruction validity condition are added to each sub-instruction. The instruction validity condition can be a certain time before the task unit of the layer starts execution, or a condition such as the completion of the precision configuration of the preposed dependency layer.

[0192] The above information is integrated and encapsulated to form a layer precision dynamic configuration instruction for the hardware device. The layer precision dynamic configuration instruction is used to instruct each hardware device to configure its low-bit calculation unit (such as arithmetic logic unit ALU, digital signal processor DSP, etc. in hardware) according to the calculation precision configuration parameter in the instruction.

[0193] Step S154: Through the dependency data preloading module, the inter-layer transmission path parameter, bandwidth allocation ratio and data transmission priority in the hierarchical deployment configuration scheme are parsed, the preposed dependency layer and the postposed dependency layer of each task unit are identified, the dependent data is preloaded from the hardware device corresponding to the preposed dependency layer to the cache unit of the hardware device corresponding to the current layer based on the data transmission priority, and a dependent data preloading instruction is generated, which includes a data identifier, a source device identifier, a target device cache address and a loading time sequence.

[0194] The data preloading module dependent on the hierarchical deployment configuration scheme parses the inter-layer transmission path parameters (path information of data transmission between layers), bandwidth allocation ratio and data transmission priority.

[0195] Step S1541: Extract the layer identifier, inter-layer transmission path parameter, bandwidth allocation ratio and data transmission priority of all layers from the hierarchical deployment configuration scheme, and construct an inter-layer dependent transmission network, which takes the layer identifier as the node and the transmission path parameter as the edge, and the weight of the edge is the data transmission priority.

[0196] From the hierarchical deployment configuration scheme, extract the layer identifier, inter-layer transmission path parameter, bandwidth allocation ratio and data transmission priority of all layers. Take the layer identifier as the node, and the transmission path parameter as the edge connecting the nodes, to construct a directed graph, i.e. an inter-layer dependent transmission network. The weight of the edge is set to the data transmission priority, and the higher the priority, the greater the weight. Through the inter-layer dependent transmission network, the data dependency relationship and transmission characteristics between layers can be displayed.

[0197] Step S1542: For each layer task unit, traverse the inter-layer dependent transmission network to identify all pre-dependent layers that provide input data to the task unit, record the layer identifier and corresponding transmission path parameter of each pre-dependent layer, and identify the post-dependent layer corresponding to the output data of the task unit for subsequent data preloading planning.

[0198] For each layer task unit, traverse the inter-layer dependent transmission network to find all layers that provide input data to the task unit, which are the pre-dependent layers of the task unit. Record the layer identifier of each pre-dependent layer and the transmission path parameter (such as data transmission interface, channel, etc.) between the pre-dependent layer and the current layer.

[0199] At the same time, identify the post-dependent layer corresponding to the output data of the task unit (i.e. the layer that receives the output data of the task unit as input). Although the data preloading of the post-dependent layer is not responsible for the dependent data preloading module of the current layer, identifying the post-dependent layer helps the overall data transmission planning and bandwidth allocation.

[0200] Step S1543: For each pre-dependent layer and the transmission path of the current layer, parse the source device identifier, target device identifier and transmission interface type in the transmission path parameter, and determine the data transmission protocol based on the transmission interface type.

[0201] For each transmission path between a pre-dependent layer and the current layer, parse the transmission path parameter to obtain the source device identifier (the hardware device where the pre-dependent layer computing task is located), the target device identifier (the hardware device where the current layer computing task is located), and the transmission interface type (such as PCIe, USB, on-chip bus, etc.).

[0202] The data transmission protocol is determined according to the transmission interface type. Different transmission interface types correspond to different data transmission protocols, for example, the PCIe interface adopts the PCIe protocol, the USB interface adopts the USB protocol, and the on-chip bus may adopt the AXI, AHB, etc. The data transmission protocol specifies the format, timing, error checking, etc. of data transmission, and ensures that the data is correctly transmitted between the source device and the target device.

[0203] Step S1544: Sort all front-dependent data transmission tasks of the current layer based on the data transmission priority, and the transmission task with high priority is preferentially planned for the preloading timing. Meanwhile, the maximum bandwidth that each transmission task can occupy is determined in combination with the bandwidth allocation ratio.

[0204] The current layer may have multiple front-dependent layers, and each front-dependent layer transmits data to the current layer to form a transmission task. The above transmission tasks are sorted based on the data transmission priority, and the task with high data transmission priority (such as the transmission task corresponding to the edge with a large weight value) is preferentially planned for the preloading timing, so as to ensure that these important data can be loaded as soon as possible.

[0205] The maximum bandwidth that each transmission task can occupy is determined in combination with the bandwidth allocation ratio. The bandwidth allocation ratio is determined in the inter-layer transmission configuration information, and each transmission task calculates the maximum bandwidth value that can be occupied according to the corresponding bandwidth allocation ratio and the total bandwidth currently available in the system. For example, if the total bandwidth of the system is 10 GB / s, and the bandwidth allocation ratio of a certain transmission task is 20%, then the maximum bandwidth that can be occupied is 2 GB / s.

[0206] Step S1545: Calculate the preloading start time of each front-dependent data: based on the task execution estimated completion time of the front-dependent layer, the data transmission amount and the occupiable bandwidth, estimate the data transmission time, subtract the transmission time estimated according to the data transmission amount and the bandwidth from the task execution estimated completion time of the front-dependent layer, and then subtract a set redundancy time, as the preloading start time.

[0207] The task execution estimated completion time of the front-dependent layer can be estimated according to the predicted calculation time and the task execution sequence of the layer. The data transmission amount is calculated according to the output data dimension and data format of the front-dependent layer (such as data amount = product of data dimension x byte number of each data element).

[0208] The data transmission time is estimated according to the data transmission amount and the occupiable bandwidth, and the data transmission time = data transmission amount / occupiable bandwidth. A redundancy time is set to cope with possible transmission delay or task execution delay.

[0209] Preload startup time = Estimated completion time of preceding dependency layer tasks - Data transfer time - Redundancy time. By calculating the preload startup time, it is ensured that the dependency data of the current layer can be loaded before its computation tasks begin execution.

[0210] Step S1546: Determine the cache unit address of the hardware device corresponding to the current layer: Based on the task unit data requirement list of the current layer, allocate a contiguous address space in the cache unit to store the preloaded dependency data, and record the start and end addresses of the address space.

[0211] The hardware device corresponding to the current layer (target device) has cache units (such as the L2 cache of the CPU, the video memory cache of the graphics processor, and the on-chip storage of the neural network processor). Based on the storage requirements for input data (such as data volume and data alignment) in the data requirement list of the current layer's task unit, a contiguous address space is allocated in the target device's cache unit to store the dependent data preloaded from the source device. The start and end addresses of this address space are recorded as the target location for data storage.

[0212] Step S1547: Integrate the source device identifier, target device identifier, transmission protocol, preload start time, cache unit address, data identifier, and bandwidth allocation quota to generate a preload sub-instruction for the preceding dependent data; integrate all preload sub-instructions for the preceding dependent data of the current layer, add the layer identifier and task unit identifier of the current layer, and encapsulate them into a dependent data preload instruction. Each preload sub-instruction in the dependent data preload instruction contains a layer identifier, task unit identifier, source device identifier, target device identifier, transmission protocol, preload start time, cache address range, data identifier, and bandwidth allocation quota.

[0213] The source device identifier (the hardware device where the preceding dependency layer is located), the target device identifier (the hardware device where the current layer is located), the transmission protocol, the preload start time, the cache unit address (including the start address and the end address, i.e. the cache address range), the data identifier (used to identify the data block being transmitted, such as the name or number of the data), and the bandwidth allocation quota (the maximum bandwidth that can be occupied) are integrated to generate the preload sub-instruction for the preceding dependency data.

[0214] The preloading sub-instructions for all preceding dependent data in the current layer are integrated, and the layer identifier and task unit identifier of the current layer are added to clarify which task unit in which layer these preloading sub-instructions serve. The integrated information is then encapsulated into a dependency data preloading instruction, which is used to control the preloading process of preceding dependent data.

[0215] Step S155: coordinating data transmission between hardware devices according to inter-layer transmission configuration information in the hierarchical deployment configuration scheme by the cross-device transmission coordination module: allocating inter-device transmission bandwidth based on bandwidth allocation ratio, scheduling transmission queue based on data transmission priority, making dependent data loaded before task execution, generating cross-device transmission coordination instructions, which include bandwidth allocation table, transmission queue order and transmission state feedback requirement.

[0216] The cross-device transmission coordination module coordinates data transmission between different hardware devices according to inter-layer transmission configuration information (including transmission path, bandwidth allocation ratio, data transmission priority, etc.) in the hierarchical deployment configuration scheme.

[0217] The actual transmission bandwidth is allocated to the transmission path between each hardware device based on the bandwidth allocation ratio. The bandwidth allocation table records the bandwidth allocation of each transmission path (source device-target device-transmission interface) in detail, including total bandwidth, allocated bandwidth, remaining bandwidth, etc.

[0218] The transmission queue is scheduled based on the data transmission priority. All data to be transmitted is arranged in order of data transmission priority from high to low to form a transmission queue. In the data transmission process, data with high priority in the transmission queue is transmitted first. For data with the same priority, they can be scheduled according to the order of preloading start time.

[0219] In order to ensure that dependent data is loaded before task execution, the cross-device transmission coordination module needs to monitor the data transmission progress in real time and adjust the order of the transmission queue and the bandwidth allocation according to the task execution sequence. If it is found that the transmission progress of a transmission path lags behind the preloading start time requirement, the bandwidth allocation ratio of the transmission path can be temporarily increased or its priority in the transmission queue can be adjusted.

[0220] The cross-device transmission coordination instructions are generated, which include the bandwidth allocation table (recording the bandwidth allocation of each transmission path), the transmission queue order (the order of data transmission) and the transmission state feedback requirement (such as requiring the transmission device to feedback the transmission progress, error information, etc.).

[0221] Step S156: sending the layer device task allocation instructions, layer precision dynamic configuration instructions, dependent data preloading instructions and cross-device transmission coordination instructions to each hardware device in the heterogeneous hardware, each hardware device receiving task units, configuring low-bit computing units, loading dependent data, executing computing tasks and feeding back transmission state according to the transmission coordination instructions.

[0222] The lightweight runtime scheduler sends the generated layer device task allocation instructions, layer precision dynamic configuration instructions, dependency data preloading instructions, and cross-device transmission coordination instructions to the central processing unit, graphics processing unit, and neural network processor in heterogeneous hardware via the system bus or network.

[0223] Upon receiving the instruction, each hardware device executes the corresponding operation according to the instruction's requirements. The hardware device receives and stores the corresponding task unit based on the layer device task allocation instruction; it configures its low-bit computing unit (such as setting the ALU's bit width, the parameters of the numerical representation format conversion circuit, and the rounding logic mode) according to the computational precision configuration parameters in the layer precision dynamic configuration instruction; based on the dependent data preloading instruction and the cross-device transmission coordination instruction, it loads dependent data from the source device into the local cache unit at the preloading startup time; once the dependent data loading is complete and the computing unit is configured, it begins executing the computational operation instructions in the task unit. During data transmission, the hardware device periodically reports the data transmission progress, error status, and other status information to the lightweight runtime scheduler according to the transmission status feedback requirements in the cross-device transmission coordination instruction.

[0224] Step S157: Through the dynamic reallocation module, the computing task execution status and load status fed back by each hardware device are received in real time. The dynamic adjustment trigger conditions in the hierarchical deployment configuration scheme are compared. If the load fluctuation of a certain device exceeds the threshold or the computing performance of a certain layer decreases by more than the threshold, the multi-objective optimization algorithm is called again to adjust the precision device selection results of the layer and related layers, generate a task reallocation instruction, and send it to the corresponding hardware device to perform task adjustment.

[0225] The dynamic reallocation module receives real-time data from each hardware device via the system feedback mechanism regarding the execution status of computing tasks (such as whether the task is completed, execution time, and accuracy of the calculation results) and the load status (such as resource utilization, response latency, and power consumption).

[0226] The received real-time status information is compared with the dynamic adjustment trigger conditions in the tiered deployment configuration scheme. Dynamic adjustment trigger conditions include hardware device load fluctuation exceeding a threshold (such as the device load fluctuation rate exceeding a set threshold), layer computing performance degradation exceeding a threshold (such as the actual computing time of a certain layer being more than 20% longer than the predicted computing time, or the actual power consumption being more than 15% higher than the predicted power consumption), and inter-layer transmission latency increase exceeding a threshold (such as the actual data transmission latency being more than 25% higher than the predicted transmission latency).

[0227] When a situation is detected that meets the dynamic adjustment triggering conditions, such as the load fluctuation rate of a hardware device exceeding a threshold, causing its performance to be unstable, or the computing efficiency of a certain layer decreasing beyond a threshold, affecting the overall network operating efficiency, the dynamic reallocation module re-invokes the multi-objective optimization algorithm used in step S140.

[0228] With the current hardware device load state, layer computing performance data, etc. as new inputs, the affected layers (such as all layers on the device with load fluctuations exceeding the threshold, or the layers with decreased computing performance and their associated layers) are reselected for adaptive low-bit computing precision and hardware devices, and a new precision device selection result is generated. Task relocation instructions are generated according to the new selection result, which includes the layer identifier to be adjusted, the new hardware device identifier, the new bit selection result, and the task migration method (such as whether to interrupt the current task, how to handle incomplete data, etc.). The task relocation instructions are sent to the corresponding hardware devices, which interrupt the current task (if necessary), release resources, receive new task units and precision configurations, and perform task adjustment according to the instructions.

[0229] Step S158: After all layers of the computing task are executed, the computing results output by each hardware device are collected, and the computing results are integrated according to the task execution sequence and the inter-layer dependency relationship to generate a hierarchical cross-device deployment execution result. The hierarchical cross-device deployment execution result includes the final neural network inference result, the execution detail log of each layer, and the dynamic adjustment record. The execution detail log includes the layer identifier, the execution device identifier, the execution precision parameter, the execution time consumption, the power consumption data, and the data transmission amount. The dynamic adjustment record includes the adjustment trigger condition, the precision device selection result before and after adjustment, and the performance change after adjustment.

[0230] When the computing tasks of all layers of the neural network are executed on the corresponding hardware devices, the lightweight runtime scheduler collects the computing results output by each hardware device. The computing results of each hardware device are organized according to the task execution sequence and the inter-layer dependency relationship, for example, the computing results of the input layer are used as the input of the first convolutional layer, the computing results of the first convolutional layer are used as the input of the pooling layer, and so on, and finally the inference result of the entire neural network is output by the output layer.

[0231] The collected computing results of each layer are integrated according to the task execution sequence and the inter-layer dependency relationship, and the computing results of the intermediate layers are passed to the subsequent layers to finally obtain the output result of the entire neural network, i.e. the final neural network inference result (for an image recognition model, the output result may be the image category label and the corresponding probability value).

[0232] At the same time, the execution detail information recorded by each hardware device during task execution is collected to generate an execution detail log. The execution detail log includes the layer identifier of each layer, the execution device identifier (the hardware device actually executing the layer), the execution precision parameter (the actual bit number, numerical representation format, operation bit width, rounding strategy, etc.), the execution time consumption (actual computing time consumption), the power consumption data (actual average power consumption), and the data transmission amount (the transmission amount of the input and output data of the layer).

[0233] The dynamic adjustment record records all adjustment operations performed by the dynamic re-allocation module during online operation, including adjustment trigger conditions (such as which device load fluctuation exceeds a threshold, which layer computing performance drops below a threshold), precision device selection results before and after adjustment (the bit number and hardware device of the layer before adjustment, and the new bit number and hardware device after adjustment), and performance changes after adjustment (such as how much the computing time is reduced after adjustment, how much the power consumption is reduced, etc.).

[0234] The final neural network inference result, the execution detail log of each layer, and the dynamic adjustment record are integrated together to generate a hierarchical cross-device deployment execution result, which reflects the dynamic deployment and execution of the neural network on the heterogeneous hardware platform.

[0235] In an exemplary embodiment, a heterogeneous-aware adaptive low-bit neural network deployment system is provided, which can be a terminal, a server, etc., and its internal structure diagram can be as shown in Figure 2 The heterogeneous-aware adaptive low-bit neural network deployment system includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface is used to exchange information between the processor and external devices. The communication interface is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, mobile cellular network, near field communication, or other technologies. The computer program is executed by the processor to implement a heterogeneous-aware adaptive low-bit neural network deployment method. The display unit is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad provided on the shell of the heterogeneous-aware adaptive low-bit neural network deployment system. It can also be an external keyboard, touchpad, or mouse, etc.

[0236] It should be noted that, in order to simplify the description of the present disclosure and help understand one or more embodiments of the present disclosure, in the foregoing description of embodiments of the present disclosure, various features are sometimes grouped into one embodiment, figure, or description thereof.

Claims

1. A method for deploying an adaptive low-bit neural network for heterogeneous sensing, characterized in that, The method includes: The hierarchical computational feature information of each layer of the neural network and the parameter information reflecting the dynamic characteristics of heterogeneous hardware are obtained to form a multi-level basic information set. The hierarchical computational feature information includes the basic computational operation type, high-order data interaction mode and inter-layer dependency weight of each layer. The dynamic hardware parameter information includes the static computing capability, dynamic resource occupation characteristics and load fluctuation trend of each hardware device in the heterogeneous hardware. Based on the multi-level basic information set, an offline performance model reflecting the dynamic characteristics of hardware is constructed for each layer of the neural network under different low-bit computation precision and for each hardware device in heterogeneous hardware. This results in a hierarchical performance correlation model, which is used to describe the temporal correlation between computation precision, hardware device dynamic characteristics and layer computation performance. During online operation, the real-time load status of each hardware device in the heterogeneous hardware and the dynamically adjusted energy efficiency constraints are tracked to generate dynamic status monitoring results. The real-time load status includes device resource utilization, computing response latency and load fluctuation rate. The dynamically adjusted energy efficiency constraints include device-level power consumption upper limit, system-level computing efficiency lower limit and load fluctuation adaptation threshold. Combining the hierarchical performance correlation model with the dynamic state monitoring results, a multi-objective optimization algorithm is used to dynamically select suitable low-bit computation precision and corresponding hardware devices for each layer of the neural network, forming a hierarchical deployment configuration scheme. The hierarchical deployment configuration scheme includes the precision selection results of each layer, the device allocation results, the inter-layer data transmission weights, and the dynamic adjustment triggering conditions. A lightweight runtime scheduler with task preloading capability is invoked. Based on the hierarchical deployment configuration scheme, the computation tasks of each layer of the neural network are allocated to the corresponding hardware devices. Inter-layer dependency data is loaded synchronously and the computation tasks are executed using the selected low bit computation precision. This realizes dynamic neural network deployment across hardware devices and obtains hierarchical cross-device deployment execution results. The hierarchical performance correlation model and the dynamic state monitoring results are combined, and a multi-objective optimization algorithm is used to dynamically select suitable low-bit computation precision and corresponding hardware devices for each layer of the neural network, forming a hierarchical deployment configuration scheme, including: The real-time load parameter set, load fluctuation analysis results and dynamic energy efficiency constraint parameters of each hardware device are extracted from the dynamic status monitoring results. Based on the dynamic energy efficiency constraint parameters, hardware devices that meet the current load constraint requirements are selected. Then, combined with the load fluctuation trend, devices with load fluctuation rate lower than a set threshold are selected to generate a dynamic available hardware device list. The dynamic available hardware device list includes device identifier, real-time load summary and load stability rating. Extract the layer-precision performance sub-model corresponding to each device in the dynamic available hardware device list from the hierarchical performance correlation model, split the layer by layer identifier to obtain multi-dimensional performance prediction data for each available device under different bit selections, integrate and generate a layer-precision dynamic device performance dataset. The multi-dimensional performance prediction data includes prediction calculation time, prediction power consumption, prediction data transmission latency and prediction performance stability. For each layer of the neural network, based on the layer's precision dynamic device performance dataset and the layer's hierarchical computational feature information, a multi-objective optimization function is constructed: the first objective function is to reduce prediction power consumption, the second objective function is to shorten prediction computation time, and the third objective function is to improve prediction performance stability. The higher-order data interaction mode and inter-layer dependency weights of the layer are transformed into constraint conditions and incorporated into the function. The constraints include the upper limit of transmission delay determined by the data interaction frequency and the range requirement of performance deviation from adjacent layers determined by the inter-layer dependency weights. The multi-objective optimization function is solved using a non-dominated sorting algorithm to generate multiple sets of non-dominated precision device candidate combinations. Each candidate combination includes the bit depth selection, hardware device identifier, and the calculation results of the corresponding first objective function, second objective function, and third objective function. Based on the inter-layer dependency weights of this layer, inter-layer correlation evaluation is performed on multiple non-dominated precision device candidate combinations. The degree of synergy between each candidate combination and the candidate combinations of adjacent layers in terms of data transmission latency, power consumption, and computational efficiency is comprehensively evaluated to obtain a comprehensive performance matching degree. The candidate combination with the highest performance matching degree is selected as the precision device selection result of this layer. The performance matching degree is a comprehensive metric value obtained based on the degree of synergy between data transmission latency, power consumption, and computational efficiency. Perform inter-layer data dependency mapping on the precision device selection results of all layers. Based on the high-order data interaction mode and inter-layer dependency weight of each layer, plan the data transmission path between adjacent layers, determine the bandwidth allocation ratio and data transmission priority of each transmission path, and generate inter-layer transmission configuration information. By integrating the precision device selection results and inter-layer transmission configuration information of all layers, and adding dynamic adjustment trigger conditions, a hierarchical deployment configuration scheme is finally formed. The dynamic adjustment trigger conditions include hardware device load fluctuation exceeding a threshold, layer computing performance degradation exceeding a threshold, and inter-layer transmission latency increase exceeding a threshold. The hierarchical deployment configuration scheme includes the layer identifier, bit length selection result, hardware device identifier, inter-layer transmission path parameters, bandwidth allocation ratio, data transmission priority, and dynamic adjustment trigger conditions for each layer.

2. The method for deploying heterogeneous sensing adaptive low-bit neural networks according to claim 1, characterized in that, The process involves acquiring hierarchical computational feature information of each layer of the neural network and parameter information reflecting the dynamic characteristics of heterogeneous hardware, forming a multi-level set of basic information, including: The structured network description file of the neural network to be deployed is parsed, and the inter-layer topological connections, structural attributes of each layer and inter-layer data dependency strength of the neural network are extracted to generate multi-level layer structure description information. The multi-level layer structure description information includes the input and output data dimensions of each layer, layer type identifier, inter-layer data flow direction and dependency weight coefficient. Based on the multi-level layer structure description information, hierarchical feature extraction is performed on each layer. First, the basic computational operation type and basic data interaction volume of the layer are extracted. Then, the data interaction frequency, data reuse rate and feature dimension of the interaction data between the layer and adjacent layers are analyzed to generate the high-order data interaction mode of the layer. Finally, the inter-layer dependency weight is determined by combining the inter-layer dependency weight coefficient, and the hierarchical computational feature information of each layer is integrated to form the hierarchical computational feature information of each layer. The basic computational operation type includes convolution operation, fully connected operation and activation function operation. The basic data interaction volume includes the layer input data volume and the layer output data volume. Dynamic parameter acquisition is performed on each hardware device in the heterogeneous hardware. First, the static computing capability parameters of each hardware device are collected. Then, the resource usage parameters, response latency parameters, and power consumption parameters of the devices are collected under different simulated loads. The variation law of parameters under different loads is analyzed to generate dynamic resource usage characteristics. Finally, the load fluctuation trend of the devices is fitted based on historical load data, and the dynamic hardware parameter information of each hardware device is integrated to form dynamic hardware parameter information of each hardware device. The static computing capability parameters include the number of floating-point operations per second, the number of parallel computing threads, and the basic memory capacity of the device. The dynamic resource usage characteristics include load utilization rate correlation curve, load latency correlation curve, and load power consumption correlation curve. The multi-level layer structure description information, the hierarchical calculation feature information of each layer, and the dynamic hardware parameter information of each hardware device are hierarchically associated and integrated. First, the hierarchical calculation feature information is mapped to the corresponding layer structure description information according to the layer identifier, and then the dynamic hardware parameter information is associated with the device type according to the device identifier to form an initial multi-level information set. For layers lacking hierarchical computational feature information, statistical methods or preset rules are used to supplement high-order data interaction patterns and inter-layer dependency weights based on the feature information already acquired by layers of the same type. For devices lacking parameter information reflecting dynamic hardware characteristics, estimation methods are used to supplement load fluctuation trends based on the parameter information already acquired by devices of the same type. Finally, a multi-level basic information set is generated. Each layer information entry in the multi-level basic information set includes a layer identifier, corresponding hierarchical computational feature information, and layer structure description information. Each device information entry includes a device identifier, corresponding dynamic hardware parameter information, and device type identifier.

3. The method for deploying heterogeneous sensing adaptive low-bit neural networks according to claim 1, characterized in that, Based on the multi-level basic information set, an offline performance model reflecting the dynamic characteristics of hardware is constructed for each layer of the neural network under different low-bit computation precision and for each hardware device in heterogeneous hardware, resulting in a hierarchical performance correlation model, including: Extract the hierarchical computational feature information of each layer and the dynamic hardware parameter information of each hardware device from the multi-level basic information set to determine the low-bit computational precision gradient range to be tested. The low-bit computational precision gradient range includes multiple bit selections that increase in bit width. Each bit selection corresponds to a set of computational precision configuration parameters, which include numerical representation format, operation bit width and rounding strategy. For each layer, select each bit depth within the low-bit computation precision gradient range and sequentially perform multiple rounds of performance data acquisition: first, encapsulate the computation task of the layer into task units adapted to the hardware device; then, allocate the task units to each hardware device in the heterogeneous hardware; execute the computation task under different simulated load conditions of the device; collect the computation time, power consumption, data transmission latency, and performance stability parameters during each round of task execution; and integrate the results of multiple rounds of acquisition to generate a multi-dimensional performance dataset of the layer on each hardware device under the selected bit depth. Based on the multi-dimensional performance dataset of all layers, a performance mapping relationship of layer computing feature hardware dynamic characteristics is constructed: First, the basic computing operation type and high-order data interaction mode in the hierarchical computing feature information are associated with the static computing capability and dynamic resource consumption characteristics in the dynamic hardware parameter information to obtain the associated features. Then, a mapping function between the associated features and the multi-dimensional performance data is established to generate the initial offline performance model. The initial offline performance model is subjected to multiple rounds of iterative optimization: the first round of optimization uses the gradient descent algorithm to adjust the weight coefficients of the mapping function, reducing the deviation between the model's predicted values ​​and the actual multi-dimensional performance data; the second round of optimization introduces a time-series correlation factor to incorporate the load fluctuation trend of hardware devices into the model, correcting the performance prediction results under different load conditions; the third round of optimization adjusts the inter-layer performance correlation parameters of the model based on inter-layer dependency weights to ensure the consistency of performance predictions of adjacent layers, generating a multi-round optimized offline performance model; Based on the offline performance model after multiple rounds of optimization, the model is split and integrated hierarchically according to the layer identifier and hardware device identifier. A corresponding device precision performance sub-model is generated for each layer. Then, the sub-models of all layers are associated through inter-layer dependency weights to form a hierarchical performance association model. Each sub-model entry in the hierarchical performance association model includes a layer identifier, bit selection, hardware device identifier, dynamic load status, corresponding multi-dimensional performance data, and inter-layer association weights.

4. The method for deploying heterogeneous sensing adaptive low-bit neural networks according to claim 1, characterized in that, During the online operation, the real-time load status of each hardware device in the heterogeneous hardware and the dynamically adjusted energy efficiency constraints are tracked to generate dynamic status monitoring results, including: A load tracing module is deployed for each hardware device in the heterogeneous hardware. The load tracing module includes a real-time data acquisition unit, a fluctuation trend analysis unit, and a constraint adaptation unit. The real-time data acquisition unit is used to collect device parameters at dynamic time intervals. The fluctuation trend analysis unit is used to analyze the parameter change pattern. The constraint adaptation unit is used to adjust the energy efficiency constraint parameters. The dynamic sampling interval is set based on the load fluctuation trend of the hardware device. When the load fluctuation rate is lower than the set first threshold, the sampling interval is increased, and when the load fluctuation rate is higher than the set second threshold, the sampling interval is decreased. The current resource utilization rate, computing response latency and instantaneous power consumption of each hardware device are collected. The resource utilization rate includes memory utilization rate, computing unit utilization rate and bandwidth utilization rate. The fluctuation trend analysis unit analyzes the collected resource occupancy rate, calculation response latency, and instantaneous power consumption within a set time window. It calculates the rate of change of each parameter within a unit of time and counts the direction of change. By combining the rate of change and the direction of change of each parameter, it evaluates and generates the load fluctuation rate of each hardware device and determines its load fluctuation direction, which includes upward trend, downward trend, and stable trend. The constraint adaptation unit obtains the initial energy efficiency constraints and dynamically adjusts the constraint parameters based on the load fluctuation trend and current load status of each hardware device: for devices with an increasing load trend, the adjustment threshold of their device-level power consumption upper limit is increased; for devices with a decreasing load trend, the adjustment threshold of the system-level computing efficiency lower limit is decreased, thus generating dynamically adjusted energy efficiency constraints. The system integrates the resource utilization rate, computing response latency, instantaneous power consumption, load fluctuation rate, load fluctuation direction, and dynamically adjusted energy efficiency constraints of each hardware device, establishes an association mapping according to the device identifier, and generates dynamic status monitoring results. Each device entry in the dynamic status monitoring results includes the device identifier, real-time load parameter set, load fluctuation analysis results, and corresponding dynamic energy efficiency constraint parameters.

5. The method for deploying heterogeneous sensing adaptive low-bit neural networks according to claim 1, characterized in that, For each layer of the neural network, a multi-objective optimization function is constructed based on the layer's precision dynamic device performance dataset and the layer's hierarchical computational feature information, including: Extract the prediction computation time, prediction power consumption and prediction performance stability data of the current analysis layer for each available hardware device under all bit selections from the layer-accuracy dynamic device performance dataset, and establish a three-dimensional data matrix according to bit selection and device identifier. Extract high-order data interaction patterns and inter-layer dependency weights from the hierarchical computational feature information of this layer, analyze the data interaction frequency, data reuse rate and interaction data feature dimensions in the high-order data interaction patterns, determine the maximum allowable value of data transmission delay, analyze the correlation coefficient between adjacent layers in the inter-layer dependency weights, and determine the maximum allowable range of performance deviation between this layer and adjacent layers. With minimizing predicted power consumption as the primary objective function, the predicted power consumption data is used as the function input variable. The function output value is set to increase as power consumption decreases. At the same time, based on the dynamic resource occupancy characteristics of hardware devices, the power consumption data corresponding to devices with load fluctuation rates below a set threshold are assigned a higher weight coefficient. The second objective function is to minimize the prediction computation time. The prediction computation time data is used as the function input variable. The function output value is set to increase as the computation time decreases. At the same time, the weights are adjusted in combination with the basic computation operation type of this layer, and higher weights are given to the time data of the computation-intensive layer. With the maximization of prediction performance stability as the third objective function, the prediction performance stability data is used as the function input variable. The function output value is set to increase as the stability coefficient increases. At the same time, the inter-layer dependency weights of the layer are associated, and higher weights are assigned to the stability data of layers with high dependency. The maximum allowable value of data transmission delay determined by the high-order data interaction mode is transformed into the first constraint, requiring that the predicted data transmission delay corresponding to the candidate combination does not exceed the allowable value; the maximum allowable range of performance deviation determined by the inter-layer dependency weight is transformed into the second constraint, requiring that the performance deviation between the candidate combination and the candidate combination of the adjacent layer does not exceed the range. By integrating the first objective function, the second objective function, and the third objective function with the first constraint and the second constraint, a multi-objective optimization function is constructed. The weight coefficient of each objective function is set based on the priority setting in the dynamically adjusted energy efficiency constraint, thus generating the multi-objective optimization function of this layer.

6. The method for deploying heterogeneous sensing adaptive low-bit neural networks according to claim 1, characterized in that, The non-dominated sorting algorithm is used to solve the multi-objective optimization function to generate multiple sets of non-dominated precision device candidate combinations, including: The combination of all bit lengths of this layer with available hardware devices is used as the initial solution set, and each combination corresponds to a set of input parameters for a multi-objective optimization function; For each combination in the initial solution set, calculate its function value under the first objective function, the second objective function, and the third objective function to obtain the three-dimensional objective function value vector for each combination; Perform the first round of non-dominated sorting: Traverse each combination in the initial solution set, compare the three-dimensional objective function value vector of the combination with all other combinations. If there exists a combination whose objective function values ​​are all no worse than the combination and at least one objective function value is better than the combination, then the combination is dominated. Otherwise, the combination is a non-dominated solution, and all non-dominated solutions are assigned to the first non-dominated layer. Remove the solutions from the first non-dominated layer from the initial solution set, perform a second round of non-dominated sorting on the remaining solutions, repeat the comparison process, and generate a second non-dominated layer; iterate in this way until all solutions are assigned to different non-dominated layers; Calculate the crowding distance for each non-dominated layer: For the solutions in each non-dominated layer, sort them by function value in each objective function dimension. Set the crowding distance to infinity for the first and last solutions in the sort. For the middle solutions, calculate the sum of the differences in function value between them and their adjacent solutions in the objective function dimension. The larger the crowding distance, the sparser the distribution of the solutions. Candidate combinations are selected based on the hierarchy of the non-dominated layer and the crowding distance of the solutions within the layer: the non-dominated layer with higher hierarchy is selected first, and the solutions within the non-dominated layer are sorted from largest to smallest according to the crowding distance. The number of solutions selected is determined based on the total number of solutions in the layer and the inter-layer dependency complexity. Each selected solution corresponds to a set of precision device candidate combinations. Extract the bit selection, hardware device identifier, and calculation results of the first, second, and third objective functions corresponding to the selected solution to generate multiple sets of non-dominated precision device candidate combinations. Each candidate combination is accompanied by an objective function value vector and a crowding distance identifier.

7. The method for deploying heterogeneous sensing adaptive low-bit neural networks according to claim 1, characterized in that, The process involves invoking a lightweight runtime scheduler with task preloading capabilities. Based on the hierarchical deployment configuration scheme, the scheduler allocates the computational tasks of each layer of the neural network to the corresponding hardware devices, synchronously loads inter-layer dependency data, and executes the computational tasks using the selected low-bit computation precision. This achieves dynamic neural network deployment across hardware devices, resulting in hierarchical cross-device deployment execution results, including: The hierarchical deployment configuration scheme is input into a lightweight runtime scheduler with task preloading capability. The lightweight runtime scheduler includes a hierarchical task management module, a precision dynamic configuration module, a dependency data preloading module, a cross-device transmission coordination module, and a dynamic reallocation module. The hierarchical task management module parses the layer identifier, hardware device identifier, and dynamic adjustment trigger conditions of each layer in the hierarchical deployment configuration scheme, sorts them according to the inter-layer dependency weights to generate a task execution sequence, and then encapsulates the computing tasks of each layer into task units containing layer identifiers, computing operation instructions, and data requirement lists, and generates layer device task allocation instructions, which are used to instruct each task unit to be sent to the corresponding hardware device. The precision dynamic configuration module parses the layer identifier and bit selection result of each layer in the hierarchical deployment configuration scheme, determines the corresponding calculation precision configuration parameters in combination with the basic calculation operation type of the layer, and generates a layer precision dynamic configuration instruction. The layer precision dynamic configuration instruction is used to instruct each hardware device to configure the low-bit calculation unit according to the parameters. The dependency data preloading module parses the inter-layer transmission path parameters, bandwidth allocation ratio, and data transmission priority in the hierarchical deployment configuration scheme, identifies the pre-dependency layer and post-dependency layer of each task unit, and starts loading dependency data from its corresponding hardware device to the cache unit of the hardware device corresponding to the current layer before the computation task of the pre-dependency layer is completed based on the data transmission priority, and generates a dependency data preloading instruction. The dependency data preloading instruction includes a data identifier, a source device identifier, a target device cache address, and a loading sequence. The cross-device transmission coordination module coordinates data transmission between hardware devices based on the inter-layer transmission configuration information in the hierarchical deployment configuration scheme: it allocates transmission bandwidth between devices based on the bandwidth allocation ratio, schedules transmission queues based on data transmission priority, ensures that dependent data is loaded before task execution, and generates cross-device transmission coordination instructions, which include a bandwidth allocation table, transmission queue order, and transmission status feedback requirements. The layer device task allocation instruction, layer precision dynamic configuration instruction, dependency data preloading instruction, and cross-device transmission coordination instruction are sent to each hardware device in the heterogeneous hardware. Each hardware device receives the task unit according to the instruction, configures the low-bit computing unit, loads the dependency data, executes the computing task, and feeds back the transmission status according to the transmission coordination instruction. Through the dynamic reallocation module, the computing task execution status and load status fed back by each hardware device are received in real time. The dynamic adjustment trigger conditions in the hierarchical deployment configuration scheme are compared. If the load fluctuation of a certain device exceeds the threshold or the computing performance of a certain layer decreases by more than the threshold, the multi-objective optimization algorithm is called again to adjust the precision device selection results of the layer and related layers, generate a task reallocation instruction, and send it to the corresponding hardware device to perform task adjustment. Once all computational tasks at each layer are completed, the computational results output by each hardware device are collected. The computational results are then integrated according to the task execution sequence and inter-layer dependencies to generate a hierarchical cross-device deployment execution result. This hierarchical cross-device deployment execution result includes the final neural network inference result, execution detail logs for each layer, and dynamic adjustment records. The execution detail logs include layer identifiers, execution device identifiers, execution precision parameters, execution time, power consumption data, and data transmission volume. The dynamic adjustment records include adjustment trigger conditions, precision device selection results before and after adjustment, and performance changes after adjustment.

8. The method for deploying heterogeneous sensing adaptive low-bit neural networks according to claim 7, characterized in that, The precision dynamic configuration module parses the layer identifier and bit selection result of each layer in the hierarchical deployment configuration scheme, determines the corresponding computational precision configuration parameters based on the basic computational operation type of that layer, and generates a layer precision dynamic configuration instruction, including: Extract the layer identifier, bit length selection result and basic calculation operation type of all layers from the hierarchical deployment configuration scheme, and establish a layer precision operation association table according to the layer identifier; For each layer identifier in the layer precision operation association table, the numerical representation format is determined based on the bit depth selection result; The operation bit width is adjusted based on the basic computation operation type of this layer: for convolution operations, an operation bit width matching the number of bits is used; for fully connected operations, a specified bit width is added based on the number of bits and the sensitivity of the operation type to error to reduce accumulated error; for activation function operations, a fixed bit width is used to adapt to nonlinear computation, and operation bit width parameters are generated. The rounding strategy is determined by combining the bit selection result with the type of basic computational operation: a rounding strategy of 4-to-5 is used for computationally intensive operations, a rounding strategy of truncated rounding is used for data interaction intensive operations, and a banker's rounding strategy is used for precision-sensitive operations. Rounding strategy parameters are generated. The numerical representation format, operation bit width parameter, and rounding strategy parameter are integrated to form the calculation precision configuration parameter for this layer. The layer identifier is associated with the calculation precision configuration parameter to generate the layer precision configuration sub-instruction for this layer. The layer-precision configuration sub-instructions of all layers are sorted in time sequence. The execution order of the instructions is determined based on the inter-layer dependency weight. The configuration instructions of the preceding dependent layers are executed first, and the configuration instructions of the subsequent layers are executed. An execution time sequence identifier is added to each sub-instruction. For each hardware device, integrate all layer precision configuration sub-instructions that need to be executed on that hardware device, sort them by execution timing identifier, add device identifier and instruction activation conditions, and encapsulate them into a layer precision dynamic configuration instruction. Each sub-instruction in the layer precision dynamic configuration instruction contains a layer identifier, a device identifier, calculation precision configuration parameters, execution timing identifier and activation conditions.

9. The method for deploying heterogeneous sensing adaptive low-bit neural networks according to claim 7, characterized in that, The dependency data preloading module parses the inter-layer transmission path parameters, bandwidth allocation ratio, and data transmission priority in the hierarchical deployment configuration scheme, identifies the pre-dependency layer and post-dependency layer of each task unit, and preloads dependency data from the hardware device corresponding to the pre-dependency layer to the cache unit of the hardware device corresponding to the current layer based on the data transmission priority, generating dependency data preloading instructions, including: Extract the layer identifiers, inter-layer transmission path parameters, bandwidth allocation ratios, and data transmission priorities of all layers from the hierarchical deployment configuration scheme, and construct an inter-layer dependent transmission network. The inter-layer dependent transmission network uses layer identifiers as nodes, transmission path parameters as edges, and the weight of the edges is the data transmission priority. For each task unit in a layer, the inter-layer dependency transmission network is traversed to identify all preceding dependency layers that provide input data to the task unit. The layer identifier and corresponding transmission path parameters of each preceding dependency layer are recorded. At the same time, the subsequent dependency layers corresponding to the output data of the task unit are identified for subsequent data preloading planning. For each transmission path between the preceding dependent layer and the current layer, the source device identifier, target device identifier, and transmission interface type in the transmission path parameters are parsed, and the data transmission protocol is determined based on the transmission interface type. Based on data transmission priority, all preceding dependent data transmission tasks of the current layer are sorted. High-priority transmission tasks are given priority in preloading timing, and the maximum bandwidth that each transmission task can occupy is determined in combination with the bandwidth allocation ratio. Calculate the preload startup time for each preceding dependency data: Based on the expected completion time of the task execution of the preceding dependency layer, the amount of data transmission and the available bandwidth, estimate the data transmission time, subtract the estimated transmission time based on the amount of data transmission and bandwidth from the expected completion time of the preceding dependency layer task execution, and then subtract a set redundancy time to obtain the preload startup time; Determine the cache unit address of the hardware device corresponding to the current layer: Based on the task unit data requirement list of the current layer, allocate a contiguous address space in the cache unit to store the preloaded dependency data, and record the start and end addresses of the address space; Integrate the source device identifier, target device identifier, transmission protocol, preload start time, cache unit address, data identifier and bandwidth allocation quota to generate the preload sub-instruction for the preceding dependent data; The preloading sub-instructions for all the preceding dependent data of the current layer are integrated, and the layer identifier and task unit identifier of the current layer are added. They are then encapsulated into a dependent data preloading instruction. Each preloading sub-instruction in the dependent data preloading instruction contains a layer identifier, a task unit identifier, a source device identifier, a target device identifier, a transmission protocol, a preloading start time, a cache address range, a data identifier, and a bandwidth allocation quota.

Images