Calculation model deployment method and electronic equipment

Abstract

The invention provides a calculation model deployment method and electronic equipment, and the method comprises the steps: determining the target quantization precision corresponding to each network layer in an original model, and determining a network layer comprising at least two continuous operators as a target network layer; quantifying each network layer of the original model based on the target quantification precision, and obtaining a target model in combination with a target fusion operator obtained by fusing at least two continuous operators of each target network layer; determining a target processing core corresponding to each network layer from a plurality of processing cores of a target deployment platform based on the target quantization precision and the calculation amount of each network layer of the target model, and obtaining calculation task distribution information; and loading the target model and the calculation task allocation information to the target deployment platform. According to the calculation model deployment method and the electronic equipment provided by the invention, the detection precision of the target model is maintained, the reasoning throughput rate and the resource utilization rate are remarkably improved, and the reasoning delay of edge equipment is effectively reduced.

Description

Technical Field

[0001] This application relates to the field of model deployment technology, and in particular to a computational model deployment method and electronic device. Background Technology

[0002] In the current model deployment process, model quantization remains at the level of uniform precision quantization and static parameter configuration. It cannot adaptively adjust the quantization bit width according to the feature complexity of different network layers, resulting in low model inference accuracy. At the same time, there are also problems such as low operator execution efficiency and large differences in load between cores, which leads to high model inference latency and high time delay, making it difficult to meet the real-time requirements of edge devices. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a method for deploying a computing model and an electronic device.

[0004] To achieve the above objectives, this application provides a method for deploying a computational model, comprising: Determine the target quantization accuracy for each network layer in the original model, and identify the network layer containing at least two consecutive operators as the target network layer; Based on the target quantization accuracy, each network layer of the original model is quantized, and the target fusion operator is obtained by performing fusion on at least two consecutive operators of each target network layer to obtain the target model. Based on the target quantization accuracy and computational cost of each network layer of the target model, the target processing core corresponding to each network layer is determined from multiple processing cores of the target deployment platform to obtain computational task allocation information; The target model and the computing task allocation information are loaded into the target deployment platform.

[0005] Optionally, determining the target quantization accuracy corresponding to each network layer in the original model includes: For the network layers containing convolutional operations in the original model, the layer entropy value of each network layer is calculated, and the target quantization accuracy is determined based on its corresponding layer entropy value. For network layers in the original model that do not contain convolution operations, the basic quantization precision is determined to be the corresponding target quantization precision; Among them, the layer entropy value is positively correlated with the target quantization accuracy.

[0006] Optionally, calculating the layer entropy value of each network layer includes: For each network layer that contains convolutional operations, obtain its output feature map; Based on its corresponding output feature map, the entropy value of each channel in the network layer is calculated, and the average value of the entropy values ​​of all channels in the network layer is determined as the layer entropy value of the network layer.

[0007] Optionally, the at least two consecutive operators include the original convolution operator and the batch normalization operator, and the target fusion operator is obtained by the following methods: For each target network layer, extract the original convolution parameters from its original convolution operator and the batch normalization parameters from its batch normalization operator; New convolution parameters are generated based on their corresponding convolution parameters and corresponding batch normalization parameters. The target fusion operator is obtained by substituting the original convolution parameters into the original convolution operator with the new convolution parameters.

[0008] Optionally, the at least two consecutive operators include the original convolution operator and the activation operator, and the target fusion operator is obtained by the following methods: For each target network layer, the computation nodes of its original convolution operator and the computation nodes of its activation operator are merged into one computation node to obtain the target fusion operator.

[0009] Optionally, the at least two consecutive operators include the original convolution operator, the batch normalization operator, and the activation operator, and the target fusion operator is obtained by the following methods: For each target network layer, extract the original convolution parameters from its original convolution operator and the batch normalization parameters from its batch normalization operator; New convolution parameters are generated based on their corresponding convolution parameters and corresponding batch normalization parameters. The new convolution parameters are substituted into the original convolution operator to replace the original convolution parameters, thus obtaining the target convolution operator. The computation nodes of the target convolution operator and the computation nodes of the activation operator are merged into one computation node to obtain the target fusion operator.

[0010] Optionally, the step of determining the target processing core corresponding to each network layer from multiple processing cores of the target deployment platform based on the target quantization accuracy and computational cost of each network layer of the target model to obtain computational task allocation information includes: The processing core with the highest computing power among the multiple processing cores of the target deployment platform is identified as the first processing core; The first processing core is determined to be the target processing core of the network layer whose target quantization accuracy is higher than the preset accuracy. For the remaining network layers whose target quantization accuracy is lower than or equal to the preset accuracy, the target processing core is determined from multiple processing cores, with the difference between the total computation amount corresponding to each processing core being less than a first preset threshold as the allocation target, so as to obtain the computation task allocation information.

[0011] Optionally, the computational complexity of each network layer of the target model is calculated using the following methods: Since the network layer of the target model is a convolutional layer, the corresponding computational cost is determined based on the output feature map size, number of channels, and convolutional kernel size of the corresponding network layer.

[0012] Optionally, it also includes: A dynamic task migration strategy is constructed based on the computing task allocation information, and the dynamic task migration strategy is loaded into the target deployment platform; The dynamic task migration strategy is as follows: when the target deployment platform is running the target model, if the real-time load rate of at least one of the multiple processing cores reaches a second preset threshold, some of the computing tasks in the corresponding processing core will be migrated to other processing cores.

[0013] Based on the same inventive concept, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0014] As can be seen from the above, the computational model deployment method and electronic device provided in this application first determine the target quantization precision for each network layer in the original model to achieve a differentiated quantization strategy; then, multi-operator fusion is performed on network layers containing at least two consecutive operators, that is, at least two consecutive operators in each target network layer are fused into a target fusion operator; finally, based on the target quantization precision and computational cost of each network layer in the target model, a suitable target processing core is determined for each network layer. In the above process, firstly, the appropriate target quantization precision is determined according to the computational complexity, information content, and other information of different network layers to avoid the precision loss caused by uniform bit-width quantization, so that after deployment on the target deployment platform, while ensuring computational precision, the overall computing power utilization of the target deployment platform can be significantly improved; secondly, by performing multi-operator fusion on network layers containing at least two consecutive operators, the consecutive operators in the original model are rewritten into a single target fusion operator, effectively reducing DDR. This application reduces memory access overhead, shortens the operator chain execution path, reduces data transfer latency between cores, and improves the execution efficiency of processing cores. Simultaneously, based on the target quantization accuracy and computational cost of each network layer of the target model, appropriate processing cores are allocated to each network layer to achieve cross-core load balancing and optimal execution paths, eliminating problems such as large load differences and idle computing power among processing cores. Through collaborative strategies such as differentiated quantization, multi-operator fusion, and inter-core task allocation, this application enables the target model to significantly improve inference throughput and resource utilization while maintaining detection accuracy, effectively reducing inference latency on edge devices, and significantly improving inference real-time performance, achieving efficient deployment suitable for practical target recognition scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a computational model deployment method according to an embodiment of this application; Figure 2 This is a schematic diagram of a computational model deployment device according to an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] In recent years, with the deep integration of the Internet of Things (IoT) and Artificial Intelligence (AI), edge AI technology has experienced explosive growth. Compared with traditional centralized AI processing in the cloud, edge AI deploys inference tasks on hardware devices close to the data acquisition end, which can significantly reduce data transmission bandwidth consumption, reduce end-to-end latency, and enhance data privacy protection through localized processing. Therefore, it has become a core technology support in scenarios with stringent requirements for real-time performance and reliability, such as intelligent security, target tracking, and industrial inspection.

[0020] Object recognition, as one of the core tasks of edge AI, relies on the synergy of two aspects: first, high-precision deep learning models, such as the YOLO series, SSD, and Faster R-CNN; and second, efficient edge hardware deployment capabilities. Among these, model quantization technology is a key means of achieving edge deployment. By converting model parameters from floating-point (FP32 / FP16) to integer (INT8 / INT4), it can significantly reduce model storage footprint (INT8 quantization can reduce storage space by 75%), improve inference speed (low-precision computing units have higher parallel efficiency), and simultaneously reduce hardware power consumption (edge ​​devices are typically limited by power supply and heat dissipation).

[0021] As one of the mainstream chips for high-performance edge computing, Rockchip's RK3588 features a built-in tri-core NPU with 6 TOPS (Tera Operations Per Second) computing power, supports INT4 / INT8 mixed precision computing, and provides a solid hardware foundation for the efficient deployment of target recognition models through its multi-core parallel architecture and dedicated convolution acceleration unit, making it the preferred platform for edge scenarios such as smart security and embodied intelligence.

[0022] In the existing technology, there are still obvious limitations in the deployment of model quantization for RK3588, mainly reflected in problems such as coarse quantization strategies, low operator execution efficiency, and insufficient utilization of hardware computing power.

[0023] First, regarding quantization precision allocation, existing technologies generally employ uniform bit width or static quantization schemes, lacking the ability to analyze and differentiate quantization based on the feature complexity of different network layers. They cannot adaptively allocate quantization bit width according to the differences in feature representation capabilities of network layers. Using fixed INT8 / INT4 precision in high-entropy regions such as feature fusion layers and detection heads easily leads to significant precision degradation, with the mean average precision (mAP) decreasing by more than 5%. On the other hand, continuing to use high bit width quantization in low-sensitivity convolutional layers wastes NPU computing power, causing the utilization rate of the RK3588's INT4 computing unit to remain below 30% for a long period, making it difficult to achieve the optimal balance between quantization precision and inference performance.

[0024] Secondly, at the operator execution level, the current deployment method still uses the operator chain structure of the original model, executing convolution operators, batch normalization (BN) operators, and activation operators independently layer by layer. There is a lack of hardware-level fusion mechanisms between operators, requiring intermediate features to be frequently written back to DDR and read again, resulting in high memory access overhead and causing inter-core data transfer latency to exceed 30%. This not only reduces operator execution efficiency but also hinders the performance of the RK3588 NPU in low-latency scenarios.

[0025] Finally, regarding multi-core scheduling, existing technologies do not incorporate effective inter-core load balancing strategies based on the RK3588 NPU's three-core heterogeneous architecture. Traditional solutions do not partition tasks based on network layer computational load (FLOPs), quantization bit width, or hierarchical structure, resulting in significant load differences between master and slave cores, potentially exceeding 40%. Furthermore, issues of core idleness and localized congestion exist, preventing the full utilization of the 6 TOPS overall computing power and leading to significantly higher inference latency.

[0026] In summary, existing deployment methods have shortcomings in areas such as quantization precision allocation, operator execution optimization, and multi-core scheduling mechanisms, which limit the accuracy, speed, and energy efficiency of the RK3588 platform when deploying target recognition models at the edge.

[0027] In view of this, this application provides a computational model deployment method. Through the coordinated optimization of differentiated quantization, multi-operator fusion and multi-core load balancing, the target model can be deployed efficiently. This method achieves coordinated optimization of high-precision inference, low memory access overhead and high core utilization, thereby significantly improving the overall inference performance of the model on embedded platforms such as RK3588. It is especially suitable for the deployment of computational models on edge devices.

[0028] Reference Appendix Figure 1 The method includes: S101. Determine the target quantization accuracy corresponding to each network layer in the original model, and determine the network layer containing at least two consecutive operators as the target network layer. Specifically, the original model is acquired and parsed to obtain its network structure and parameters. Then, based on the network structure and parameters, the target quantization accuracy of each network layer in the original model is determined, and the network layer containing at least two consecutive operators is identified as the target network layer. The original model can be YOLO, SSD, U-Net, ResNet, etc. Among them, the YOLO model is a target detection model known for its real-time performance. It completes target localization and classification in a single forward inference through an end-to-end convolutional neural network and is widely used in scenarios requiring high-speed recognition, such as autonomous driving, video surveillance, security inspection, drone perception, and industrial inspection. Optionally, the original model can be parsed using a model deployment tool. For example, the "rknn.load_pytorch()" interface of the RKNN Toolkit can be called to parse custom modules such as C3 and SPPF of the original model (e.g., YOLOv8) to obtain its network structure and model parameters.

[0029] S102. Based on the target quantization accuracy, quantize each network layer of the original model, and combine the target fusion operator obtained by performing fusion on the at least two consecutive operators of each target network layer to obtain the target model; Specifically, the target quantization precision can be INT4 or INT8, or other quantization precisions, without limitation. In step S102, the original model can be quantized first, and then the continuous operators of the target network layers in the quantized model can be fused. That is, based on the target quantization precision corresponding to each network layer in the original model, the corresponding network layer is quantized to obtain the first model; at least two continuous operators in each target network layer in the first model are fused into a target fusion operator, thereby obtaining the target model. Alternatively, the original model can be fused first, and then the fused model can be quantized. That is, at least two continuous operators in each target network layer in the original model are fused into a target fusion operator to obtain the second model; then, based on the target quantization precision corresponding to each network layer in the second model, the corresponding network layer is quantized to obtain the target model. The execution order of the quantization of the original model and the operator fusion is not limited.

[0030] In this step, the target quantization precision of each network layer can be further used to guide the priority of operator fusion. The target network layer may have two, three or more consecutive operators. The target quantization precision can be used to determine the specific fusion strategy for each target network layer. For example, the target network layer with high target quantization precision can be preferentially fused with the original convolution operator, batch normalization operator and activation operator, without any specific restrictions.

[0031] S103. Based on the target quantization accuracy and computational load of each network layer of the target model, determine the target processing core corresponding to each network layer from multiple processing cores of the target deployment platform to obtain computational task allocation information; Specifically, the target processing core of a network layer is used to execute the computation process of the corresponding network layer. The target deployment platform can be the RK3588 platform or other platforms, without specific restrictions. After the model is loaded onto the target deployment platform, the processing core of the target deployment platform is used to execute the computation steps of the model. Taking the RK3588 platform as an example, its Neural Processing Unit (NPU) consists of three processing cores, each of which can independently undertake part of the computation work of the network layer. The three processing cores have certain differences in computing power, computing scale, and supported bit width types. If the computation tasks of each network layer of the model are distributed to the three processing cores in a fixed order or on an average basis, high-computation layers are often scheduled to low-performance cores, while low-computation layers occupy high-performance cores, resulting in unbalanced load between cores, decreased overall throughput, and increased critical path latency. Therefore, it is necessary to dynamically select the most suitable target processing core for each layer based on the target quantization precision (e.g., INT4 and INT8 correspond to different computing units) and computational load of each network layer, so as to achieve load balancing between cores and make full use of each processing core. Optionally, the computational cost of each network layer of the target model can be quantized using floating-point operations per second (FLOPs).

[0032] In this step, the hierarchical dependencies between network layers in the target model can be further considered to collaboratively optimize the allocation of processing cores for each network layer. Specifically, in the computational model, some network layers have sequential dependencies during computation, meaning that the computation of a later network layer requires the output of a previous network layer as input. These network layers constitute a hierarchically dependent network layer group with continuous computational relationships. For multiple network layers with hierarchical dependencies, to avoid data transfer across processing cores and repeated writing of intermediate results back to external storage, it is preferable to allocate the entire network layer group to the same target processing core for execution. This ensures continuous data flow within the processing core, thereby reducing memory access overhead and inter-core communication latency.

[0033] Building upon this foundation, for network layer groups with hierarchical dependencies, the target processing core can be prioritized based on the network layer with the highest target quantization accuracy or the largest computational load. The remaining network layers with the same hierarchical dependency are then synchronously allocated to this target processing core. This ensures that computationally intensive and high-precision-requirement network layers receive priority access to more powerful processing cores, while the remaining network layers share the core while maintaining continuous execution. Thus, by comprehensively considering target quantization accuracy, computational load, and hierarchical dependencies, overall collaborative optimization of network layer processing core allocation is achieved. This avoids additional data transfer caused by inter-layer splitting and improves the utilization efficiency of high-computation-load processing cores, thereby effectively reducing overall inference latency and improving the computing power utilization of the target deployment platform.

[0034] The computational task allocation information includes the identification information of the network layer and the identification information of the target processing core, and contains the correspondence between the network layer identification information and the target processing core identification information. Therefore, after loading the target model and computational task allocation information into the target deployment platform, the platform can allocate network layer computational tasks to different processing cores according to the correspondence between the network layer identification information and the processing core identification information.

[0035] S104. Load the target model and the computing task allocation information into the target deployment platform.

[0036] Specifically, the obtained target model is converted into a dedicated model format supported by the target deployment platform and then loaded onto the target deployment platform. When the target deployment platform is RK3588, the model building interface of RKNN Toolkit is called to directly convert the target model into the supported .rknn format, skipping the ONNX intermediate format, achieving direct format conversion, and then the .rknn format is directly loaded onto the RK3588 platform.

[0037] In computational models, convolution operations typically consist of a sequence of multiple complementary operators, such as a convolution (Conv) operator followed by a batch normalization (BN) operator, or a convolution operator followed by activation operators (e.g., SiLU, ReLU). The convolution operator performs weighted convolution calculations on the input feature map, extracting spatial features through the convolution kernel; it is the most important numerical computation unit in the entire computation chain. The batch normalization operator, executed after convolution, standardizes the convolution output, stabilizing its distribution and improving the convergence of training and inference. Activation operators apply nonlinear transformations to the output of convolution or BN, enabling the network to model complex patterns.

[0038] Network layers containing convolution operations typically have at least two consecutive operators. In the original model, these operators are treated as independent operator nodes. If directly mapped to the target deployment platform, intermediate results (such as the computation results of convolution operators) require multiple reads and writes to DDR. Taking a convolution operator + activation operator as an example, after the convolution operator completes its computation, the result is written to DDR, and the activation operator reads data from DDR for computation. This process requires multiple reads and writes to DDR, resulting in high memory access overhead and high latency. At the same time, switching between operators also incurs additional hardware scheduling costs, preventing the full utilization of the processing core's computing power and resources.

[0039] In this application, based on steps S101-S104, firstly, the target quantization precision for each network layer in the original model is determined to achieve a differentiated quantization strategy; then, multi-operator fusion is performed on network layers containing at least two consecutive operators, that is, at least two consecutive operators in each target network layer are fused into a target fusion operator; finally, based on the target quantization precision and computational cost of each network layer in the target model, a suitable target processing kernel is determined for each network layer. In the above process, the appropriate target quantization precision is first determined according to the computational complexity and information content of different network layers to avoid the precision loss caused by uniform bit-width quantization, thus ensuring its... After deployment on the target deployment platform, the system ensures computational accuracy while significantly improving the overall computing power utilization of the platform. Secondly, by performing multi-operator fusion on network layers containing at least two consecutive operators, the consecutive operators in the original model are rewritten into a single target fusion operator, effectively reducing DDR memory access overhead, shortening the operator chain execution path, reducing inter-core data transfer latency, and improving the execution efficiency of the processing cores. Simultaneously, based on the target quantization accuracy and computational load of each network layer in the target model, appropriate processing cores are allocated to each network layer, achieving cross-core load balancing and optimal execution paths, eliminating problems such as large load differences and idle computing power among processing cores. In this application, through collaborative strategies such as differentiated quantization, multi-operator fusion, and inter-core task allocation, the target model maintains detection accuracy while significantly improving inference throughput and resource utilization, effectively reducing inference latency on edge devices, and effectively improving inference real-time performance, achieving efficient deployment suitable for practical target recognition scenarios.

[0040] In some embodiments, determining the target quantization accuracy corresponding to each network layer in the original model includes: For the network layers containing convolutional operations in the original model, the layer entropy value of each network layer is calculated, and the target quantization accuracy is determined based on its corresponding layer entropy value. For network layers in the original model that do not contain convolution operations, the basic quantization precision is determined to be the corresponding target quantization precision; Among them, the layer entropy value is positively correlated with the target quantization accuracy.

[0041] Specifically, an entropy threshold can be preset, and this preset entropy threshold can be used to determine the target quantization accuracy of each network layer. The preset entropy threshold can be obtained through multiple experiments and tests. Taking the RK3588 platform as an example, 10 sets of comparative experiments can be conducted on the COCO dataset to test the impact of different thresholds on mAP and inference speed, thus determining the preset entropy threshold. The COCO (CommonObjects in Context) dataset is an open-source computer vision dataset built by the Microsoft team, containing 80 target classes.

[0042] When the layer entropy value When the corresponding network layer is a critical layer in the original model, such as the detector head, feature fusion layer, or some intermediate convolutional layers, a high target quantization precision is determined for the corresponding network layer to ensure the model's detection accuracy. For example, a target quantization precision of INT8 is chosen to effectively preserve the features of the critical layer, with mAP loss ≤1%. When the specified value is used, it indicates that the corresponding network layer is a non-critical layer in the original model, such as some intermediate convolutional layers, input layers, or shallow convolutional layers. In this case, the target quantization precision of the corresponding network layer can be set to be lower, for example, INT4, thereby improving the inference speed of the model. Optionally, the preset entropy threshold can be 0.7, or it can be set to other values ​​according to the specific target deployment platform, such as 0.8, 0.75, etc., without any specific restrictions.

[0043] For network layers in the original model that do not contain convolutional operations, a base quantization precision is determined as the corresponding target quantization precision. For example, the base quantization precision INT4 can be directly determined as the target quantization precision for the corresponding network layer. After determining the target quantization precision for each network layer in the original model, a "Network Layer ID - Target Quantization Precision" configuration file is generated. This file can be in JSON format, and the quantization operation of the original model can be achieved by directly calling the corresponding configuration file during quantization.

[0044] In this embodiment, the layer entropy value is calculated for each network layer containing convolutional operations in the original model, and the target quantization precision is determined based on the layer entropy value. This allows different network layers to adopt differentiated quantization bit width configurations according to their feature complexity. Network layers with higher layer entropy values ​​typically carry richer feature patterns and are more sensitive to quantization errors. Therefore, assigning them higher quantization precision can significantly reduce the cumulative error introduced by quantization and effectively ensure detection accuracy. Conversely, network layers with lower layer entropy values ​​have more stable feature distributions and are more tolerant of quantization perturbations. Using lower quantization precision can effectively compress computational load and storage bandwidth, thereby improving inference speed and reducing power consumption. At the same time, for network layers that do not contain convolutional operations, a uniform base quantization precision is used as their target quantization precision, which can reduce unnecessary precision allocation overhead and make the overall quantization strategy simpler and easier to implement. Thus, a balance between accuracy preservation and computational efficiency can be maintained, making full use of the computing resources of the target deployment platform (such as RK3588) and significantly improving the problems of excessive accuracy loss or wasted computing power caused by traditional uniform quantization strategies.

[0045] In some embodiments, calculating the layer entropy value of each network layer includes: For each network layer that contains convolutional operations, obtain its output feature map; Based on its corresponding output feature map, the entropy value of each channel in the network layer is calculated, and the average value of the entropy values ​​of all channels in the network layer is determined as the layer entropy value of the network layer.

[0046] Specifically, the output feature maps of each network layer can be obtained by calling the model deployment tool. For example, the feature extraction interface (rknn.extract_feature()) of RKNNToolkit can be called to obtain the output feature maps of each network layer. The output feature map data of each network layer can be stored in a temporary cache in the structure of "network layer ID-channel ID-pixel value" for subsequent calculation.

[0047] The specific calculation process for the layer entropy value of each network layer containing convolutional operations is as follows: Iterate through all channels of the network layer, perform histogram statistics on the pixel values ​​of the feature map for each channel, and count the occurrence frequency of each value (pixel value range: 0-255). Then, calculate the entropy value of each channel and take the average of the entropy values ​​of all channels in the network layer as the layer entropy value. The entropy value of each channel can be calculated using the following formula: ; in, The entropy value. n This represents the total number of pixels in a single channel (e.g., when the feature map size is 64×64, n=64×64=4096). For the first in this channel i The probability distribution of pixel values It is the ratio of the frequency of a pixel value to the total number of pixels.

[0048] In some embodiments, the at least two consecutive operators include the original convolution operator and the batch normalization operator, and the target fusion operator is obtained by the following methods: For each target network layer, extract the original convolution parameters from its original convolution operator and the batch normalization parameters from its batch normalization operator; New convolution parameters are generated based on their corresponding convolution parameters and corresponding batch normalization parameters. The target fusion operator is obtained by substituting the original convolution parameters into the original convolution operator with the new convolution parameters.

[0049] Specifically, the original convolution parameters include weights. Bias Batch normalization parameters include the mean. ,variance Scaling factor and offset The new convolution parameters include new weights. With the new bias It can be calculated using the following formula: ; ; in, = To avoid the denominator being 0.

[0050] New weights With the new bias Replace the original convolution parameter weights Bias Substituting the values ​​into the original convolution operator yields the target fusion operator, thereby achieving parameter-level fusion of the operators. The target fusion operator can directly replace the calculation of the original convolution operator and the batch normalization operator. After the model is deployed and running, the target fusion operator is calculated directly, eliminating the need to execute the original convolution operator and the batch normalization operator.

[0051] After parametrically fusing the convolution and batch normalization operators for each target network layer to obtain the target fusion operator, corresponding hardware execution instructions can be generated. Taking the RK3588 target deployment platform as an example, after obtaining the target fusion operator, the target fusion operator is bound to the NPU's Conv+BN composite instruction through the custom operator interface of the RKNN Toolkit, and the corresponding hardware execution unit is specified to generate hardware instructions that can be directly run on the target deployment platform. With the help of operator fusion and hardware-level instruction binding, the data transfer process that originally required "Conv→DDR→BN" is simplified to "target fusion operator→output buffer". The intermediate results do not need to be written back to DDR, which reduces the number of data interactions by about 50% and reduces the operator chain execution latency from 15ms to 8ms.

[0052] In this embodiment, by fusing the parameters of the original convolution operator and batch normalization operator of the target network layer, the originally separately executed convolution and batch normalization operations are rewritten into a single convolution operator. This effectively reduces the number of operators, avoids the intermediate memory access process of writing convolution results back and forth to DDR, and allows data to flow directly within the same execution unit. This significantly reduces data transfer overhead, reduces execution latency during the inference stage, improves the computing power utilization of the target deployment platform, and simultaneously improves the model's running efficiency and response speed on edge devices.

[0053] In some embodiments, the at least two consecutive operators include a primary convolution operator and an activation operator, and the target fusion operator is obtained by the following methods: For each target network layer, the computation nodes of its original convolution operator and the computation nodes of its activation operator are merged into one computation node to obtain the target fusion operator.

[0054] Specifically, for a target network layer containing at least two consecutive operators, the computation nodes of its original convolution operators and activation operators are merged into a single computation node in the computation graph to obtain a target fusion operator that replaces the original operator chain. This target fusion operator is a fusion computation node that can be executed continuously within the same hardware execution unit of the target deployment platform. This allows the convolution output to directly enter subsequent activation computations without having to write back to external DDR storage, reducing the number of memory accesses for intermediate feature data, lowering data transfer overhead, and effectively compressing the overall inference latency.

[0055] In deep learning inference, the model is typically represented as a computation graph consisting of multiple computation nodes. Each node corresponds to an operator (such as convolution, batch normalization, activation, etc.), and each operator node is executed by an independent execution unit in hardware. Taking the unfused "Conv→activation" structure as an example, the convolution operator node first runs in the convolution execution unit of the NPU, and the convolution output is written to external DDR. Subsequently, the activation operator node reads the result from DDR and completes the computation in the activation execution unit. This separate execution method across nodes and execution units results in multiple redundant data transfers and storage accesses.

[0056] After fusing the computation nodes of the original convolution operator and the activation operator to obtain the target fused operator, the computations that originally belonged to different operator nodes are merged into a single fused computation node. This node can perform convolution and activation operations continuously in one go by the same hardware execution unit within the NPU, without needing to write back to DDR. For the computation graph, the original two sequential nodes are replaced by a single fused node with a simpler structure, reducing the graph depth and data dependencies between nodes. For hardware execution, the fused node allows the convolution result to be directly activated within the execution unit, significantly reducing memory access pressure, improving computational pipeline efficiency, and thus significantly improving overall inference throughput and reducing execution latency.

[0057] After fusing the original convolution and activation operators of each target network layer to obtain the target fusion operator, corresponding hardware execution instructions can be further generated. Taking the target deployment platform as RK3588 as an example, after obtaining the target fusion operator, the target fusion operator is bound to the NPU's Conv+SiLU composite instruction through the custom operator interface of RKNN Toolkit, and the corresponding hardware execution unit is specified to generate hardware instructions that can be directly run on the target platform. With the help of operator fusion and hardware-level instruction binding, the data transfer process that originally required "Conv→DDR→activation layer" is simplified to internal flow within the fusion operator. The intermediate results do not need to be written back to DDR, eliminating the independent memory access step of the SiLU activation layer and effectively reducing the number of data interactions.

[0058] In this embodiment, by merging the computation nodes of the original convolution operators and activation operators of the target network layer into one computation node, the content originally executed by different hardware execution units is merged into the same hardware execution unit for computation, reducing the number of DDR accesses, improving data transmission efficiency, thereby reducing model computation overhead and increasing model push speed.

[0059] In some embodiments, the at least two consecutive operators include the original convolution operator, the batch normalization operator, and the activation operator, and the target fusion operator is obtained by the following methods: For each target network layer, extract the original convolution parameters from its original convolution operator and the batch normalization parameters from its batch normalization operator; New convolution parameters are generated based on their corresponding convolution parameters and corresponding batch normalization parameters. The new convolution parameters are substituted into the original convolution operator to replace the original convolution parameters, thus obtaining the target convolution operator. The computation nodes of the target convolution operator and the computation nodes of the activation operator are merged into one computation node to obtain the target fusion operator.

[0060] Specifically, when the target network layer includes a continuous operator chain structure of the original convolution operator, batch normalization operator, and activation operator, it is possible to fuse only the original convolution operator and the batch normalization operator, or to fuse the original convolution operator, the batch normalization operator, and the activation operator together, without any specific restrictions.

[0061] When it is necessary to simultaneously fuse the original convolution operator, batch normalization operator, and activation operator, new convolution parameters are first generated using the convolution parameters of the original convolution operator and the batch normalization parameters of the batch normalization operator. Then, the target convolution operator is obtained based on the new convolution parameters. Finally, the computation nodes of the target convolution operator and the activation operator are merged into a single computation node in the computation graph, thus obtaining the target fusion operator. The original data transfer process of "Conv→DDR→BN→DDR→Activation Operator" is simplified to an internal flow within the fusion operator, namely "Target Convolution Operator→Activation Operator→Output Buffer". The intermediate results do not need to be written back to DDR, eliminating the independent memory access step of the SiLU activation layer, effectively reducing the number of data interactions and DDR accesses, improving data transfer efficiency, effectively reducing model computational overhead, and improving model push speed.

[0062] In some embodiments, a data flow-level optimization strategy can also be configured to improve the execution efficiency of the fusion operator. Specifically, by configuring "enable_zero_copy=True" in RKNN-Toolkit2 to enable the data flow-level optimization strategy, and in conjunction with asynchronous pipelined processing (preprocessing / DMA transfer / inference parallelism), the target model, during loading, writes the data to be processed directly to the shared memory area of ​​the NPU and GPU via the DMA controller based on the AXI bus shared memory pool. The calculation results of the volume operator are not written to DDR, but are directly synchronized to the BN operator hardware unit through the cache coherence protocol, avoiding the data "computation-storage-read" cycle.

[0063] In some embodiments, the step of determining the target processing core corresponding to each network layer from multiple processing cores of the target deployment platform based on the target quantization accuracy and computational cost of each network layer of the target model, and obtaining computational task allocation information, includes: The processing core with the highest computing power among the multiple processing cores of the target deployment platform is identified as the first processing core; The first processing core is determined to be the target processing core of the network layer whose target quantization accuracy is higher than the preset accuracy. For the remaining network layers whose target quantization accuracy is lower than or equal to the preset accuracy, the target processing core is determined from multiple processing cores, with the difference between the total computation amount corresponding to each processing core being less than a first preset threshold as the allocation target, so as to obtain the computation task allocation information.

[0064] Specifically, the target deployment platform typically has multiple processing cores with varying computing capabilities, such as the three-core NPU of the RK3588. The peak computing power, parallelism, and supported instruction combinations of these different processing cores are not entirely consistent. Therefore, when allocating execution cores to each network layer of the target model, it is necessary to consider the computing power of different processing cores as well as the target quantization accuracy and computational load of the network layer itself, ensuring that the load difference between multiple processing cores is less than a first preset threshold. This first preset threshold can be 10%, or it can be 5%, 8%, 10%, 11%, 12%, 13%, 14%, 15%, or other values; there are no specific restrictions.

[0065] The processing core with the highest computational power among multiple processing cores is identified and designated as the first processing core. Since high quantization precision (such as INT8 or INT16) typically implies greater computational load and stricter numerical precision requirements, all network layers with target quantization precision higher than the preset precision are preferentially assigned to this first processing core to ensure inference accuracy and execution stability during critical computation stages. Then, for the remaining network layers with target quantization precision lower than or equal to the preset precision, allocation is performed with the goal of balancing the total computational load among the multiple processing cores.

[0066] The above process will be further illustrated using the RK3588's triple-core NPU as an example.

[0067] The RK3588 NPU has three processing cores with different computing capabilities, typically including one master core and two slave cores. The master core has higher computing power, wider vector units, and better memory bandwidth, making it suitable for executing high-precision, computationally intensive network layers. The slave cores have relatively lower computing power and are more suitable for executing network layers with lower computational requirements or lower quantization bit widths. Assume that in the target model's network layers, some layers have a target quantization precision of INT8 determined by previous steps, while others have a target quantization precision of INT4. Set INT4 as the preset precision, and the first preset threshold is 10%.

[0068] First, the processing core with the highest computational power among the three processing cores is designated as the first processing core, i.e., the NPU master core. Since the INT8 network layer has a higher quantization bit width, its computational load and precision requirements are higher than those of the INT4 layer. Therefore, all network layers with a target quantization precision higher than the preset precision (INT4) are preferentially assigned to the first processing core for execution; that is, the INT8 network layers are assigned to the NPU master core for processing. For network layers with a target quantization precision lower than or equal to the preset precision (i.e., the INT4 layer), the allocation target is that the difference between the total computational load corresponding to the master core and the two slave cores is less than 10%. In the actual allocation of a certain target model, the final allocation result is as follows: the total computational load of the master core is 2.8e8, the total computational loads of slave core 1 and slave core 2 are 2.7e8 and 2.6e8 respectively, and the difference in the total computational load among the three processing cores is less than 10%.

[0069] When it is necessary to consider the hierarchical dependencies of each network layer in the target model, the network layers of the target model can be grouped first, and then the target processing kernel of each network can be determined by considering the target quantization accuracy and computational cost of the network layers, as follows: The network layers with hierarchical dependencies in the target model are identified as network layer groups, and the network layer with the highest target quantization accuracy and / or the highest computational cost in each network layer group is identified as the core network layer. For each core network layer and each independent network layer without hierarchical dependencies, the target quantization precision for each network layer is determined based on the target quantization precision and computational cost. The specific execution is as follows: The processing core with the highest computing power among the multiple processing cores of the target deployment platform is identified as the first processing core; The first processing core is determined to be the target processing core of the independent network layer and the core network layer whose target quantization accuracy is higher than the preset accuracy; For all other network layers, the target processing cores corresponding to each processing core are determined from multiple processing cores, with each network layer in each network layer group being the same target processing core and the difference between the total computational workloads corresponding to each processing core being less than a first preset threshold as the allocation constraint condition. This yields the computational task allocation information.

[0070] Therefore, by comprehensively considering the target quantization accuracy, computational load, and hierarchical dependencies, the overall collaborative optimization of the allocation of processing cores in each network layer is achieved. This avoids the additional data transfer caused by inter-layer splitting and improves the utilization efficiency of high-computation-load processing cores, thereby effectively reducing overall inference latency and improving the computing power utilization of the target deployment platform.

[0071] In some embodiments, the computational cost of each network layer of the target model is calculated using the following methods: Since the network layer of the target model is a convolutional layer, the corresponding computational cost is determined based on the output feature map size, number of channels, and convolutional kernel size of the corresponding network layer.

[0072] Specifically, the computational cost of each network layer in the quantized model (i.e., the target model) is equivalently converted based on the quantization precision, and the computational cost of each network layer can be measured in FLOPs. The computational cost of a convolutional layer can be determined based on its output feature map size, number of channels, and kernel size. The computational cost under different quantization precisions is equivalently converted based on the quantization precision. For example, the FLOPs of INT4 are approximately 1 / 4 of those of INT8. That is, the computational cost of an INT8 convolutional layer = number of channels × feature map size × kernel size, and the computational cost of an INT4 convolutional layer is 1 / 4 of that of INT8.

[0073] In some embodiments, it also includes: A dynamic task migration strategy is constructed based on the computing task allocation information, and the dynamic task migration strategy is loaded into the target deployment platform; The dynamic task migration strategy is as follows: when the target deployment platform is running the target model, if the real-time load rate of at least one of the multiple processing cores reaches a second preset threshold, some of the computing tasks in the corresponding processing core will be migrated to other processing cores.

[0074] Specifically, a dynamic task migration strategy is used to achieve load balancing adjustment of each processing core. The second preset threshold can be 90%, 85%, 88%, 95%, etc., or can be set to other values ​​according to the actual situation; there are no specific restrictions.

[0075] During the actual operation of the target deployment platform, the real-time utilization rate of each processing core is monitored in real time. When the utilization rate of a certain processing core reaches a second preset threshold (e.g., ≥90%), the first processing core (e.g., the main core) can migrate a portion of the low computational network layers of that processing core (e.g., shallow convolutional blocks with quantization accuracy of INT4) to a processing core with significantly lower real-time utilization (e.g., a processing core with ≤70%). This dynamically balances the workload among multiple processing cores, ensuring that the load difference among multiple processing cores remains within a preset range (e.g., ≤10%), further improving overall inference efficiency and response stability.

[0076] In some embodiments, the dynamic task migration strategy may further be: When the target deployment platform runs the target model, if the cache hit rate of at least one of the multiple processing cores is lower than the first preset cache threshold, some computational tasks with low computational load but high feature reuse rate in that processing core will be migrated to a processing core with a cache hit rate higher than the second preset cache threshold.

[0077] When the target deployment platform runs the target model, the cache hit rate of each processing core is monitored in real time. If the cache hit rate of at least one processing core is lower than a first preset cache threshold (e.g., 75%), the computational tasks with lower computational load and higher feature reuse rate in that processing core are migrated to other processing cores with a cache hit rate higher than a second preset cache threshold (e.g., 85%) to reduce the external memory access pressure on that processing core. During the migration process, cross-core data synchronization is automatically completed based on the cache consistency protocol to avoid duplicate calculations caused by cache inconsistency, thereby achieving dynamic load optimization among processing cores during inference.

[0078] After loading the computation task allocation information into the target deployment platform, the platform needs to use this information to schedule and execute network layer tasks on the multi-core NPU. Taking the RK3588 as an example, its NPU has three processing cores, and the platform can use different inter-core communication and scheduling mechanisms to execute the allocated computation tasks. In one implementation, the Mailbox (the inter-core communication hardware unit built into the RK3588 chip, including task sending, status, and progress registers) mechanism can be used to complete the task distribution and synchronization between the master core and slave cores. Based on the computation task allocation information, the master core sends a task packet containing the layer ID, computation address, and quantization precision to each slave core through the Mailbox's task sending register; after receiving the packet, the slave core sends a "ready" signal back through its status register, and the master core synchronously starts the three-core computation accordingly, and uses the progress register to monitor the execution status of each core in real time to trigger subsequent tasks in batches. This method has a simple structure and is suitable for inference scenarios with relatively fixed task flows.

[0079] In another implementation, more flexible inter-core scheduling can be achieved based on the Remote Processor Messaging (RPMsg) virtual communication channel provided by the RK3588. In this approach, the three-core NPU can be functionally divided into inference cores (high priority), preprocessing cores (medium priority), and post-processing cores (low priority), and inter-core state synchronization is achieved using RPMsg message queues, replacing the semaphore communication of traditional Mailboxes. During scheduling, considering the computational load, quantization accuracy, and core priority of each network layer, high-priority tasks are preferentially assigned to idle cores for execution. Tasks can be transferred in real time between different cores via the "rpsg_send() / rpmsg_recv()" interface, allowing idle cores to quickly take over remaining tasks, further improving multi-core utilization and throughput performance. This method offers greater flexibility and scheduling efficiency, significantly improving overall inference speed.

[0080] In another alternative implementation, a more refined inter-core load balancing mechanism can be achieved based on Dynamic Voltage and Frequency Scaling (DVFS). This approach first establishes a computational load-frequency mapping table. After loading the computational task allocation information, the target deployment platform allocates basic tasks to multiple processing cores based on the computational load of each network layer. Subsequently, the DVFS module monitors the load of each processing core in real time. When a processing core has a high load, its operating frequency is automatically increased, while the frequency of a less loaded processing core is decreased to improve overall energy efficiency. Simultaneously with dynamic frequency adjustment, the platform synchronizes task migration via interrupt signals. If a high-load core still cannot meet real-time inference requirements after frequency increase, some low-priority or low-computation network layer tasks on that core can be migrated to other processing cores with higher idle rates. Compared to scheduling methods that rely solely on Mailboxes, this approach achieves coordinated scheduling of "frequency adjustment + task migration," improving the overall utilization of the three cores while maintaining inference stability.

[0081] Taking the RK3588 as the target deployment platform as an example, the computational model deployment method of this application can significantly improve the inference performance of the target recognition model at the edge by combining the hardware characteristics of the RK3588 (such as 6TOPS NPU, 8K video processing capability, and multi-interface expansion capability), achieving a comprehensive optimization effect of high precision, low latency, and low power consumption. Since the target model has already completed fine-grained quantization precision allocation, operator-level fusion, and processing kernel mapping before deployment, the target model can make fuller use of the RK3588's three-core NPU architecture and composite instruction system, showing significant advantages in edge application scenarios.

[0082] In smart cockpit scenarios, the RK3588 supports multi-screen display and HDMI output. The multi-core load balancing strategy of this application enables the model to simultaneously complete "camera video stream inference + real-time interface rendering" during operation, avoiding the stuttering caused by uneven load between cores in traditional deployments. By reducing memory access through operator fusion, the deployment method of this application can stably achieve the "1080p@60FPS real-time" requirement in automotive scenarios and maintain detection accuracy under low power consumption conditions. It can serve scenarios such as automotive DMS (driver status monitoring, such as fatigue / distraction recognition) and cockpit target tracking (passenger and object positioning).

[0083] In 8K ultra-high-definition intelligent conferencing scenarios, the 8K decoding capability of the RK3588 can work synergistically with the NPU optimization method of this application. This application reduces data transfer through hardware-level fusion of "convolution-Batch Normalization-Activation," enabling parallel inference of multiple video streams (e.g., four 4K streams) and compressing post-processing latency such as subtitle overlay from tens of milliseconds to less than 20ms, meeting the conferencing scenario's requirements for smooth real-time interaction, multi-target identity recognition, and real-time subtitle generation. Through dynamic processing core scheduling, the deployment method of this application can maintain stable throughput under parallel input of multiple video sources, supporting functions such as speaker tracking, real-time subtitle generation, and identity recognition.

[0084] The target model deployed using the method described in this application can achieve at least the following technical effects: (1) Accuracy and speed are optimized in tandem to meet the dual requirements of real-time performance and accuracy. The complexity of model layer features is quantized by channel entropy analysis, and the quantization precision is dynamically allocated according to the layer entropy value of the network layer. For example, INT8 / INT4 / mixed precision is used, with INT8 used to maintain precision for critical layers and INT4 used to improve speed for non-critical layers. As a result, the mAP@0.5 loss is controlled within 1%, the inference speed is improved by 41.5% (single frame latency is reduced from 65ms to 38ms), and the utilization rate of the INT4 computing unit of RK3588 is increased from 30% to 85%, which can meet the core scenario requirements of intelligent security real-time detection (≥30FPS) and target tracking dynamic locking.

[0085] (2) Hardware performance is fully activated, balancing high performance and low power consumption. Hardware adaptation was optimized for the characteristics of the RK3588 NPU. On the one hand, the data transfer latency between cores was reduced by 47% through parameter fusion of convolution and BN operators and hardware-level composite instructions. On the other hand, the Mailbox module was used to achieve load balancing of the three cores, with the load difference between the three groups ≤10%, and the high-load core tasks could be adjusted in real time, which increased the NPU computing power utilization from 58% to 89%, and the total computing power of 6 TOPS was fully activated. At the same time, the power consumption of the whole machine was kept below 12W, which met the power supply and heat dissipation constraints of edge devices.

[0086] (3) Deployment efficiency is greatly improved, and the technical threshold and cost are reduced. In this application, one-click quantization deployment was achieved. Taking RK388 as the target deployment platform, an end-to-end toolchain can be built based on the RKNN Toolkit to automate the process of "inputting a floating-point model → outputting a deployed model," directly converting from a PyTorch model to an RKNN model, skipping the ONNX intermediate format, avoiding the 0.5% mAP loss, integrating fully automated logic, eliminating the need for manual debugging of quantization parameters, reducing model deployment time from 1 hour to 15 minutes, reducing debugging costs by 90%, and requiring only one command from ordinary developers to complete the operation without needing to understand the underlying details of RK3588, significantly accelerating the iteration of edge AI projects.

[0087] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0088] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0089] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a computational model deployment device.

[0090] refer to Figure 2 The device includes: The quantization accuracy calculation module 201 is used to determine the target quantization accuracy corresponding to each network layer in the original model, and to determine the network layer containing at least two consecutive operators as the target network layer. The quantization fusion module 202 is used to quantize each network layer of the original model based on the target quantization accuracy, and combine the target fusion operator obtained by performing fusion on the at least two consecutive operators of each target network layer to obtain the target model; The inter-core task allocation module 203 is used to determine the target processing core corresponding to each network layer from multiple processing cores of the target deployment platform based on the target quantization accuracy and computational load of each network layer of the target model, and obtain computational task allocation information. The loading module 204 is used to load the target model and the computing task allocation information into the target deployment platform.

[0091] In some embodiments, the quantization accuracy calculation module 201 is further configured to: For the network layers containing convolutional operations in the original model, the layer entropy value of each network layer is calculated, and the target quantization accuracy is determined based on its corresponding layer entropy value. For network layers in the original model that do not contain convolution operations, the basic quantization precision is determined to be the corresponding target quantization precision; Among them, the layer entropy value is positively correlated with the target quantization accuracy.

[0092] In some embodiments, calculating the layer entropy value of each network layer includes: For each network layer that contains convolutional operations, obtain its output feature map; Based on its corresponding output feature map, the entropy value of each channel in the network layer is calculated, and the average value of the entropy values ​​of all channels in the network layer is determined as the layer entropy value of the network layer.

[0093] In some embodiments, the at least two consecutive operators include the original convolution operator and the batch normalization operator, and the target fusion operator is obtained by the following methods: For each target network layer, extract the original convolution parameters from its original convolution operator and the batch normalization parameters from its batch normalization operator; New convolution parameters are generated based on their corresponding convolution parameters and corresponding batch normalization parameters. The target fusion operator is obtained by substituting the original convolution parameters into the original convolution operator with the new convolution parameters.

[0094] In some embodiments, the at least two consecutive operators include a primary convolution operator and an activation operator, and the target fusion operator is obtained by the following methods: For each target network layer, the computation nodes of its original convolution operator and the computation nodes of its activation operator are merged into one computation node to obtain the target fusion operator.

[0095] In some embodiments, the at least two consecutive operators include the original convolution operator, the batch normalization operator, and the activation operator, and the target fusion operator is obtained by the following methods: For each target network layer, extract the original convolution parameters from its original convolution operator and the batch normalization parameters from its batch normalization operator; New convolution parameters are generated based on their corresponding convolution parameters and corresponding batch normalization parameters. The new convolution parameters are substituted into the original convolution operator to replace the original convolution parameters, thus obtaining the target convolution operator. The computation nodes of the target convolution operator and the computation nodes of the activation operator are merged into one computation node to obtain the target fusion operator.

[0096] In some embodiments, the inter-core task allocation module 203 is further configured to: The processing core with the highest computing power among the multiple processing cores of the target deployment platform is identified as the first processing core; The first processing core is determined to be the target processing core of the network layer whose target quantization accuracy is higher than the preset accuracy. For the remaining network layers whose target quantization accuracy is lower than or equal to the preset accuracy, the target processing core is determined from multiple processing cores, with the difference between the total computation amount corresponding to each processing core being less than a first preset threshold as the allocation target, so as to obtain the computation task allocation information.

[0097] In some embodiments, the computational cost of each network layer of the target model is calculated using the following methods: Since the network layer of the target model is a convolutional layer, the corresponding computational cost is determined based on the output feature map size, number of channels, and convolutional kernel size of the corresponding network layer.

[0098] In some embodiments, it also includes: The migration strategy module is used to construct a dynamic task migration strategy based on the computing task allocation information, and load the dynamic task migration strategy into the target deployment platform; The dynamic task migration strategy is as follows: when the target deployment platform is running the target model, if the real-time load rate of at least one of the multiple processing cores reaches a second preset threshold, some of the computing tasks in the corresponding processing core will be migrated to other processing cores.

[0099] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0100] The apparatus described above is used to implement a corresponding computational model deployment method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0101] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a computing model deployment method as described in any of the above embodiments.

[0102] Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0103] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0104] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0105] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0106] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0107] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0108] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0109] The electronic devices described above are used to implement a corresponding computational model deployment method in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0110] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute a computing model deployment method as described in any of the above embodiments.

[0111] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0112] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute a computing model deployment method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0113] Based on the same concept, corresponding to any of the above embodiments, this application also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to perform the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0115] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0116] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0117] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0118] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0119] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0120] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0121] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the claims of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for deploying a computational model, characterized in that, include: Determine the target quantization accuracy for each network layer in the original model, and identify the network layer containing at least two consecutive operators as the target network layer; Based on the target quantization accuracy, each network layer of the original model is quantized, and the target fusion operator is obtained by performing fusion on at least two consecutive operators of each target network layer to obtain the target model. Based on the target quantization accuracy and computational cost of each network layer of the target model, the target processing core corresponding to each network layer is determined from multiple processing cores of the target deployment platform to obtain computational task allocation information; The target model and the computing task allocation information are loaded into the target deployment platform.

2. The computational model deployment method according to claim 1, characterized in that, The determination of the target quantization accuracy corresponding to each network layer in the original model includes: For the network layers containing convolutional operations in the original model, the layer entropy value of each network layer is calculated, and the target quantization accuracy is determined based on its corresponding layer entropy value. For network layers in the original model that do not contain convolution operations, the basic quantization precision is determined to be the corresponding target quantization precision; Among them, the layer entropy value is positively correlated with the target quantization accuracy.

3. The computational model deployment method according to claim 2, characterized in that, The calculation of the layer entropy value for each network layer includes: For each network layer that contains convolutional operations, obtain its output feature map; Based on its corresponding output feature map, the entropy value of each channel in the network layer is calculated, and the average value of the entropy values ​​of all channels in the network layer is determined as the layer entropy value of the network layer.

4. The computational model deployment method according to claim 1, characterized in that, The at least two consecutive operators include the original convolution operator and the batch normalization operator, and the target fusion operator is obtained by the following methods: For each target network layer, extract the original convolution parameters from its original convolution operator and the batch normalization parameters from its batch normalization operator; New convolution parameters are generated based on their corresponding convolution parameters and corresponding batch normalization parameters. The target fusion operator is obtained by substituting the original convolution parameters into the original convolution operator with the new convolution parameters.

5. The computational model deployment method according to claim 1, characterized in that, The at least two consecutive operators include the original convolution operator and the activation operator, and the target fusion operator is obtained by the following methods: For each target network layer, the computation nodes of its original convolution operator and the computation nodes of its activation operator are merged into one computation node to obtain the target fusion operator.

6. The computational model deployment method according to claim 1, characterized in that, The at least two consecutive operators include the original convolution operator, the batch normalization operator, and the activation operator. The target fusion operator is obtained through the following methods: For each target network layer, extract the original convolution parameters from its original convolution operator and the batch normalization parameters from its batch normalization operator; New convolution parameters are generated based on their corresponding convolution parameters and corresponding batch normalization parameters. The new convolution parameters are substituted into the original convolution operator to replace the original convolution parameters, thus obtaining the target convolution operator. The computation nodes of the target convolution operator and the computation nodes of the activation operator are merged into one computation node to obtain the target fusion operator.

7. The computational model deployment method according to claim 1, characterized in that, Based on the target quantization accuracy and computational cost of each network layer of the target model, the target processing core corresponding to each network layer is determined from multiple processing cores of the target deployment platform to obtain computational task allocation information, including: The processing core with the highest computing power among the multiple processing cores of the target deployment platform is identified as the first processing core; The first processing core is determined to be the target processing core of the network layer whose target quantization accuracy is higher than the preset accuracy. For the remaining network layers whose target quantization accuracy is lower than or equal to the preset accuracy, the target processing core is determined from multiple processing cores, with the difference between the total computation amount corresponding to each processing core being less than a first preset threshold as the allocation target, so as to obtain the computation task allocation information.

8. The computational model deployment method according to claim 1, characterized in that, The computational complexity of each network layer in the target model is calculated using the following methods: Since the network layer of the target model is a convolutional layer, the corresponding computational cost is determined based on the output feature map size, number of channels, and convolutional kernel size of the corresponding network layer.

9. A computational model deployment method according to claim 1, characterized in that, Also includes: A dynamic task migration strategy is constructed based on the computing task allocation information, and the dynamic task migration strategy is loaded into the target deployment platform; The dynamic task migration strategy is as follows: when the target deployment platform is running the target model, if the real-time load rate of at least one of the multiple processing cores reaches a second preset threshold, some of the computing tasks in the corresponding processing core will be migrated to other processing cores.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the computational model deployment method as described in any one of claims 1 to 9.

Images