Model optimization method and device, terminal equipment and computer program product
By performing performance analysis and optimization on the chip model, configuring the candidate optimization layer as the SRAM-first mode, and combining operator and pipeline fusion rules, the problem of high-computational-load models occupying DDR bandwidth was solved, thereby achieving DDR bandwidth optimization and improving task processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2026-03-27
AI Technical Summary
High-computational-capacity models deployed on chips tend to consume high DDR bandwidth, leading to overall system performance bottlenecks, which existing technologies have not been able to effectively address.
By performing performance analysis on the model to be optimized, candidate optimization layers are identified and configured as SRAM-preferred modes, which prioritize storing intermediate data in SRAM. Combined with operator fusion and pipeline fusion rules, the model structure is optimized to reduce DDR bandwidth usage.
This reduces the model's DDR bandwidth usage, avoids frequent DDR read/write operations, and improves task processing efficiency and system stability.
Smart Images
Figure CN121743032A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer data processing technology, and in particular relates to a model optimization method, apparatus, terminal equipment and computer program product. Background Technology
[0002] In fields such as intelligent driving and in-vehicle intelligent devices, it is necessary to deploy multi-task models (e.g., models that perform tasks such as object detection, driving status monitoring, and facial recognition) on embedded chips to achieve complex business functions. In such scenarios, the chip's computing power (Neural Processing Unit (NPU) performance) and bandwidth (Double Data Rate Synchronous Dynamic Random Access Memory (DDR) data transfer rate) work together to determine the model's task processing efficiency.
[0003] Due to the high algorithmic accuracy requirements of the aforementioned business needs, computationally intensive models are gradually replacing traditional low-computational-intensive models. However, computationally intensive models deployed on chips tend to consume significant DDR bandwidth, which can lead to performance bottlenecks in the overall system and even prevent it from processing tasks properly.
[0004] There is still no effective solution to the problem that high computational complexity models deployed on chips in related technologies consume a lot of DDR bandwidth. Summary of the Invention
[0005] This application provides a model optimization method, apparatus, terminal device, and computer program product to at least solve the problem in related technologies where high-computational-load models deployed on chips consume high DDR bandwidth.
[0006] In a first aspect, embodiments of this application provide a model optimization method, the method comprising: performing performance analysis on a model to be optimized deployed on a chip to determine candidate optimization layers in the model to be optimized; calculating the SRAM usage and double-rate synchronous dynamic random access memory (DDR) bandwidth savings of the candidate optimization layers after they are pre-configured as a Static Random Access Memory (SRAM) priority mode, wherein the SRAM priority mode indicates that intermediate data generated by the candidate optimization layers are preferentially stored in the SRAM of the chip; selecting multiple layers to be optimized from multiple candidate optimization layers in descending order of DDR bandwidth savings, such that the sum of the SRAM usage of the multiple layers to be optimized is less than or equal to the total storage capacity of the SRAM; and configuring the layers to be optimized as the SRAM priority mode.
[0007] In some embodiments, the step of performing performance analysis on the model to be optimized deployed on the chip and determining candidate optimization layers in the model to be optimized includes: performing performance analysis on the model to be optimized to obtain the number of DDR access clock cycles and the number of neural network processing unit (NPU) computation clock cycles required for each network layer in the model to process each frame of data at a preset frequency; and marking the network layer as a candidate optimization layer if the ratio of the number of DDR access clock cycles to the number of NPU computation clock cycles of the network layer is greater than a first preset threshold.
[0008] In some embodiments, the SRAM usage represents the storage space required to process each frame of data after the candidate optimization layer is configured to the SRAM priority mode; before configuring the layer to be optimized to the SRAM priority mode, the method further includes: obtaining the processing frame rate of the model to be optimized; and setting the refresh period of the SRAM based on the processing frame rate so that the refresh period is equal to the time required for the model to be optimized to process one frame of data.
[0009] In some embodiments, after configuring the layer to be optimized to the SRAM priority mode, the method further includes: calculating the sum of the DDR bandwidth savings of all the layers to be optimized; if the sum of the DDR bandwidth savings is greater than or equal to a second preset threshold, determining that the model to be optimized has completed SRAM optimization, and obtaining a first model.
[0010] In some embodiments, after determining that the model to be optimized has completed SRAM optimization and obtained a first model, the method further includes: obtaining the operator fusion rule of the chip from the chip's configuration file; based on the operator fusion rule, fusing at least two network layers in the first model that match the operator fusion rule into a fused network layer to obtain a second model, wherein the inter-layer data generated by the network layers in the fused network layer processing each frame of data will be directly input to the next network layer in the fused network layer; matching the operator fusion rule means that: the at least two network layers are arranged consecutively and the at least two network layers conform to the operator fusion rule; or, the at least two network layers are not arranged consecutively and the at least two network layers conform to the operator fusion rule, and moving the network layer between the at least two network layers to before / after the at least two network layers does not affect the calculation results of the at least two network layers.
[0011] In some embodiments, after fusing at least two network layers matching the operator fusion rule in the first model into a fused network layer to obtain a second model based on the operator fusion rule, the method further includes: obtaining the pipeline fusion rule of the chip from the configuration file; fusing at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers in the second model into a fused pipeline based on the pipeline fusion rule to obtain a third model, wherein the inter-layer data generated by the network layers or fused network layers in the fused pipeline for processing each frame of data will be directly input to the next network layer or fused network layer in the fused pipeline; matching the pipeline fusion rule means that: the at least two network layers, or the at least one network layer and the at least one fused network layer, or the at least two fused network layers are arranged consecutively, and the at least two network layers, or the at least one network layer and the at least one fused network layer, or the at least two fused network layers conform to the pipeline fusion rule and have the same input / output dimensions.
[0012] In some embodiments, after fusing at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers in the second model that match the pipeline fusion rule into a fused pipeline to obtain a third model, the method further includes: obtaining the DDR access granularity of the chip from the configuration file; setting the number of input / output channels of each network layer, each fused network layer, or each fused pipeline in the third model to an integer multiple of the DDR access granularity to obtain an optimized model.
[0013] Secondly, embodiments of this application provide a model optimization apparatus, the apparatus comprising: an analysis module, configured to perform performance analysis on a model to be optimized deployed on a chip, and determine candidate optimization layers in the model to be optimized; a calculation module, configured to calculate the SRAM usage and DDR bandwidth savings of the candidate optimization layers after they are pre-configured as a Static Random Access Memory (SRAM) priority mode, wherein the SRAM priority mode indicates that intermediate data generated by the candidate optimization layers are preferentially stored in the SRAM of the chip; a selection module, configured to select multiple layers to be optimized from multiple candidate optimization layers in descending order of DDR bandwidth savings, such that the sum of the SRAM usage of the multiple layers to be optimized is less than or equal to the total storage capacity of the SRAM; and a configuration module, configured to configure the layers to be optimized as the SRAM priority mode.
[0014] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model optimization method described in any one of the first aspects above.
[0015] Fourthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the model optimization method described in any one of the first aspects to be executed.
[0016] Compared to related technologies, the model optimization method, apparatus, terminal device, and computer program product provided in this application can perform performance analysis on a model to be optimized deployed on a chip to determine candidate optimization layers in the model; can calculate the SRAM usage and DDR bandwidth savings after the candidate optimization layers are pre-configured in SRAM-priority mode, where SRAM-priority mode means that intermediate data generated by the candidate optimization layers is preferentially stored in the chip's SRAM; can select multiple optimization layers from multiple candidate optimization layers in descending order of DDR bandwidth savings, so that the sum of the SRAM usage of the multiple optimization layers is less than or equal to the total SRAM storage capacity; and can configure the optimization layers in SRAM-priority mode. In this way, multiple optimization layers with high DDR bandwidth savings can be configured in SRAM-priority mode, and the intermediate data generated by these optimization layers can be preferentially stored in the chip's SRAM, avoiding frequent read and write operations on the chip's DDR, thereby reducing the DDR bandwidth occupied by the model while avoiding SRAM data overflow. This application solves the problem in related technologies that high-computational-load models deployed on chips occupy a large amount of DDR bandwidth, and achieves the technical effect of reducing the DDR bandwidth occupied by models deployed on chips.
[0017] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart of a model optimization method according to an embodiment of this application; Figure 2This is a flowchart of a candidate optimization layer screening method according to an embodiment of this application; Figure 3 This is a flowchart of a model optimization method according to another embodiment of this application; Figure 4 This is a schematic diagram of the structure of a model optimization device according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. Detailed Implementation
[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0021] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0022] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0023] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."
[0024] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0025] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0026] In fields such as intelligent driving and in-vehicle intelligent devices, it is necessary to deploy multi-task models (e.g., models that perform tasks such as object detection, driving status monitoring, and facial recognition) on embedded chips to achieve complex business functions. In such scenarios, the chip's computing power (NPU performance) and bandwidth (DDR data transfer rate) work together to determine the model's task processing efficiency.
[0027] Due to the high algorithmic accuracy requirements of the aforementioned business needs, computationally intensive models are gradually replacing traditional low-computational-intensive models. However, computationally intensive models deployed on chips tend to consume significant DDR bandwidth, which can lead to performance bottlenecks in the overall system and even prevent it from processing tasks properly.
[0028] There is still no effective solution to the problem that high computational complexity models deployed on chips in related technologies consume a lot of DDR bandwidth.
[0029] In view of this, embodiments of this application provide a model optimization method that can perform performance analysis on a model to be optimized deployed on a chip to determine candidate optimization layers in the model; calculate the SRAM usage and DDR bandwidth savings after the candidate optimization layers are pre-configured in SRAM-priority mode, where SRAM-priority mode means that intermediate data generated by the candidate optimization layers is preferentially stored in the chip's SRAM; select multiple optimization layers from multiple candidate optimization layers in descending order of DDR bandwidth savings, such that the sum of the SRAM usage of the multiple optimization layers is less than or equal to the total SRAM storage capacity; and configure the optimization layers in SRAM-priority mode. In this way, multiple optimization layers with high DDR bandwidth savings can be configured in SRAM-priority mode, and the intermediate data generated by these optimization layers can be preferentially stored in the chip's SRAM, avoiding frequent reads and writes to the chip's DDR, thereby reducing the DDR bandwidth occupied by the model while avoiding SRAM data overflow. This application solves the problem in related technologies where high-computational-load models deployed on chips occupy high DDR bandwidth, achieving the technical effect of reducing the DDR bandwidth occupied by models deployed on chips.
[0030] The following will combine Figure 1 For an explanation of the model optimization method provided in one embodiment of this application, please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart of a model optimization method according to an embodiment of this application, such as... Figure 1 As shown, the method includes: Step S101: Perform performance analysis on the model to be optimized deployed on the chip to determine the candidate optimization layers in the model to be optimized.
[0031] In this embodiment, a performance analysis tool (e.g., Profiler) can be used to collect key parameters of each network layer in the model to be optimized. Based on the key parameters of each network layer, candidate optimization layers with a Double Data Rate Synchronous Dynamic Random Access Memory (DDR) bandwidth bottleneck (DDR access time when a network layer processes a frame of data is much greater than the computation time of the Neural Processing Unit (NPU)) can be identified. The candidate optimization layers are network layers in the model to be optimized with high data reuse rate and high read / write frequency.
[0032] The following will combine Figure 2 For an exemplary description of the candidate optimization layer selection method, please refer to [link to documentation]. Figure 2 , Figure 2This is a flowchart of a candidate optimization layer screening method according to an embodiment of this application, such as... Figure 2 As shown, step S101 above may include: Step S201: Perform performance analysis on the model to be optimized to obtain the number of DDR access clock cycles and the number of NPU calculation clock cycles required for each network layer in the model to process each frame of data at a preset frequency.
[0033] Step S202: If the ratio of the number of DDR access clock cycles to the number of NPU computation clock cycles in the network layer is greater than a first preset threshold, the network layer is marked as a candidate optimization layer.
[0034] In this embodiment, a performance analysis tool, such as Profiler, can be used to collect the number of DDR access clock cycles and NPU computation clock cycles required by each network layer in the model to be optimized to process each frame of data at a preset frequency.
[0035] In addition, performance analysis tools can be used to obtain the DDR bandwidth usage and time required for each network layer to process each frame of data, for subsequent analysis and processing.
[0036] DDR Cycles refers to the number of clock cycles consumed when the data generated by the network layer is transferred between the DDR and NPU in the chip while the network layer processes each frame of data; NPU Cycles refers to the number of clock cycles consumed when the operators of the network layer execute computational logic in the NPU while the network layer processes each frame of data.
[0037] After setting DDR Cycles and NPU Cycles to the same frequency (preset frequency), the size relationship between DDR Cycles and NPU Cycles can be directly compared. For example, if the preset frequency is set to the operating frequency of the NPU (e.g., 1000MHz, which means the NPU can run 1 billion times per second), and at this preset frequency, DDR Cycles is 100M and NPU Cycles is 5M, then the ratio of DDR Cycles to NPU Cycles of the network layer can be calculated to be 20. If this ratio of 20 is greater than a first preset threshold (e.g., the first preset threshold can be set to 10), then it is determined that the network layer has a DDR bandwidth bottleneck (i.e., the DDR access time when the network layer processes a frame of data is much greater than the computation time of the neural network processing unit), and the network layer can be marked as a candidate optimization layer.
[0038] It is understood that the aforementioned preset frequency and first preset threshold can be set according to user expectations and the actual application scenario of the method. This application does not impose any restrictions on the specific values of the preset frequency and the first preset threshold.
[0039] Using the above methods, we can accurately locate candidate optimization layers in the model to be optimized that have DDR bandwidth bottlenecks, and further screen and optimize these candidate optimization layers in subsequent processes.
[0040] Step S102: Calculate the SRAM usage and double-rate synchronous dynamic random access memory (DDR) bandwidth savings after the candidate optimization layer is pre-configured as a Static Random-Access Memory (SRAM) priority mode. SRAM priority mode means that the intermediate data generated by the candidate optimization layer will be stored in the chip's SRAM first.
[0041] In this embodiment, the candidate optimization layer with DDR bandwidth bottleneck can be configured in SRAM-priority mode. That is, the intermediate data generated by the candidate optimization layer will be stored in the chip's SRAM first, rather than in the chip's DDR. SRAM is a high-speed cache memory. Storing the intermediate data generated by the candidate optimization layer in SRAM first can reduce the number of interactions between intermediate data and DDR and NPU, thereby reducing the DDR bandwidth usage of the candidate optimization layer.
[0042] Furthermore, due to the physical limitations of the chip, the storage capacity of the SRAM on the chip is subject to certain constraints. For example, in the Rockchip RK3576 chip, the SRAM storage capacity is 512KB (kilobyte). Therefore, in order to avoid data overflow caused by insufficient SRAM storage capacity, further screening of candidate optimization layers is required.
[0043] Step S103: Select multiple layers to be optimized from multiple candidate optimization layers in descending order of DDR bandwidth savings, so that the sum of the SRAM usage of the multiple layers to be optimized is less than or equal to the total storage of SRAM.
[0044] In this embodiment, the DDR bandwidth saving amount represents the difference between the DDR bandwidth required for preprocessing each frame of data when the candidate optimization layer is configured in SRAM-preferred mode and the DDR bandwidth required for postprocessing each frame of data when the candidate optimization layer is configured in SRAM-preferred mode.
[0045] Multiple candidate optimization layers can be sorted in descending order of DDR bandwidth savings. The candidate optimization layers with the highest current sequence number are selected from the sequence. The total SRAM usage of the currently selected candidate optimization layers is calculated to see if it is greater than the total SRAM storage. If so, the candidate optimization layer with the lowest sequence number is removed from the selected candidate optimization layers, and the calculation steps are continued until the total SRAM usage of the selected candidate optimization layers is just less than or equal to the total SRAM storage.
[0046] By using the above methods, data overflow caused by insufficient SRAM storage capacity can be avoided while maximizing the optimization of DDR bandwidth usage of the model to be optimized.
[0047] Step S104: Configure the layer to be optimized to SRAM priority mode.
[0048] In this embodiment, after selecting the layer to be optimized from the candidate optimization layers, the layer to be optimized can be configured as an SRAM priority mode so that the intermediate data generated by the layer to be optimized is preferentially stored in the chip's SRAM, and only when the storage capacity of the SRAM is insufficient, the intermediate data generated by the layer to be optimized is stored in the chip's DDR.
[0049] By using the above methods, performance analysis tools are used to collect DDR Cycles and NPU Cycles of each network layer in the model to be optimized. Based on the DDR Cycles and NPU Cycles, candidate optimization layers with DDR bandwidth bottlenecks are screened. This allows for the identification of network layers that are compatible with the SRAM optimization mentioned above, avoiding blind optimization. At the same time, considering the small SRAM storage capacity of the chip, the optimization priority of candidate optimization layers is determined, and candidate optimization layers with higher DDR bandwidth savings are identified as the layers to be optimized. In addition, a constraint is added to the selection of the layers to be optimized: the total SRAM usage of the layers to be optimized is less than or equal to the total SRAM storage capacity. This can maximize the optimization of the DDR bandwidth usage of the model to be optimized while avoiding data overflow caused by insufficient SRAM storage capacity, thereby improving the task processing efficiency of the model.
[0050] In one embodiment, if the business scenario of the model to be optimized is sensitive to the model's time consumption, a SRAM frame storage strategy can be adopted: split a single frame of data into 2 to 3 blocks and store them into SRAM in the order of calculation. This reduces the increase in inference time while maintaining the optimization effect of DDR bandwidth usage.
[0051] In one embodiment, prior to step S104 above, the method further includes: obtaining the processing frame rate of the model to be optimized; and setting the refresh period of the SRAM based on the processing frame rate so that the refresh period is equal to the time required for the model to be optimized to process one frame of data.
[0052] In this embodiment, the SRAM usage of the selected candidate optimization layer represents the storage space required to process each frame of data after the candidate optimization layer is configured in SRAM priority mode.
[0053] In this embodiment, the refresh cycle of the chip's SRAM can be set based on the processing frame rate of the model to be optimized, thus avoiding data overflow due to insufficient SRAM storage capacity. For example, if the processing frame rate of the model to be optimized is 25 frames per second, the SRAM refresh cycle can be set to 40 milliseconds.
[0054] In one embodiment, after step S104 above, the method further includes: calculating the sum of DDR bandwidth savings for all layers to be optimized; and if the sum of DDR bandwidth savings is greater than or equal to a second preset threshold, determining that the model to be optimized has completed SRAM optimization, and obtaining a first model.
[0055] In this embodiment, after SRAM optimization of the model to be optimized, the optimization effect can be clearly defined, and the completion of SRAM optimization can be determined based on the optimization effect.
[0056] As an example, the total DDR bandwidth savings of all layers to be optimized can be calculated. If the total DDR bandwidth savings are greater than or equal to the second preset threshold (for example, it can be set to 10MB (Megabit) / frame), it can be determined that the optimization model has been initially completed. If the total DDR bandwidth savings are less than the second preset threshold, it can be determined that the current optimization effect is not good. The sorting in step S103 can be adjusted, and steps S103 to S104 can be re-executed until the total DDR bandwidth savings of all layers to be optimized are greater than or equal to the second preset threshold.
[0057] It is understood that the aforementioned second preset threshold can be set according to user expectations, the actual application scenario of the method, and the SRAM storage capacity of the chip. This application does not impose any restrictions on the specific value of the second preset threshold.
[0058] By monitoring the optimization effect of the DDR bandwidth usage of the model to be optimized and making timely adjustments when the optimization effect is not good, until the optimization effect meets the predetermined standard, the DDR bandwidth usage of the optimized first model can be reduced as much as possible. This avoids problems such as task processing timeouts or incompatibility with other tasks caused by the DDR bandwidth usage of the model exceeding the actual available threshold of the chip, thereby improving the task processing efficiency and running stability of the first model.
[0059] The following will combine Figure 3For an explanation of another embodiment of the model optimization method provided in this application, please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart of a model optimization method according to another embodiment of this application, such as... Figure 3 As shown, after determining that the model to be optimized has completed SRAM optimization and obtained the first model, the method further includes: Step S301: Obtain the chip's operator fusion rules from the chip's configuration file.
[0060] Step S302: Based on the operator fusion rule, at least two network layers in the first model that match the operator fusion rule are fused into a fused network layer to obtain the second model.
[0061] In this embodiment, the model structure of the first model can be optimized based on the operator fusion characteristics of the chip. For example, the RK3576, RK3568, RK3588, RK3566, and RK3562 chips of the Rockchip platform all have operator fusion characteristics. The model structure of the first model can be optimized based on these operator fusion characteristics. Since the fused operator combination can be regarded as a single computing unit by the chip NPU, its inter-layer data does not need to be interacted through DDR (that is, the inter-layer data generated by the network layer in the fused network layer for processing each frame of data will be directly input to the next network layer in the fused network layer). On the basis of the above SRAM optimization, the DDR bandwidth usage can be further reduced by 20% to 30%.
[0062] In this embodiment, matching the operator fusion rule means: at least two network layers are arranged consecutively, and at least two network layers conform to the operator fusion rule; or, at least two network layers are not arranged consecutively, and at least two network layers conform to the operator fusion rule; and, moving a network layer between at least two network layers to before / after at least two network layers does not affect the calculation results of at least two network layers.
[0063] The operator fusion rules mentioned above are related to the chip model. For example, for the RK3576 chip, the operator fusion rules may include: merging Conv (convolutional) layers and ReLU (linear rectified) layers into a ConvReLU layer; merging Conv layers and PreLU (parameterized ReLU) layers into a ConvPReLU layer; merging Conv layers and Clip layers into a ConvClip layer; merging Conv layers and Sigmoid layers into a ConvSigmoid layer; merging Conv layers and Add layers into a ConvAdd layer; and merging Conv layers, Activation layers, and Add layers into a ConvActivationAdd layer.
[0064] Taking the operator fusion rule mentioned above as an example, if at least two network layers are arranged consecutively, and these at least two network layers are exactly a Conv layer and a ReLU layer, then it is determined that these at least two network layers meet the operator fusion rule, and these at least two network layers can be fused into a fused network layer, ConvReLU layer. Alternatively, if at least two network layers are not arranged consecutively, for example, if there is a Transpose layer between the Conv layer and the Clip layer, since moving the Transpose layer to the Clip layer does not affect the calculation results of the Conv layer and the Clip layer, the order of the Transpose layer and the Clip layer can be interchanged, and the Conv layer and the Clip layer can be fused into a fused network layer, ConvClip layer.
[0065] Since the fused network layer can be regarded as a single computing unit by the chip's NPU hardware, the inter-layer data of the fused network layer does not need to be interacted through DDR. That is, the inter-layer data generated by the network layer in the fused network layer in processing each frame of data will be directly input to the next network layer in the fused network layer. Therefore, the DDR bandwidth usage of the second model can be further reduced.
[0066] Step S303: Obtain the chip pipeline fusion rules from the configuration file.
[0067] Step S304: Based on the pipeline fusion rule, at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers in the second model that match the pipeline fusion rule are fused into a fused pipeline to obtain the third model.
[0068] Similar to the operator fusion optimization mentioned above, some chips also support pipeline fusion. For example, Rockchip's RK3576, RK3568, RK3588, RK3566, and RK3562 chips all have pipeline fusion features. Based on this pipeline fusion feature, the model structure of the second model can be further optimized. Since the inter-layer data generated by the network layer or fused network layer in the fused pipeline for processing each frame of data will be directly input to the next network layer or fused network layer in the fused pipeline, on the basis of the above SRAM optimization and operator fusion optimization, the DDR bandwidth usage can be further reduced by 15% to 20%.
[0069] In this embodiment, matching the pipeline fusion rule means: at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers are arranged consecutively, and at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers conform to the pipeline fusion rule and have the same input / output dimensions.
[0070] The pipeline fusion rules described above are related to the chip model. For example, for the RK3576 chip, the pipeline fusion rules may include: at least two network layers with similar computation patterns, or at least one network layer and at least one fused network layer, or at least two fused network layers arranged consecutively, and at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers having the same input / output dimensions, can be fused into a fused pipeline.
[0071] Taking the pipeline fusion rule mentioned above as an example, if three network layers are arranged consecutively and have the same input / output dimensions, for example, if the three network layers are a Conv layer, a BatchNorm (batch normalization) layer, and a ReLU layer arranged consecutively and have the same input / output dimensions, then these three network layers can be fused into a fusion pipeline, ConvPipeline. Alternatively, if two fusion network layers, ConvReLU layers, are arranged consecutively and have the same input / output dimensions, then these two fusion network layers can be fused into a fusion pipeline, ConvPipeline.
[0072] Interlayer data in the fusion pipeline does not need to interact through DDR. That is, the interlayer data generated by the network layer or fusion network layer in the fusion pipeline for processing each frame of data will be directly input to the next network layer or fusion network layer in the fusion pipeline. Therefore, the DDR bandwidth usage of the third model can be further reduced.
[0073] In addition, the number of shortcut branches in the second model can be reduced, or dimension adaptation layers can be added to the branches (by using 1×1 convolutional blocks to adjust the input / output dimensions of the branches to be consistent with the trunk, so that the branch layers and the trunk layers have the same input / output dimensions), thereby increasing the number of pipelined or fused network layers and further reducing the DDR bandwidth consumption caused by inter-layer data copying.
[0074] Step S305: Obtain the DDR access granularity of the chip from the configuration file.
[0075] Step S306: Set the number of input / output channels for each network layer, each fused network layer, or each fused pipeline in the third model to an integer multiple of the DDR access granularity to obtain the optimized model.
[0076] In this embodiment, the DDR access granularity of the chip can also be determined by combining the chip's DDR read / write friendly rules (for example, in the RK3576 chip, the number of input / output channels of the Conv operator is an integer multiple of 32 under Int8 precision).
[0077] While maintaining the computational load of the model, the number of input / output channels for each network layer, each fusion network layer, or each fusion pipeline in the third model is adjusted to an integer multiple of 32, thereby improving DDR read / write efficiency.
[0078] Furthermore, taking the RK3576 chip as an example, the backbone of the third model can be replaced with a lightweight structure of NPU hardware adapted to the RK3576 chip (such as RepVGG, MobileNetV5, EfficientNet-Lite, etc.). This structure has the characteristics of friendly inter-layer fusion, easy control of computational load, and easy adaptation of the number of channels to integer multiples of 32. While ensuring the accuracy of the algorithm, it can improve the utilization of hardware resources and maximize the synergistic effect of various optimization methods.
[0079] Through the above-mentioned SRAM optimization, operator fusion optimization, pipeline fusion optimization, and channel number adaptation optimization, the DDR bandwidth usage of the model can be optimized in a coordinated manner, minimizing the DDR bandwidth usage of the model. This avoids problems such as task processing timeouts or incompatibility with other tasks caused by the DDR bandwidth usage of the model exceeding the actual available threshold of the chip, thereby improving the task processing efficiency and running stability of the optimized model.
[0080] Through the above steps S101 to S104, performance analysis can be performed on the model to be optimized deployed on the chip to determine candidate optimization layers in the model; the SRAM usage and DDR bandwidth savings after the candidate optimization layers are pre-configured as SRAM-priority mode can be calculated, where SRAM-priority mode means that intermediate data generated by the candidate optimization layers are preferentially stored in the chip's SRAM; multiple optimization layers can be selected from multiple candidate optimization layers in descending order of DDR bandwidth savings, so that the sum of the SRAM usage of the multiple optimization layers is less than or equal to the total SRAM storage capacity; and the optimization layers can be configured as SRAM-priority layers. In this way, multiple optimization layers with high DDR bandwidth savings can be configured as SRAM-priority layers, and the intermediate data generated by these optimization layers can be preferentially stored in the chip's SRAM, avoiding frequent read and write operations on the chip's DDR, thereby reducing the DDR bandwidth occupied by the model while avoiding SRAM data overflow. This application solves the problem in related technologies where high-computational-load models deployed on chips occupy high DDR bandwidth, achieving the technical effect of reducing the DDR bandwidth occupied by models deployed on chips.
[0081] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0082] Corresponding to the model optimization method described in the above embodiments, Figure 4 A schematic diagram of a model optimization apparatus according to an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.
[0083] Please see Figure 4 The model optimization device 4 includes: an analysis module 40, used to perform performance analysis on the model to be optimized deployed on the chip and determine the candidate optimization layers in the model to be optimized; a calculation module 41, used to calculate the SRAM usage and double-rate synchronous dynamic random access memory (DDR) bandwidth savings after the candidate optimization layers are pre-configured as SRAM priority mode, where SRAM priority mode means that intermediate data generated by the candidate optimization layers are preferentially stored in the chip's SRAM; a selection module 42, used to select multiple optimization layers from multiple candidate optimization layers in descending order of DDR bandwidth savings, so that the sum of the SRAM usage of multiple optimization layers is less than or equal to the total storage of SRAM; and a configuration module 43, used to configure the optimization layers as SRAM priority mode.
[0084] In one embodiment, the analysis module 40 is further configured to perform performance analysis on the model to be optimized, and obtain the number of DDR access clock cycles and the number of NPU computation clock cycles required by each network layer in the model to process each frame of data at a preset frequency; if the ratio of the number of DDR access clock cycles to the number of NPU computation clock cycles of the network layer is greater than a first preset threshold, the network layer is marked as a candidate optimization layer.
[0085] In one embodiment, SRAM occupancy represents the storage space required to process each frame of data after the candidate optimization layer is configured in SRAM priority mode; the configuration module 43 is also used to obtain the processing frame rate of the model to be optimized; based on the processing frame rate, the SRAM refresh period is set so that the refresh period is equal to the time required for the model to be optimized to process one frame of data.
[0086] In one embodiment, the model optimization device 4 further includes a determination module for calculating the sum of DDR bandwidth savings for all layers to be optimized; if the sum of DDR bandwidth savings is greater than or equal to a second preset threshold, it is determined that the model to be optimized has completed SRAM optimization, and a first model is obtained.
[0087] In one embodiment, the model optimization device 4 further includes an operator fusion optimization module, used to obtain the operator fusion rules of the chip from the chip's configuration file; based on the operator fusion rules, to fuse at least two network layers in the first model that match the operator fusion rules into a fused network layer to obtain a second model, wherein the inter-layer data generated by the network layers in the fused network layer for processing each frame of data will be directly input into the next network layer in the fused network layer; matching the operator fusion rules means that: at least two network layers are arranged consecutively and at least two network layers conform to the operator fusion rules; or, at least two network layers are not arranged consecutively and at least two network layers conform to the operator fusion rules, and moving a network layer between at least two network layers to before / after at least two network layers does not affect the calculation results of at least two network layers.
[0088] In one embodiment, the model optimization device 4 further includes a pipeline fusion optimization module, used to obtain the pipeline fusion rules of the chip from the configuration file; based on the pipeline fusion rules, at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers in the second model that match the pipeline fusion rules are fused into a fusion pipeline to obtain the third model. The inter-layer data generated by the network layers or fused network layers in the fusion pipeline for processing each frame of data will be directly input to the next network layer or fused network layer in the fusion pipeline; matching the pipeline fusion rules means that at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers are arranged consecutively, and at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers conform to the pipeline fusion rules and have the same input / output dimensions.
[0089] In one embodiment, the model optimization device 4 further includes a channel number adaptation optimization module, which is used to obtain the DDR access granularity of the chip from the configuration file; and set the number of input / output channels of each network layer, each fused network layer, or each fused pipeline in the third model to an integer multiple of the DDR access granularity to obtain the optimized model.
[0090] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0092] Figure 5 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application. Figure 5 As shown, the terminal device 5 includes: at least one processor 50 ( Figure 5 (Only one is shown in the diagram) a processor, a memory 51, and a computer program 52 stored in the memory 51 and capable of running on at least one processor 50, wherein the processor 50 executes the computer program 52 to implement the steps in any of the above-described model optimization method embodiments.
[0093] Terminal device 5 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 5 may include, but is not limited to, processor 50 and memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of terminal device 5 and does not constitute a limitation on terminal device 5. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.
[0094] The processor 50 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0095] In some embodiments, memory 51 may be an internal storage unit of terminal device 5, such as a hard disk or memory of terminal device 5. In other embodiments, memory 51 may be an external storage device of terminal device 5, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on terminal device 5. In other embodiments, memory 51 may include both internal storage units and external storage devices of terminal device 5. Memory 51 is used to store operating system, application programs, bootloader, data, and other programs, such as the program code of computer program 52. Memory 51 may also be used to temporarily store data that has been output or will be output.
[0096] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described model optimization method embodiments.
[0097] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described model optimization method embodiments.
[0098] This application implements all or part of the processes in the methods of the above embodiments, which can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a model optimization device or terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.
[0099] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0100] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0101] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0103] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A model optimization method, characterized by, The method comprises: performing performance analysis on a to-be-optimized model deployed in a chip to determine candidate optimization layers in the to-be-optimized model; calculating SRAM occupancy and DDR bandwidth saving of the candidate optimization layers after the candidate optimization layers are pre-configured in a SRAM priority mode, the SRAM priority mode indicating that intermediate data generated by the candidate optimization layers are preferentially stored in SRAM of the chip; selecting a plurality of to-be-optimized layers from the plurality of candidate optimization layers in descending order of the DDR bandwidth saving, so that a sum of the SRAM occupancy of the plurality of to-be-optimized layers is less than or equal to a total storage amount of the SRAM; configuring the to-be-optimized layers in the SRAM priority mode.
2. The method of claim 1, wherein, The performance analysis on the to-be-optimized model deployed in the chip to determine the candidate optimization layers in the to-be-optimized model comprises: performing performance analysis on the to-be-optimized model to obtain a number of DDR access clock cycles and a number of NPU calculation clock cycles required by each network layer in the to-be-optimized model to process each frame of data at a preset frequency; in a case where a ratio of the number of DDR access clock cycles to the number of NPU calculation clock cycles of the network layer is greater than a first preset threshold, marking the network layer as a candidate optimization layer.
3. The method of claim 1, wherein, The SRAM occupancy indicates a storage space required by the candidate optimization layers to process each frame of data after the candidate optimization layers are configured in the SRAM priority mode; before the to-be-optimized layers are configured in the SRAM priority mode, the method further comprises: obtaining a processing frame rate of the to-be-optimized model; based on the processing frame rate, setting a refresh period of the SRAM, so that the refresh period is equal to a required duration of processing one frame of data by the to-be-optimized model.
4. The method according to any one of claims 1 to 3, characterized in that, after the to-be-optimized layers are configured in the SRAM priority mode, the method further comprises: calculating a sum of the DDR bandwidth saving of all the to-be-optimized layers; in a case where the sum of the DDR bandwidth saving is greater than or equal to a second preset threshold, determining that the to-be-optimized model has completed SRAM optimization to obtain a first model.
5. The method of claim 4, wherein, after the to-be-optimized model is determined to have completed SRAM optimization to obtain the first model, the method further comprises: obtaining an operator fusion rule of the chip from a configuration file of the chip; based on the operator fusion rule, fusing at least two network layers in the first model that match the operator fusion rule into a fused network layer to obtain a second model, layer data generated by a network layer in the fused network layer processing each frame of data will be directly input to a next network layer in the fused network layer; Matching the operator fusion rule means that: the at least two network layers are arranged consecutively and the at least two network layers conform to the operator fusion rule; or, the at least two network layers are not arranged consecutively and the at least two network layers conform to the operator fusion rule, and moving a network layer between the at least two network layers to before / after the at least two network layers does not affect the calculation results of the at least two network layers.
6. The method of claim 5, wherein, After fusing at least two network layers in the first model that match the operator fusion rule into a fused network layer to obtain the second model, the method further includes: Obtain the pipeline fusion rules for the chip from the configuration file; Based on the pipeline fusion rule, at least two network layers, or at least one network layer and at least one fusion network layer, or at least two fusion network layers in the second model that match the pipeline fusion rule are fused into a fusion pipeline to obtain the third model. The inter-layer data generated by the network layer or fusion network layer in the fusion pipeline for processing each frame of data will be directly input to the next network layer or fusion network layer in the fusion pipeline. Matching the pipeline fusion rule means that the at least two network layers, or the at least one network layer and the at least one fused network layer, or the at least two fused network layers are arranged consecutively, and the at least two network layers, or the at least one network layer and the at least one fused network layer, or the at least two fused network layers conform to the pipeline fusion rule and have the same input / output dimensions.
7. The method of claim 6, wherein, After fusing at least two network layers, or at least one network layer and at least one fused network layer, or at least two fused network layers in the second model into a fused pipeline based on the pipeline fusion rule to obtain the third model, the method further includes: Obtain the DDR access granularity of the chip from the configuration file; The optimized model is obtained by setting the number of input / output channels of each network layer, each fused network layer, or each fused pipeline in the third model to an integer multiple of the DDR access granularity.
8. A model optimization apparatus characterized by comprising: The device includes: The analysis module is used to perform performance analysis on the model to be optimized deployed on the chip and determine the candidate optimization layers in the model to be optimized. The calculation module is used to calculate the SRAM usage and the bandwidth saving of double-rate synchronous dynamic random access memory (DDR) after the candidate optimization layer is pre-configured as a Static Random Access Memory (SRAM) priority mode. The SRAM priority mode means that the intermediate data generated by the candidate optimization layer is preferentially stored in the SRAM of the chip. The selection module is used to select multiple layers to be optimized from multiple candidate optimization layers in descending order of DDR bandwidth savings, so that the sum of the SRAM usage of the multiple layers to be optimized is less than or equal to the total storage of the SRAM. A configuration module is used to configure the layer to be optimized as the SRAM priority mode.
9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the model optimization method as described in any one of claims 1 to 7.
10. A computer program product, characterised in that, Includes a computer program, which, when run, causes the model optimization method as described in any one of claims 1 to 7 to be performed.