Global mask iteration pruning method and device, equipment and storage medium

Abstract

The invention provides a global mask iteration pruning method and device, equipment and a storage medium, and belongs to the technical field of deep learning. The method comprises the following steps: constructing a target loss function, wherein the target loss function comprises task loss, pruning loss and hardware resource loss; in any iteration training process, determining a pruning weight matrix of the target model in the current iteration process according to the global mask matrix, determining target loss of the target model in the current iteration process, and updating the global mask matrix in the next iteration process according to the target loss; and when the target model converges to a target state, obtaining a trained target model. The invention aims to dynamically adjust the calculation complexity and the hardware resource use efficiency of the model.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of deep learning, and in particular, to a global mask iterative pruning method, apparatus, device, and storage medium. Background Art

[0002] With the continuous development of deep learning technology, neural network models have achieved remarkable results in fields such as image recognition and natural language processing; however, the computing and storage resource requirements of deep learning models are also growing rapidly, making it challenging to deploy deep learning models in scenarios such as edge computing, IoT devices, and mobile terminals. Especially in these resource-constrained environments, the computing cost and memory usage of the model become the main bottlenecks.

[0003] Currently, model compression methods such as pruning are commonly used to try to reduce the model size while ensuring model performance. However, current pruning schemes often fail to fully consider the resource constraints of the target hardware, such as the CPU utilization, memory usage, and power consumption of the device running the model, resulting in computing bottlenecks and resource waste when the model is actually deployed. Summary of the invention

[0004] The embodiments of the present application provide a global mask iterative pruning method, apparatus, device and storage medium, which aim to dynamically adjust the computational complexity of the model and the efficiency of hardware resource utilization.

[0005] In a first aspect, an embodiment of the present application provides a global mask iterative pruning method, the method comprising: Constructing a target loss function, wherein the target loss function includes task loss, pruning loss and hardware resource loss, wherein the pruning loss is calculated according to the pruning weight matrix in each iteration process; the hardware resource loss is determined according to the real-time hardware resource information of the deployment device of the target model in the iteration process; Perform iterative training on the target model based on the target loss function. In any iterative training process, determine the pruning weight matrix of the target model in this iterative process according to the global mask matrix, determine the target loss of the target model in this iterative process, and update the global mask matrix of the next iterative process according to the target loss. Any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation; When the target model converges to the target state, the trained target model is obtained.

[0006] Optionally, the objective loss function is constructed as:

[0007]

[0008] Among them, L is the target loss function, is the task loss of the target model; is the pruning loss; Loss of hardware resources; is the loss weight of pruning loss; is the loss weight of hardware resource loss; is the element in the pruning weight matrix in any iteration process.

[0009] Optionally, the method further comprises: During the iteration process of the target model, the hardware resource information of the deployment device of the target model during the iterative training process is obtained, and the hardware resource information includes the CPU utilization rate, memory utilization rate, power consumption of the deployment device and temperature of the deployment device.

[0010] Optionally, the formula for hardware resource loss is:

[0011] in, CPU usage of the deployment device; Memory usage of the deployment device; The power consumption of the deployed equipment; The temperature of the deployed equipment; These are the weight parameters of CPU usage, memory usage, power consumption, and temperature.

[0012] Optionally, the method further comprises: Initialize the global mask matrix of the target model, the size of the global mask matrix is ​​consistent with the network weight matrix of the target model, and each element in the initialized global mask matrix is ​​1.

[0013] Optionally, in any iterative training process, the pruning weight matrix of the target model in this iterative process is determined according to the global mask matrix, the target loss of the target model in this iterative process is determined, and the global mask matrix of the next iterative process is updated according to the target loss, including: In any iteration process, the pruning weight matrix is ​​obtained by multiplying the global mask matrix and the network weight matrix of this iteration process; Based on the pruning weight matrix, calculate the target loss of the target model in this iteration process; According to the target loss, the global mask matrix and network weight matrix of the next iteration process are gradient updated respectively.

[0014] Optionally, after gradient updating the global mask matrix and the network weight matrix of the next iteration process respectively according to the target loss, the method further includes: Determine the size of any mask element in the global mask matrix after gradient update and the pruning threshold; When any mask element is smaller than the pruning threshold, the mask element is set to 0.

[0015] In a second aspect, an embodiment of the present application provides a global mask iterative pruning device, the device comprising: A construction module is used to construct a target loss function, wherein the target loss function includes task loss, pruning loss and hardware resource loss, wherein the pruning loss is calculated according to the pruning weight matrix in each iteration process; and the hardware resource loss is determined according to the real-time hardware resource information of the deployment device of the target model in the iteration process; An iterative training module, used to perform iterative training on the target model based on the target loss function. In any iterative training process, the pruning weight matrix of the target model in this iterative process is determined according to the global mask matrix, the target loss of the target model in this iterative process is determined, and the global mask matrix of the next iterative process is updated according to the target loss. Any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation; The model generation module is used to obtain the trained target model when the target model converges to the target state.

[0016] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the global mask iterative pruning method described in the first aspect of the embodiment is performed.

[0017] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the global mask iterative pruning method described in the first aspect of the embodiment is performed.

[0018] Beneficial effects: Construct a target loss function, which includes task loss, pruning loss and hardware resource loss. The pruning loss is calculated based on the pruning weight matrix in each iteration process; the hardware resource loss is determined based on the real-time hardware resource information of the deployment device of the target model during the iteration process.

[0019] Iterative training of the target model is performed based on the target loss function. In any iterative training process, the pruning weight matrix of the target model in this iterative process is determined according to the global mask matrix, the target loss of the target model in this iterative process is determined, and the global mask matrix of the next iterative process is updated according to the target loss. Any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation; when the target model converges to the target state, the trained target model is obtained.

[0020] This method sets a global mask matrix during the training process of the target model to determine whether any weight element in the network weight matrix participates in the operation, thereby reducing the computational complexity and hardware resources occupied by the target model; and adds pruning loss and hardware resource loss when constructing the target loss function of the model. The sparse ratio of the global mask matrix can be dynamically adjusted during the model training process, thereby dynamically adjusting the pruning intensity. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 is a flowchart of the steps of the global mask iterative pruning method proposed in one embodiment of the present application; Figure 2 is a flowchart of the steps of the global mask iterative pruning method proposed in one embodiment of the present application; Figure 3 is a schematic diagram of a global mask iterative pruning method provided by an embodiment of the present application; Figure 4 It is a functional module diagram of a global mask iterative pruning device proposed in an embodiment of the present application; Figure 5 is a schematic diagram of a computer device proposed in one embodiment of the present application; Figure 6 It is a schematic diagram of a non-volatile readable storage medium provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] RAGMIP: Resource-Aware Global Mask Iterative Pruning, resource-aware global mask iterative pruning; Global mask: refers to the mask matrix defined on all layers of the neural network, which is used to mark and control the retention and pruning of weights in the neural network.

[0025] With the continuous development of deep learning technology, neural network models have achieved remarkable results in fields such as image recognition and natural language processing; however, the computing and storage resource requirements of deep learning models are also growing rapidly, making it challenging to deploy deep learning models in scenarios such as edge computing, IoT devices, and mobile terminals. Especially in these resource-constrained environments, the computing cost and memory usage of the model become the main bottlenecks.

[0026] Currently, model compression methods such as pruning and quantization are commonly used to try to reduce the model size while ensuring model performance. However, current pruning schemes often fail to fully consider the resource constraints of the target hardware, such as the CPU utilization, memory usage, and power consumption of the device running the model, resulting in computing bottlenecks and resource waste when the model is actually deployed.

[0027] In addition, traditional model pruning techniques are usually based on static analysis and cannot adapt to real-time changes in input data. They are also difficult to adjust according to dynamic changes in hardware resources, resulting in poor results in real-time scenarios.

[0028] Therefore, an embodiment of the present application provides a global mask iterative pruning method that optimizes resource utilization while ensuring model performance, aiming to dynamically adjust the computational complexity of the model and the efficiency of hardware resource utilization, so that the model is suitable for resource-constrained environments.

[0029] Reference Figure 1 , shows a flowchart of the steps of a global mask iterative pruning method in an embodiment of the present application, and the method may specifically include the following steps: S101: Construct a target loss function, which includes task loss, pruning loss and hardware resource loss. The pruning loss is calculated based on the pruning weight matrix in each iteration process; the hardware resource loss is determined based on the real-time hardware resource information of the deployment device of the target model during the iteration process.

[0030] In this embodiment, in order to reduce the problem of occupying more hardware resources due to large amount of computation when the target model is deployed on a device with limited hardware resources, pruning is performed in combination with the hardware resources of the deployment device for training the target model during the target model training process.

[0031] First, a new objective function of the target model is constructed, and the constructed objective loss function is:

[0032]

[0033] Among them, L is the target loss function, is the task loss of the target model; is the pruning loss; Loss of hardware resources; is the loss weight of pruning loss; is the loss weight of hardware resource loss; is the element in the pruning weight matrix in any iteration process.

[0034] In the actual implementation process, the loss weight of the pruning loss and the loss weight of hardware resource loss Custom configuration can be performed according to actual application requirements.

[0035] The hardware resource information includes the usage of various hardware resources of the deployed device. For example, the hardware resource information may include at least one of the CPU usage rate, memory usage rate, power consumption and temperature of the deployed device. In the actual implementation process, the content of the hardware resource information can also be expanded according to the needs of the actual application.

[0036] Taking the hardware resource information including CPU usage, memory usage, power consumption of deployed devices, and temperature of deployed devices as an example, the formula for hardware resource loss in the target loss function of the target model is:

[0037] in, CPU usage of the deployment device; Memory usage of the deployment device; The power consumption of the deployed equipment; The temperature of the deployed equipment; These are the weight parameters of CPU usage, memory usage, power consumption, and temperature.

[0038] Initialize a global mask matrix, which is used to implement the pruning of the computational complexity of the target model. For example, each element in the initialized global mask matrix is ​​1, indicating that during the first iterative training of the target model, any weight element in the network weight matrix participates in the forward propagation of the target model.

[0039] S102: Perform iterative training on the target model based on the target loss function. In any iterative training process, the pruning weight matrix of the target model in this iterative process is determined according to the global mask matrix, the target loss of the target model in this iterative process is determined, and the global mask matrix of the next iterative process is updated according to the target loss. Any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation.

[0040] During the forward propagation process of any iteration, the pruning weight matrix is ​​obtained by multiplying the global mask matrix and the network weight matrix of this iteration process; based on the pruning weight matrix and the hardware resources of the deployed device, the target loss of the target model in this iteration process is calculated.

[0041] According to the target loss, the model parameters of the target model are updated by using the loss gradient descent method, including updating the global mask matrix and the network weight matrix to obtain the global mask matrix and the network weight matrix in the next iteration.

[0042] S103: When the target model converges to the target state, a trained target model is obtained.

[0043] In the actual implementation process, the target model converges to the target state, which may include that the number of iterations of the target model reaches the target number, or the target loss function value of the target model is less than the target loss threshold. Then, the training of the target model can be stopped to obtain the trained target model. At this time, the target model obtained is a sparse model compared to the model that does not adopt the pruning method.

[0044] Reference Figure 2 , shows a flowchart of a global mask iterative pruning method provided by an embodiment of the present application, and the method may specifically include the following steps: S1: Construct a target loss function, which includes task loss, pruning loss and hardware resource loss.

[0045] The pruning loss is calculated based on the pruning weight matrix in each iteration process; the hardware resource loss is determined based on the real-time hardware resource information of the deployment device of the target model during the iteration process.

[0046] In this embodiment, in order to reduce the problem of occupying more hardware resources due to the large amount of calculation when the target model is deployed on a device with limited hardware resources, during the target model training process, pruning is performed in combination with the hardware resources of the deployment device for training the target model. Specifically, a new objective function of the target model is constructed, and the constructed objective loss function is:

[0047]

[0048] Among them, L is the target loss function, is the task loss of the target model; is the pruning loss; Loss of hardware resources; is the loss weight of pruning loss; is the loss weight of hardware resource loss; is the element in the pruning weight matrix in any iteration process.

[0049] In the actual implementation process, the loss weight of the pruning loss and the loss weight of hardware resource loss Custom configuration can be performed according to actual application requirements.

[0050] Pruning loss It can be used to measure the efficiency of the model in removing unnecessary weights during the pruning process, while controlling the sparsity of the model. The goal of pruning loss is to reduce unnecessary weights as much as possible while maintaining task performance, thereby compressing the model. By applying appropriate pruning loss, the model can be prompted to automatically adjust the global mask matrix and remove some weights that contribute less to the task. Pruning loss is constructed using L1 regularization to force the model weights to tend to be sparse and encourage weights to be close to 0.

[0051] By adding pruning loss and hardware resource loss to the original task loss of the target model to determine the pruning intensity, a resource-aware pruning process is achieved. Compared with the traditional pruning method based on the importance or redundancy of parameters within the model, this method can use the resource-aware pruning process to consider the actual operating conditions of the target model on the deployed device, thereby optimizing the performance of the model on the target platform.

[0052] Specifically, during the iterative training of the target model, it is necessary to obtain the hardware resource information of the deployed device. As the number of iterations and the amount of computation increase, the resources occupied by the training process of the target model may increase. For example, the hardware resource information of the iteration process can be obtained before each iteration starts. Then, during the continuous iterative training of the target model, the pruning intensity can be adjusted according to the trend of the model occupying hardware resources.

[0053] The hardware resource information includes the usage of various hardware resources of the deployed device. For example, the hardware resource information may include at least one of the CPU usage rate, memory usage rate, power consumption and temperature of the deployed device. In the actual implementation process, the content of the hardware resource information can also be expanded according to the needs of the actual application.

[0054] Taking the hardware resource information including CPU usage, memory usage, power consumption of deployed devices, and temperature of deployed devices as an example, the formula for hardware resource loss in the target loss function of the target model is:

[0055] in, CPU usage of the deployment device; Memory usage of the deployment device; The power consumption of the deployed equipment; The temperature of the deployed equipment; These are the weight parameters of CPU usage, memory usage, power consumption, and temperature.

[0056] S2: Initialize a global mask matrix.

[0057] After constructing the target loss function of the target model, a global mask matrix is ​​initialized. The global mask matrix is ​​used to implement the pruning of the computational complexity of the target model, thereby dynamically reducing the occupied hardware resources or adapting to the hardware resources of the deployed device.

[0058] The size of the global mask matrix is ​​consistent with the network weight matrix of the target model. The network weight matrix of the model has the function of feature selection and feature change in the model. In any iteration process, the global mask matrix and the network weight matrix are multiplied to obtain the pruning weight matrix that actually participates in the forward propagation of the target model in this iteration.

[0059] For example, each element in the initialized global mask matrix is ​​1, which indicates that during the first iterative training of the target model, any weight element in the network weight matrix participates in the operation in the forward propagation of the target model.

[0060] S3: In any iteration process, the pruning weight matrix is ​​obtained by multiplying the global mask matrix and the network weight matrix of this iteration process.

[0061] For example, in the forward propagation process of each iteration of the target model, the global mask matrix and the network weight matrix of this iteration are multiplied to obtain the pruning weight matrix, and the calculation formula is as follows:

[0062] Where W is the global mask matrix of this iteration process; M is the network weight matrix of this iteration process; is the pruning weight matrix of this iteration process; is the matrix multiplication operation.

[0063] During the first iteration of the target model, because the global mask matrix is ​​a matrix with all mask elements being 1, all weight elements in the initial network weight matrix participate in the calculation.

[0064] S4: Based on the pruning weight matrix, calculate the target loss of the target model in this iteration process.

[0065] Based on the obtained pruning weight matrix, during the forward propagation process of the target model, the target model will be operated layer by layer, and finally the target loss will be calculated.

[0066] S5: According to the target loss, the global mask matrix and network weight matrix of the next iteration process are gradient updated respectively.

[0067] Then, according to the target loss, the model parameters of the target model are updated by using the loss gradient descent method, including updating the global mask matrix and the network weight matrix to obtain the global mask matrix and the network weight matrix in the next iteration.

[0068] In a feasible implementation, after determining the global mask matrix after gradient update, the size of any mask element in the global mask matrix after gradient update and the pruning threshold can also be determined; when any mask element is smaller than the pruning threshold, the mask element is set to 0.

[0069] The process of training the target model is to reduce the loss of the target loss function, and then iteratively update the global mask matrix. The number of zero items in the global mask matrix will gradually increase, that is, the sparsity in the global mask matrix will gradually increase. While the increase in zero items in the global mask matrix will reduce the number of weight elements involved in the operation in the network weight matrix, thereby reducing the computational complexity of the target model and the hardware resources occupied.

[0070] However, this embodiment sets a pruning threshold, such as 0.5. When any mask element in the updated global mask matrix is ​​less than the pruning threshold, the mask element is configured to 0, so that the amount of calculation can be gradually reduced to slowly reduce the influence of the weight element, instead of immediately pruning any weight element in each iteration process in a non-0 or 1 manner.

[0071] By dynamically adjusting the sparsity ratio in the global mask matrix, the pruning intensity is gradually increased until the predetermined sparsity or hardware resource constraints are reached.

[0072] S6: When the target model converges to the target state, the trained target model is obtained.

[0073] In the actual implementation process, the target model converges to the target state, which may include that the number of iterations of the target model reaches the target number, or the target loss function value of the target model is less than the target loss threshold. Then, the training of the target model can be stopped to obtain the trained target model. At this time, the target model obtained is a sparse model compared to the model that does not adopt the pruning method.

[0074] In the actual implementation process, after the target model is obtained through training, the target model can be further compressed and quantized. For example, during the model iteration process, the weight elements with mask elements of 0 in the network weight matrix are deleted, which significantly reduces the size and computational complexity of the model. The memory usage is further reduced through sparse matrix storage or efficient model representation methods. The remaining weights are quantized to reduce the floating point precision, memory requirements and computational complexity. In the actual implementation process, the quantization range of different layers can be dynamically adjusted according to the distribution of activation data to ensure that the model does not lose significant accuracy while being compressed.

[0075] Reference Figure 3 , shows a schematic diagram of a global mask iterative pruning method provided in an embodiment of the present application. In a feasible implementation, First, construct the target loss function, which includes task loss, pruning loss and hardware resource loss.

[0076] For example, a new objective function of the target model is constructed, and the constructed objective loss function is:

[0077]

[0078]

[0079] Among them, L is the target loss function, is the task loss of the target model; is the pruning loss; Loss of hardware resources; is the loss weight of pruning loss; is the loss weight of hardware resource loss; is the element in the pruning weight matrix in any iteration process; CPU usage of the deployment device; Memory usage of the deployment device; The power consumption of the deployed equipment; The temperature of the deployed equipment; These are the weight parameters of CPU usage, memory usage, power consumption, and temperature.

[0080] Then, the global mask weights are initialized.

[0081] For example, each element in the initialized global mask matrix is ​​1.

[0082] Next, the network weight matrix of the target model is initialized.

[0083] Then, during the forward propagation process of each iteration, the hardware resource information of the current deployed device can be monitored, and the global mask matrix and the network weight matrix of this iteration process can be multiplied to obtain the pruning weight matrix. Based on the pruning weight matrix and the hardware resource information, the target loss can be calculated.

[0084] Next, the model parameters including the global mask matrix and the network weight matrix are updated according to the target loss, and compression and quantization processes are performed to reduce memory usage.

[0085] Finally, when the target model is iteratively trained to the target state, a trained target model is obtained.

[0086] The method provided in this embodiment has at least the following beneficial effects: 1. During the pruning process, a global mask matrix is ​​added to prune all layers of the model's neural network, reducing the computational load while maintaining model performance. The global mask matrix can make pruning uniform and effective, avoiding the problem of irregular network structure that may be caused by local pruning; 2. When constructing the loss function, hardware resource loss is introduced. By real-time monitoring of changes in hardware resources of deployed devices during model iteration training, including CPU usage, memory usage, power consumption, and temperature, the pruning decision is made based on dynamic adjustment of pruning intensity and computing allocation strategy, thereby effectively reducing the amount of model computation and occupied hardware resources. 3. Optimize the loss function design, comprehensively consider the task loss, pruning loss and hardware resource regularization loss to achieve more efficient resource utilization, and optimize the hardware utilization efficiency in a resource-constrained environment.

[0087] Reference Figure 4 , shows a functional module diagram of a global mask iterative pruning device provided in an embodiment of the present application, the device comprising: A construction module 100 is used to construct a target loss function, wherein the target loss function includes task loss, pruning loss and hardware resource loss, wherein the pruning loss is calculated according to the pruning weight matrix in each iteration process; and the hardware resource loss is determined according to the real-time hardware resource information of the deployment device of the target model in the iteration process; An iterative training module 200 is used to perform iterative training on a target model based on a target loss function. In any iterative training process, a pruning weight matrix of the target model in this iterative process is determined according to a global mask matrix, a target loss of the target model in this iterative process is determined, and the global mask matrix of the next iterative process is updated according to the target loss. Any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation. The model generation module 300 is used to obtain a trained target model when the target model converges to a target state.

[0088] Optionally, the target loss function constructed by the construction module is:

[0089]

[0090] Among them, L is the target loss function, is the task loss of the target model; is the pruning loss; Loss of hardware resources; is the loss weight of pruning loss; is the loss weight of hardware resource loss; is the element in the pruning weight matrix in any iteration process.

[0091] Optionally, the device further comprises: The hardware resource information acquisition module is used to obtain the hardware resource information of the deployment device of the target model during the iterative training process of the target model, and the hardware resource information includes the CPU utilization rate, memory utilization rate, power consumption of the deployment device and temperature of the deployment device.

[0092] Optionally, the formula for hardware resource loss constructed by the building module is:

[0093] in, CPU usage of the deployment device; Memory usage of the deployment device; The power consumption of the deployed equipment; The temperature of the deployed equipment; These are the weight parameters of CPU usage, memory usage, power consumption, and temperature.

[0094] Optionally, the device further comprises: An initialization module is used to initialize the global mask matrix of the target model. The size of the global mask matrix is ​​consistent with the network weight matrix of the target model, and each element in the initialized global mask matrix is ​​1.

[0095] Optionally, the iterative training module includes: A pruning weight matrix calculation unit is used to obtain a pruning weight matrix in any iteration process by multiplying the global mask matrix and the network weight matrix of this iteration process; The target loss calculation unit is used to calculate the target loss of the target model in this iteration process based on the pruning weight matrix; The gradient update unit is used to perform gradient updates on the global mask matrix and the network weight matrix of the next iteration process according to the target loss.

[0096] Optionally, the device further comprises: The judgment unit, Yang Hongyu judges the size of any mask element in the global mask matrix after the gradient update and the pruning threshold; The setting unit is used to set any mask element to 0 when the mask element is smaller than the pruning threshold.

[0097] Reference Figure 5 , shows a schematic diagram of a computer device provided in an embodiment of the present application, comprising: at least one processor 501, and a memory 502, wherein the memory 502 stores a computer program that can be run on the processor 501, wherein the processor 501 executes the global mask iterative pruning method described in the embodiment when executing the computer program.

[0098] When the processor 501 executes the computer program and executes the global mask iterative pruning method described in the embodiment, it specifically includes the following steps: constructing a target loss function, wherein the target loss function includes task loss, pruning loss and hardware resource loss, and the pruning loss is calculated based on the pruning weight matrix in each iteration process; the hardware resource loss is determined based on the real-time hardware resource information of the deployment device of the target model in the iteration process; performing iterative training of the target model based on the target loss function, and in any iterative training process, determining the pruning weight matrix of the target model in this iteration process according to the global mask matrix, determining the target loss of the target model in this iteration process, and updating the global mask matrix of the next iteration process according to the target loss, wherein any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation; when the target model converges to the target state, a trained target model is obtained.

[0099] Reference Figure 6, shows a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium 600 stores a computer program 601, wherein the computer program 601, when executed by a processor, performs the global mask iterative pruning method described in the embodiment.

[0100] When the computer program 601 is executed by the processor to perform the global mask iterative pruning method described in the embodiment, it specifically includes the following steps: constructing a target loss function, wherein the target loss function includes task loss, pruning loss and hardware resource loss, wherein the pruning loss is calculated based on the pruning weight matrix in each iteration process; the hardware resource loss is determined based on the real-time hardware resource information of the deployment device of the target model in the iteration process; performing iterative training of the target model based on the target loss function, and in any iterative training process, determining the pruning weight matrix of the target model in this iteration process according to the global mask matrix, determining the target loss of the target model in this iteration process, and updating the global mask matrix of the next iteration process according to the target loss, wherein any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation; when the target model converges to the target state, a trained target model is obtained.

[0101] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0102] It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, devices or computer program products. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0103] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0106] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0107] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0108] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A global mask iterative pruning method, characterized in that: The method comprises: Constructing a target loss function, wherein the target loss function includes task loss, pruning loss and hardware resource loss, wherein the pruning loss is calculated according to the pruning weight matrix in each iteration process; the hardware resource loss is determined according to the real-time hardware resource information of the deployment device of the target model in the iteration process; Perform iterative training on the target model based on the target loss function. In any iterative training process, determine the pruning weight matrix of the target model in this iterative process according to the global mask matrix, determine the target loss of the target model in this iterative process, and update the global mask matrix of the next iterative process according to the target loss. Any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation; When the target model converges to the target state, the trained target model is obtained.

2. The method according to claim 1, characterized in that The constructed objective loss function is: Among them, L is the target loss function, is the task loss of the target model; is the pruning loss; Loss of hardware resources; is the loss weight of pruning loss; is the loss weight of hardware resource loss; is the element in the pruning weight matrix in any iteration process.

3. The method according to claim 2, characterized in that The method further comprises: During the iteration process of the target model, the hardware resource information of the deployment device of the target model during the iterative training process is obtained, and the hardware resource information includes the CPU utilization rate, memory utilization rate, power consumption of the deployment device and temperature of the deployment device.

4. The method according to claim 3, characterized in that The formula for hardware resource loss is: in, The CPU usage of the deployment device; Memory usage of the deployment device; The power consumption of the deployed equipment; The temperature of the deployed equipment; These are the weight parameters of CPU usage, memory usage, power consumption, and temperature.

5. The method according to claim 3, characterized in that: The method further comprises: Initialize the global mask matrix of the target model, the size of the global mask matrix is ​​consistent with the network weight matrix of the target model, and each element in the initialized global mask matrix is ​​1.

6. The method according to claim 5, characterized in that In any iterative training process, the pruning weight matrix of the target model in this iterative process is determined according to the global mask matrix, the target loss of the target model in this iterative process is determined, and the global mask matrix of the next iterative process is updated according to the target loss, including: In any iteration process, the pruning weight matrix is ​​obtained by multiplying the global mask matrix and the network weight matrix of this iteration process; Based on the pruning weight matrix, calculate the target loss of the target model in this iteration process; According to the target loss, the global mask matrix and network weight matrix of the next iteration process are gradient updated respectively.

7. The method according to claim 1, characterized in that After respectively performing gradient updates on the global mask matrix and the network weight matrix of the next iteration process according to the target loss, the method further includes: Determine the size of any mask element in the global mask matrix after gradient update and the pruning threshold; When any mask element is smaller than the pruning threshold, the mask element is set to 0.

8. A global mask iterative pruning device, characterized in that: The device comprises: A construction module is used to construct a target loss function, wherein the target loss function includes task loss, pruning loss and hardware resource loss, wherein the pruning loss is calculated according to the pruning weight matrix in each iteration process; and the hardware resource loss is determined according to the real-time hardware resource information of the deployment device of the target model in the iteration process; An iterative training module, used to perform iterative training on the target model based on the target loss function. In any iterative training process, the pruning weight matrix of the target model in this iterative process is determined according to the global mask matrix, the target loss of the target model in this iterative process is determined, and the global mask matrix of the next iterative process is updated according to the target loss. Any mask element in the global mask matrix is ​​used to determine whether any weight element in the network weight matrix of the target model participates in the operation; The model generation module is used to obtain the trained target model when the target model converges to the target state.

9. A computer device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the global mask iterative pruning method according to any one of claims 1 to 7 is executed.

10. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the global mask iterative pruning method according to any one of claims 1 to 7.

Images