Method, device and computer-readable storage medium for determining a pruning module

Abstract

The present application discloses a method, device, and computer-readable storage medium for determining a pruning module. The method for determining a pruning module includes: obtaining input tasks and the number of tasks of a neural network to be pruned, and output information of the component modules of the neural network to be pruned; determining the task association of the component modules based on the input tasks, the number of tasks, and the output information; determining the input information retention of the component modules based on the number of tasks and the output information; determining an importance index value of the component modules based on the task association and the input information retention; and determining the pruning module in the neural network to be pruned based on the importance index value.

Description

Technical Field

[0001] The present application relates to the field of neural network compression technology, and in particular to a method and device for determining a pruning module and a computer-readable storage medium. Background Art

[0002] Neural networks have achieved significant breakthroughs in numerous fields, including computer vision and natural language processing. However, in practical applications, their computational complexity and parameter storage requirements are relatively high, making them difficult to deploy on resource-constrained devices. To expand the applicability of neural networks, neural network pruning is often used to compress and reduce their computational complexity and parameter storage requirements.

[0003] However, existing neural network pruning solutions typically assess the importance of different parts of the network structure based on the weight parameters or data characteristics of the output feature graphs, and then prune them based on this importance. This data-driven assessment of the importance of different parts of the network structure only considers the data characteristics of the output data itself, resulting in a need for further improvement in the accuracy of pruning based on the assessed importance. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, and computer-readable storage medium for determining a pruning module, aiming to address the technical problem that, when evaluating the importance of different parts of a network structure based on data-driven analysis, only the data characteristics of the output data itself are considered, resulting in a need for further improvement in the accuracy of pruning when pruning based on the assessed importance.

[0005] An embodiment of the present application provides a method for determining a pruning module, the method comprising:

[0006] Obtaining input tasks and task quantities of the neural network to be pruned, as well as output information of the component modules of the neural network to be pruned;

[0007] Determining the task relevance of the component modules according to the input tasks, the number of tasks, and the output information;

[0008] Determining the input information retention degree of the component modules according to the number of tasks and the output information;

[0009] Determining the importance index value of the component module according to the task relevance and the input information retention;

[0010] A pruning module in the neural network to be pruned is determined according to the importance index value.

[0011] In one embodiment, the step of determining the task relevance of the component modules according to the input tasks, the number of tasks, and the output information includes:

[0012] Get the target sequence number of the component module in the current network layer;

[0013] Determine a target component module in the current network layer whose sequence number is less than or equal to the target sequence number;

[0014] Determining the input task retention degree and the sum of input and output information of each target component module according to the output information of each target component module and the input task;

[0015] The task association degree of the component modules is determined according to the task quantity, the input task retention degree and the sum of the input and output information.

[0016] In one embodiment, the input task includes at least one input task graph, the output information includes an output feature graph, and the step of determining the input task retention degree and the sum of the input and output information of each target component module based on the output information of each target component module and the input task includes:

[0017] Adjusting the resolution of the output feature map of each target component module so that the resolution of the output feature map is consistent with the resolution of the input task map;

[0018] Performing binarization processing on the input task graph and the adjusted output feature graph;

[0019] Taking the intersection of the processed input task graph and the processed output feature graph to obtain the input task retention degree of each target component module;

[0020] The processed input task graph and the processed output feature graph are taken as a union to obtain the sum of input and output information of each target component module.

[0021] In one embodiment, the step of determining the task association of the component modules according to the number of tasks, the input task retention, and the sum of the input and output information includes:

[0022] Obtaining a ratio of the input task retention degree to the sum of the input and output information to obtain a task retention ratio of each target component module;

[0023] Summing up the retention ratios of each task to obtain a task retention ratio sum value;

[0024] The quotient of the task retention ratio and value and the number of tasks is determined as the task association degree of the component module.

[0025] In one embodiment, the step of determining the input information retention degree of the component module according to the number of tasks and the output information includes:

[0026] Obtaining the target sequence number of the component module in the current network layer and the number of images included in the input task;

[0027] Determine a target component module in the current network layer whose sequence number is less than or equal to the target sequence number;

[0028] Determining a first energy value of each target component module in a current network layer according to output information of each target component module, and determining a second energy value of a component module in a previous network layer according to output information of each component module in a previous network layer;

[0029] The input information retention degree of the component module is determined according to the number of tasks, the first energy value, and the second energy value.

[0030] In one embodiment, the step of determining the first energy value of each target component module in the current network layer according to the output information of each target component module, and determining the second energy value of the component module in the previous network layer according to the output information of each component module in the previous network layer includes:

[0031] Calculating the norm of output information of each target component module, and using the square of the norm as the first energy value of each target component module;

[0032] The variance of the output information of each component module in the previous network layer is calculated to obtain the second energy value.

[0033] In one embodiment, the step of determining the input information retention of the component module according to the number of tasks, the first energy value, and the second energy value includes:

[0034] Obtaining a ratio between each of the first energy values ​​and the second energy value to obtain an energy proportion corresponding to each of the target component modules;

[0035] Summing up the energy proportions to obtain a sum of the energy proportions;

[0036] A quotient of the energy proportion and value and the number of tasks is determined as the input information retention degree of the component module.

[0037] In one embodiment, the step of determining the importance index value of the component module according to the task relevance and the input information retention includes:

[0038] Normalizing the task relevance and the input information retention;

[0039] The sum of the normalized task relevance and the normalized input information retention is used as the importance index value of the component module;

[0040] Alternatively, the product of the normalized task relevance and the normalized input information retention is used as the importance index value of the component module.

[0041] In addition, to achieve the above-mentioned purpose, the present application also provides a determination device for a pruning module, wherein the determination device for the pruning module includes a memory, a processor, and a pruning program of a network model stored in the memory and capable of running on the processor. When the pruning program of the network model is executed by the processor, the steps of the pruning method of the network model as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a pruning module determination program. When the pruning module determination program is executed by a processor, the steps of the pruning module determination method described above are implemented.

[0043] The pruning module determination method, apparatus, and computer-readable storage medium provided in the embodiments of the present application obtain input tasks and the number of tasks of a neural network to be pruned, as well as output information of the component modules of the neural network to be pruned. Then, the task association of each component module is determined based on the input tasks, the number of tasks, and the output information, and the input information retention of the component module is determined based on the number of tasks and the output information. Subsequently, an importance index value of the component module is determined based on the task association and the input information retention, so as to determine the pruning module in the neural network to be pruned based on the importance index value. This allows the output information of each component module to be associated with the input task to implement task-driven neural network pruning. This avoids the situation in which important modules associated with the input task are easily removed when evaluating the importance of each component module based solely on the data characteristics of the output data itself, thereby affecting the neural network's processing of the input task. This improves the accuracy of pruning module determination and thus improves the accuracy of neural network pruning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram of the structure of a device for determining a pruning module in a hardware operating environment according to an embodiment of the present application;

[0045] Figure 2 This is a flowchart of the first embodiment of the method for determining the pruning module of the present application;

[0046] Figure 3This is a flow chart of a second embodiment of the method for determining a pruning module of the present application;

[0047] Figure 4 This is a flowchart of a third embodiment of the method for determining a pruning module of the present application;

[0048] Figure 5 This is a flowchart of the fourth embodiment of the method for determining a pruning module of the present application.

[0049] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings are only an embodiment diagram, not the entire application. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0051] The main solution of the present application is: obtaining the input tasks and the number of tasks of the neural network to be pruned, as well as the output information of the constituent modules of the neural network to be pruned; determining the task association of the constituent modules based on the input tasks, the number of tasks and the output information; determining the input information retention of the constituent modules based on the number of tasks and the output information; determining the importance index value of the constituent modules based on the task association and the input information retention; and determining the pruning module in the neural network to be pruned based on the importance index value.

[0052] Most current neural network pruning solutions are based on data-driven evaluation of the importance of each component module. However, this approach only considers the data characteristics of the output data itself, while ignoring the relationship between the output data and the input task. As a result, components related to the input task are easily pruned during pruning, which is not conducive to improving pruning accuracy. Therefore, the above-mentioned solution proposed in this application aims to improve pruning accuracy.

[0053] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a device for determining a pruning module in a hardware operating environment according to an embodiment of the present application.

[0054] like Figure 1As shown, the determination device of the pruning module may include: a communication bus 1002, a processor 1001, such as a CPU, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art will understand that Figure 1 The structure of the determination device of the pruning module shown in the figure does not constitute a limitation on the determination device of the pruning module, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0056] exist Figure 1 In the device for determining a pruning module shown, the network interface 1004 is primarily used to connect to a backend server and communicate data with the backend server; the user interface 1003 is primarily used to connect to a client (user end) and communicate data with the client; and the processor 1001 can be used to call a control program of the device for determining a pruning module stored in the memory 1005 and execute the relevant steps of each embodiment of the following method for determining a pruning module.

[0057] Based on the system architecture of the above-mentioned pruning module determination device, the first embodiment of the pruning module determination method of the present application is proposed. Figure 2 In this embodiment, the method for determining the pruning module includes the following steps:

[0058] Step S10: Obtaining input tasks and task quantities of the neural network to be pruned, as well as output information of the component modules of the neural network to be pruned;

[0059] Step S20: determining the task relevance of the component modules according to the input tasks, the number of tasks and the output information;

[0060] Step S30: determining the input information retention degree of the component module according to the number of tasks and the output information;

[0061] Step S40: determining the importance index value of the component module according to the task relevance and the input information retention;

[0062] Step S50: Determine a pruning module in the neural network to be pruned according to the importance index value.

[0063] It should be noted that the neural network to be pruned refers to a neural network that needs to be pruned to achieve neural network compression, such as convolutional neural networks (CNN), deep neural networks (DNN), and recurrent neural networks (RNN); the input task of the neural network to be pruned refers to the input data that needs to be processed by the neural network to be pruned, such as a data table, sequence table, or image table; the output information of the component modules of the neural network to be pruned refers to the output data of the component modules of the neural network to be pruned, such as the output feature map. Each neural network to be pruned may include multiple component modules, and the component modules may include at least one of filters, channels, and parameters; each input task may include at least one input data, such as at least one input image.

[0064] Due to the high computational complexity and parameter storage requirements of neural networks, compression is necessary to improve processing efficiency and expand their applicability. Neural network pruning is a commonly used neural network compression method. However, some implementations often use data-driven methods to assess the importance of neural network components. For example, importance is determined by numerically ranking the results of L1 regularization, L2 regularization, or other regularization terms calculated for channel output feature maps; alternatively, importance is determined by sorting all elements in each weight parameter matrix from smallest to largest absolute value; or, filter contribution is determined by calculating the norm of the filter (L1 regularization, L2 regularization, or other regularization terms), with the norm being proportional to the contribution. This approach only considers the output characteristics of the output data itself when assessing the importance of different components within the network structure, ignoring the connection between the output data and the input task. Consequently, the accuracy of pruning neural networks based on assessed importance needs to be further improved. Therefore, the method for determining the pruning module proposed in this application can combine the connection between the output data of each component module and the input task to evaluate the importance of the neural network to be pruned, so as to more accurately determine the pruning module among the component modules of the neural network to be pruned, thereby improving the pruning accuracy when pruning the neural network to be pruned.

[0065] Specifically, after the input task is input into the neural network to be pruned, the output information of each component module of the neural network to be pruned can be obtained accordingly, and the input task input into the neural network to be pruned and the number of tasks corresponding to the input task are recorded, so as to determine the importance of each component module of the neural network to be pruned based on the input task and the number of tasks of the neural network to be pruned and the output information of the component modules of the neural network to be pruned.

[0066] When determining the importance of each component module of a neural network to be pruned, specifically: determining the task relevance of each component module based on the input tasks and number of tasks of the neural network to be pruned and the output information of the component modules of the neural network to be pruned; and determining the input information retention of each component module based on the number of tasks of the neural network to be pruned and the output information of each component module of the neural network to be pruned, and then determining the importance index value of each component module based on the task relevance and input information retention of each component module. Among them, the task relevance of each component module refers to the relevance between the output information of each component module and the input tasks of the neural network to be pruned; the input information retention of each component module refers to the amount of information of its input information contained in the output information of each component module.

[0067] Optionally, when determining the task relevance of each component module based on the input tasks and the number of tasks of the neural network to be pruned and the output information of the component modules of the neural network to be pruned, the influence of information transmission between the component modules of the same network layer is taken into account. Specifically, the following method can be used: the input tasks of the neural network to be pruned and the output information of the first N component modules in the current network layer, including the current component module N (N is a positive integer), are respectively intersected to obtain the amount of input task information contained in the first N component modules, and the amount of input task information contained in these first N component modules is summed, and then the sum values ​​obtained are averaged according to the number of tasks to obtain the task relevance corresponding to the current component module N; or, after determining the intersection and union of the output information of each component module and the input task, the ratio of the intersection and union corresponding to the first N component modules is summed according to the sequence number N of the current component module in the current network layer, and then the sum values ​​obtained are averaged according to the number of tasks to obtain the task relevance corresponding to the current component module, etc.

[0068] It is understandable that there may be other ways to determine the task relevance of each component module, which are merely listed here without any specific limitation.

[0069] Optionally, when determining the input information retention of each component module based on the number of tasks of the neural network to be pruned and the output information of each component module of the neural network to be pruned, specifically, it can be: considering the mutual influence between the component modules of the same network layer, after determining the energy values ​​corresponding to the output information of the first N component modules in the current network layer, including the current component module N (N is a positive integer), and summing the determined N energy values, the summed values ​​are averaged according to the number of tasks to obtain the input information retention of the current component module; or, considering the information flow between adjacent network layers and the mutual influence between the component modules of the same network layer, after determining the energy ratio corresponding to the output information of each component module and the output information of the previous network layer, the energy ratio corresponding to the first N component modules is summed according to the sequence number N of the current component module in the current network layer, and the average is taken according to the number of tasks to obtain the input information retention of the current component module, etc.

[0070] It is understandable that the method for determining the input information retention degree of each component module may also be other methods, which are only listed here and are not specifically limited.

[0071] Optionally, when determining the importance index value of each component module based on the task relevance and input information retention of each component module, it can be specifically: taking the sum of the task relevance and the input information retention as the importance index value; or taking the product of the task relevance and the input information retention as the importance index value; or taking the weighted sum of the task relevance and the input information retention as the importance index value, etc., which is not specifically limited here.

[0072] After determining the importance index values ​​of each component module, the pruned modules in the neural network to be pruned can be determined based on the importance index values. The pruned modules in the neural network to be pruned refer to the components of the neural network to be pruned that can be pruned, such as at least one of a filter, a channel, and a parameter. After determining the pruned modules in the neural network to be pruned, the pruned modules can be pruned to reduce the computational complexity and data storage requirements of the neural network to be pruned.

[0073] Optionally, the method for determining the pruned modules in the neural network to be pruned based on the importance index value may be: determining as pruned modules those modules whose importance index values ​​are within a preset range; or determining as pruned modules those modules whose importance index values ​​are higher than those of the number of pruned modules based on a preset pruning rate, etc., which are not specifically limited here. The preset range and the preset pruning rate can be set according to actual needs and are not specifically limited here.

[0074] This embodiment obtains the input tasks and number of tasks of the neural network to be pruned, as well as the output information of the component modules of the neural network to be pruned. Then, the task association of the component modules is determined based on the input tasks, number of tasks, and output information. The input information retention of the component modules is determined based on the number of tasks and output information. The importance index value of each component module is determined based on the task association and input information retention. The pruned modules in the neural network to be pruned are determined based on the importance index value. This allows the output information of each component module to be associated with the input tasks of the neural network to be pruned to evaluate the importance of each component module based on a task-driven approach. This avoids the situation where the importance is evaluated based solely on the data characteristics of the output data itself while ignoring the role of the input tasks in information processing of the neural network. This can easily lead to the mistaken pruning of component modules with a high degree of association with the input tasks, resulting in an impact on the processing of the input tasks. In this way, by associating the output information of each component module with the input tasks of the neural network to be pruned and evaluating the importance of each component module based on a task-driven approach, the pruning accuracy of the neural network to be pruned can be improved.

[0075] Based on the above first embodiment, a second embodiment of the method for determining the pruning module of the present application is proposed. Figure 3 In this embodiment, step S20 may include:

[0076] Step S21: obtaining the target sequence number of the component module in the current network layer;

[0077] Step S22: determining a target component module in the current network layer whose sequence number is less than or equal to the target sequence number;

[0078] Step S23: determining the input task retention degree and the sum of input and output information of each target component module according to the output information of each target component module and the input task;

[0079] Step S24: determining the task association of the component modules according to the number of tasks, the input task retention degree and the sum of the input and output information.

[0080] It should be noted that the neural network to be pruned may include multiple network layers, and each network layer may include multiple component modules. Based on this, different component modules in the same network layer can be numbered based on the connection relationship or processing order of each component module, so as to distinguish different component modules in the same network layer. For example, they can be sorted in ascending order according to the connection relationship; or they can be sorted in ascending order according to the processing order, with the earlier the processing order, the smaller the corresponding sorting number.

[0081] Because components at the same level can influence each other, thus affecting the task relevance of the current component module, before determining the task relevance of the current component module, the target component module in the current network level that has an impact on the information processing of the current component module can be first determined. Then, the task relevance of the current component module can be determined by combining the impact of the target component module and the processing capacity of the current component module itself. Specifically, the target sequence number of the current component module in the current network level can be determined first. For example, the current filter is the component module of the current network level. Component modules with sequence numbers less than the target sequence number are considered to have an impact on the task relevance of the current component module. In this way, the component modules in the current network level with sequence numbers less than or equal to the target sequence number can be determined as target components. For example, when the target sequence number is 1, the first component module is determined as the target component module. When the target sequence number is 3, the first, second, and third components are all determined as target components. Then, the task relevance of the current component module can be determined based on the output information of the target component module and the input tasks and number of tasks of the neural network to be pruned.

[0082] Specifically, when determining the task relevance of the current component module, the input task retention of each target component module and the sum of the input and output information of each target component module can be determined based on the output information of each target component module and the input tasks of the neural network to be pruned. Then, the task relevance of the current component module can be determined based on the number of tasks input into the neural network to be pruned and the input task retention and sum of the input and output information of each target component module. Among them, the input task retention of each target component module refers to the amount of information about the input task contained in the output information of each target component module; the sum of the input and output information of each target component module refers to the sum of the task information contained in the output information of each target component module and the input tasks of the neural network to be pruned.

[0083] Optionally, when determining the input task retention degree and the sum of input and output information of each target component module based on the output information of each target component module and the input task of the neural network to be pruned, specifically: taking the intersection of the output information of the target component module and the input task of the neural network to be pruned to obtain the input task retention degree of each target component module; and taking the union of the output information of the target component module and the input task of the neural network to be pruned to obtain the sum of input and output information of each target component module.

[0084] Optionally, when the input task is image classification as an example, if the input task of the neural network to be pruned includes at least one input task graph, and correspondingly, the output information of each component module includes at least one output feature graph, then in order to facilitate information comparison, before determining the input task retention degree and the sum of input and output information of each target component module, the resolution of the output feature graph of each target component module can be adjusted to adjust the resolution of the output feature graph to be consistent with the resolution of the input task graph.

[0085] Optionally, in order to reduce the amount of computation and highlight the outline of the target of interest, the output feature map and the input task map may be binarized before determining the input task retention and the sum of input and output information of each target component module.

[0086] Optionally, when determining the task association of the current component module based on the number of tasks input into the neural network to be pruned and the input task retention and the sum of input and output information of each target component module, the specific method can be: first calculate the ratio of the input task retention and the sum of input and output information of each target component module to obtain the task retention ratio; then sum the task retention ratios of each target component module to obtain the total task retention ratio; then divide the total task retention ratio by the number of tasks to obtain the average task retention value corresponding to each task, and use the average task retention value as the task association of the current component module.

[0087] In a specific application example, the resolution of the output feature map of each target component module can be adjusted to adjust the resolution of the output feature map to be consistent with the resolution of the input task map; then the input task map is binarized to obtain a binarized input task map; and the adjusted output feature map is binarized to obtain a binarized output feature map; then the intersection of the binarized input task map and the binarized output feature map is taken to obtain the input task retention degree of each target component module; the union of the binarized input task map and the binarized output feature map is taken to obtain the sum of input and output information of each target component module; then, the ratio of the input task retention degree of each target component module to the sum of input and output information is obtained to obtain the task retention ratio of each target component module; and the obtained task retention ratios are summed to obtain the task retention ratio sum value of each target component module; then, the quotient of the task retention ratio sum value of each target component module and the number of tasks is calculated to obtain the task association degree of the current component module.

[0088] For example, for the first Layer Assume that the grayscale image of the input task graph x is recorded as , the first Layer The network weight parameters of the component modules are , and its corresponding output feature map is recorded as , then calculate the first Layer The task relevance of each component module may include the following steps:

[0089] 1) Bilinear interpolation can be used to The resolution of the task map is adjusted to the same resolution as the input task map x, and the output feature map after adjustment is recorded as .

[0090] 2) Based on set threshold ,right and Perform binarization processing to obtain the input task graph after binarization processing, which is recorded as , and the output feature map after binarization processing, recorded as The calculation formula is as follows:

[0091]

[0092] It is worth noting that the threshold set The difference in will not affect the final calculated importance index value.

[0093] (3) If the task retention ratio of each target component module with a sequence number less than or equal to i is recorded as , then according to the number of tasks d Taking the average value, we can get the first Layer The task relevance of the component modules is recorded as The specific calculation formula is as follows:

[0094]

[0095] It is worth noting that The larger the Layer The stronger the ability of a component module to extract semantic information of the input task, the more important it is to the input task. In other words, each component module can be regarded as a semantic extractor related to the input task, and the output feature map of each component module can be regarded as a feature map containing the semantic information of the input task. By determining how much semantic information related to the input task is contained in the output feature map, the connection between the input task and the feature map is evaluated, thereby determining the semantic extraction ability of the filter that outputs this feature map for the task.

[0096] This embodiment determines the task relevance of the current component module by the number of tasks corresponding to the input tasks of the neural network to be pruned, the input task retention of the target component module, and the sum of input and output information. This not only allows the output information of the current component module to be associated with the input tasks, but also fully considers the impact of the component modules that are ranked higher in the current network hierarchy on the task relevance of the current component module. This can improve the accuracy of determining the task relevance of the current component module, and further improve the accuracy of determining the importance index value of the current component module, thereby improving pruning accuracy.

[0097] Based on the above first embodiment, a third embodiment of the method for determining the pruning module of the present application is proposed. Figure 4 In this embodiment, step S30 may include:

[0098] Step S31: obtaining the target sequence number of the component module in the current network layer and the number of images included in the input task;

[0099] Step S32: determining a target component module in the current network layer whose sequence number is less than or equal to the target sequence number;

[0100] Step S33: determining a first energy value of each target component module in the current network layer according to the output information of each target component module, and determining a second energy value of the component module in the previous network layer according to the output information of each component module in the previous network layer;

[0101] Step S34: determining the input information retention degree of the component module according to the number of tasks, the first energy value, and the second energy value.

[0102] It should be noted that the neural network to be pruned may include multiple network layers, and each network layer may include multiple component modules. Based on this, different component modules in the same network layer can be numbered based on the connection relationship or processing order of each component module, so as to distinguish different component modules in the same network layer. For example, they can be sorted in ascending order according to the connection relationship; or they can be sorted in ascending order according to the processing order, with the earlier the processing order, the smaller the corresponding sorting number.

[0103] Since there is mutual influence between the components of the same layer, but there is no mutual influence between the components of the interlayer, the information flow in the components of the adjacent layers can be measured by analyzing the amount of information acquisition between the components of the adjacent layers to evaluate the input information retention of the current component module. Specifically, before determining the input information retention of the current component module, the target component module in the current network layer that has an impact on the input information retention of the current component module can be determined. For example, the target sequence number of the current component module in the current network layer where it is located can be determined first, for example, the number of filters in the current network layer where the current filter is located. Component modules with a sequence number less than the target sequence number are considered to be target component modules that have an impact on the input information retention of the current component module. In this way, the component modules with sequence numbers less than or equal to the target sequence number in the current network layer can be determined as target component modules, and then the input information retention of the current component module can be determined based on the output information of the target component module and the number of tasks corresponding to the input tasks of the neural network to be pruned.

[0104] When determining the input information retention of the current component module based on the output information of the target component module and the number of tasks in the neural network to be pruned, the following steps may be performed: first, a first energy value of each target component module in the current network layer is determined based on the output information of each target component module; and second energy values ​​of the component modules in the previous network layer are determined based on the output information of each component module in the previous network layer; and then, the input information retention of the current component module is determined based on the number of tasks, the first energy value, and the second energy value. The first energy value refers to the energy value corresponding to the output information of each target component module, and the second energy value refers to the energy value corresponding to the output information of the previous network layer.

[0105] Optionally, the norm of the output information of each target component module can be calculated, and the square of the calculated norm can be used as the first energy value of each target component module. The norm here specifically refers to the L2 norm to improve the accuracy of the first energy value calculation. Of course, other norms, such as the L1 norm, can also be used in other embodiments, which are not specifically limited here. The L2 norm refers to the square root of the sum of the squares of each element in the vector; the L1 norm refers to the sum of the absolute values ​​of each element in the vector, also known as the "sparse rule operator."

[0106] Optionally, the variance of the output information of each component group module in the previous network layer can be obtained as the second energy value corresponding to the previous network layer.

[0107] Optionally, when determining the input information retention of the current component module based on the number of tasks, the first energy value and the second energy value, the ratio between the first energy value and the second energy value of each target component module can be obtained first to obtain the energy proportion corresponding to each target component module, and then the energy proportions are summed to obtain the energy proportion sum value, and then the obtained energy proportion sum value is divided by the number of tasks to obtain the input information retention of the current component module.

[0108] In a specific application example, when the input task is image classification, if the input task of the neural network to be pruned includes at least one input task graph, and the output information of each component module includes at least one output feature graph, then for the first Layer Assume that the network Layer The network weight parameters of the component modules are , and its corresponding output feature map is recorded as , the energy proportion of each target component module is recorded as , the energy ratio is calculated based on the number of tasks d input into the neural network to be pruned. Taking the average value, we can get Layer The input information retention of each component module. The specific calculation formula is as follows:

[0109]

[0110] in, Indicates that when the input task graph is When the network Layer filters (the network weight parameters are )’s output feature map; express The square of the L2 norm of ; express The variance of all output feature maps of the layer. It can be used to characterize the first Layer The information acquisition capability of each filter is calculated as follows: Here, the calculated The larger the Layer The more information a filter can obtain about the input task, the more important it is to the input task.

[0111] This embodiment determines the input information retention of the component modules by the number of tasks corresponding to the input tasks of the neural network to be pruned, the first energy value and the second energy value corresponding to the input tasks of the neural network to be pruned, so that the input information retention of the component modules can be evaluated according to the information flow between different network layers, so as to improve the accuracy of determining the input information retention of the component modules, thereby improving the accuracy of determining the importance index value of the current component module and improving the pruning accuracy.

[0112] Based on the above first embodiment, a fourth embodiment of the method for determining the pruning module of the present application is proposed. Figure 5 In this embodiment, step S40 may include:

[0113] Step S41: normalizing the task relevance and the input information retention;

[0114] Step S42: taking the sum of the normalized task relevance and the normalized input information retention as the importance index value of the component module; or,

[0115] Step S43: taking the product of the normalized task relevance and the normalized input information retention as the importance index value of the component module.

[0116] Since the importance of each component module in the neural network to be pruned is related to the task relevance and input information retention of each component module, the importance index value of each component module can be determined according to the task relevance and input information retention to evaluate the importance of each component module.

[0117] For example, the task relevance and input information retention of each component module can be summed, and the sum obtained can be used as the importance index value of each component module; or, the product of the task relevance and input information retention of each component module can be used as the importance index value of each component module; or, after summing the task relevance and input information retention of each component module and calculating the product of the task relevance and input information retention of each component module, different weight values ​​can be assigned to the obtained sum and product, and the sum and product can be weightedly summed with the assigned weight values, and the weighted sum obtained by the weighted summation can be used as the importance index value of each component module, etc.

[0118] In one embodiment, in order to unify the magnitude and reduce the interference of abnormal data, the task relevance and input information retention of each component module can be normalized first, and the sum or product of the normalized task relevance and input information retention can be obtained as the importance index value of each component module. Specifically, the maximum task relevance among the task relevances of all component modules in the current network layer and the maximum input information retention among the input information retention of all component modules in the current network layer can be obtained first; then, the quotient between the task relevance of the current component module and the maximum task relevance can be obtained to obtain the normalized task relevance, and the quotient between the input information retention of the current component module and the maximum input information retention can be obtained to obtain the normalized input information retention; then, the normalized task relevance and the normalized input information retention can be summed to obtain the importance index value of the corresponding component module; alternatively, the product of the normalized task relevance and the normalized input information retention can be obtained to obtain the importance index value of the corresponding component module.

[0119] This embodiment obtains the importance index value of each component module by summing or multiplying the normalized task relevance and input information retention, so that the importance index value of each component module can be determined based on task driving, so as to improve the accuracy of the importance index value and thus improve the pruning accuracy of the neural network to be pruned.

[0120] In a specific application example, if the component modules are filters in a neural network to be pruned, the input information of the neural network to be pruned includes at least one input task graph, and the output information of each component module includes an output feature graph, the importance index value of each filter can be evaluated by evaluating the connection between the input task and the output feature graph (task correlation) and measuring the information flow in the filters between adjacent network layers (input information retention), and then the pruned modules in the neural network to be pruned are determined based on the importance index value. The specific process is as follows:

[0121] 1. Evaluate the connection between the input task and the feature map (task relevance)

[0122] Each filter can be regarded as a semantic extractor related to the input task, and the output feature map of the filter can be regarded as a feature map containing the semantic information of the input task. The connection between the input task and the feature map is evaluated by determining how much semantic information related to the input task is contained in the output feature map, thereby determining the semantic extraction ability of the filter that outputs the output feature map for the input task, and using this ability as an indicator to evaluate the importance of the filter.

[0123] Specifically: Assume that the input task graph is , get its grayscale image Input the input task graph into the neural network to be pruned. Assume that the first Layer The network weight parameters of the filter are , the corresponding output feature map can be recorded as Then, the output feature map is expanded to an image with the same resolution as the input task map by bilinear interpolation, and based on the set threshold, the following formula (1) is used to implement and Binarization operation:

[0124]

[0125] in, and For and The result obtained after the binarization operation; and setting thresholds of different values , which has no impact on the final determined importance index value.

[0126] Assume that the first Layer The importance quantification score of the filter (task retention ratio) is , then for all input tasks we get The final average can be obtained Layer Quantified importance score of each filter (task relevance) , The semantic extraction capability of the filter corresponding to the current output feature map for the input task can be used to characterize the connection between the input task and the output feature map. The specific calculation process is shown in formula (2):

[0127]

[0128] in, represents the number of input task graphs in the input task, that is, the number of tasks, and It can be obtained by calculating the connection between the input task and the feature map. The specific calculation process is to find and The intersection-and-union ratio of Here, the calculated The larger the Layer The stronger the ability of a filter to extract the semantic information of the input task, the more important it is to the input task.

[0129] 2. Determine the information flow in filters between adjacent layers (input information retention)

[0130] During forward inference in a neural network to be pruned, information from the input task flows between adjacent layers, layer by layer, from shallow to deep. The filters in a given layer input the output feature map of the previous layer and output the output feature map of the current layer, meaning that filters in adjacent layers have no influence on each other. Therefore, the filters in each layer can be viewed as finite state machines, and the information flow between filters in adjacent layers can be measured by analyzing the amount of information acquired between filters in adjacent layers.

[0131] Specifically, assuming that when the task image is input When the first Layer The importance quantification score (energy ratio) of each filter is , then for all input tasks we get The final average can be obtained Layer The importance quantification score of each filter (input information preservation) , to represent the first Layer The information acquisition capability of each filter. The specific calculation process is shown in formula (3):

[0132]

[0133] in, Indicates that when the task picture is input When the network Layer filters (the network weight parameters are )’s output feature map; express The square of the L2 norm of ; express The variance of all output feature maps of the layer. It can be calculated when the input task picture When the network Layer The information acquisition ability of each filter is obtained, and the specific calculation process is: Here, the calculated The larger the Layer The more information a filter can obtain about the input task, the more important it is to the input task.

[0134] 3. Combine to obtain the final task-driven importance assessment strategy

[0135] The above obtained and Combine them to calculate the importance index value of each filter, as shown in formula (4):

[0136]

[0137] here, and They represent the first All filters in the layer The maximum value in , and ( It is the first The maximum value among the number of filters that the layer has.

[0138] To validate the performance of this invention, at least four typical neural networks, such as VGG-16, ResNet-56, ResNet-110, and ResNet-50, were used as the neural networks to be pruned. Network model pruning tests were conducted on two test datasets, CIFAR-10 and ImageNet (ILSVRC2012). Experimental results demonstrate that the task-driven pruning scheme proposed in this application achieves superior network pruning performance compared to traditional data-driven pruning schemes, including higher compression rates, reduced parameter storage requirements, and lower computational complexity.

[0139] That is, both of the above two combination calculation schemes of importance index values ​​can achieve good results and can be selected according to specific circumstances.

[0140] 4. Joint Training Framework

[0141] Assume that the loss function of the task is as shown in formula (5):

[0142]

[0143] in is the output of the network, and is each input training example The loss function calculation formula (which can be cross entropy loss or other specific calculation formula). is the training example The corresponding groundtruth.

[0144] The specific process of pruning is as follows: Assume that The pruning rate of the layer is preset to , we get the The importance index value of each filter in the layer ( It is the first The number of filters in the layer can be obtained by The first The filters with smaller importance quantization scores in the layer are removed while the filters with larger importance quantization scores are retained to achieve pruning. The sparsity of the layer is preset to , then 10% of the filters in the first layer will be retained, and the remaining 90% of the filters will be pruned. The importance index value of each filter in the layer , the first 10% The filter with the largest value is retained, and the rest of the filters are removed to finally achieve the first Layer pruning.

[0145] The specific training framework is as follows:

[0146] 1) Input: training data pair , pre-trained model parameters ( is the number of layers in the neural network to be pruned), the pre-set network pruning rate , learning rate , the number of training iterations , training stop condition .

[0147] 2) Output: The final pruned network model parameters .

[0148] The task-driven importance index value of the neural network to be pruned is calculated by the above formulas (1) to (4) ;

[0149] 3) Order ;

[0150] 4) Pass and Prune the neural network layer by layer;

[0151] 5) When hour, ;

[0152] 6) Output the pruned network model parameters .

[0153] It should be noted that step 3 is the process of pruning the neural network model based on task-driven; steps 4 to 5 are the process of retraining and fine-tuning. The number of training iterations represented by The pruned model parameters obtained at this time The number of training iterations is ( The pruned model parameters obtained at this time The L2 norm of the difference between . , the pruned network model parameters will be iteratively retrained and fine-tuned continuously.

[0154] In addition, an embodiment of the present application also provides a device for determining a pruning module, wherein the device for determining a pruning module includes a memory, a processor, and a pruning module determination program stored on the processor and runnable on the processor. When the processor executes the pruning module determination program, the steps of the pruning module determination method described above are implemented.

[0155] In addition, an embodiment of the present application further provides a computer-readable storage medium, on which a pruning module determination program is stored. When the pruning module determination program is executed by a processor, the steps of the pruning module determination method described above are implemented.

[0156] It should be noted that, in this document, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0157] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0159] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for determining a pruning module, characterized in that: The method for determining the pruning module comprises the following steps: Obtaining input tasks and the number of tasks of the neural network to be pruned, and output information of the component modules of the neural network to be pruned, wherein the input tasks include at least one input task graph, and the output information includes an output feature graph; Get the target sequence number of the component module in the current network layer; Determine a target component module in the current network layer whose sequence number is less than or equal to the target sequence number; Adjusting the resolution of the output feature map of each target component module so that the resolution of the output feature map is consistent with the resolution of the input task map; Performing binarization processing on the input task graph and the adjusted output feature graph; Taking the intersection of the binarized input task graph and the binarized output feature graph to obtain the input task retention degree of each target component module; Taking the union of the binarized input task graph and the binarized output feature graph to obtain the sum of input and output information of each target component module; Determining the task association of the component modules according to the number of tasks, the input task retention degree, and the sum of the input and output information, wherein the task association of each component module refers to the association between the output information of each component module and the input task of the neural network to be pruned; Determining the input information retention of the component modules according to the number of tasks and the output information, wherein the input information retention of each component module refers to the amount of its input information contained in the output information of each component module; Determining the importance index value of the component module according to the task relevance and the input information retention; A pruning module in the neural network to be pruned is determined according to the importance index value.

2. The method for determining a pruning module according to claim 1, wherein: The step of determining the task association of the component modules according to the number of tasks, the input task retention and the sum of the input and output information includes: Obtaining a ratio of the input task retention degree to the sum of the input and output information to obtain a task retention ratio of each target component module; Summing up the retention ratios of each task to obtain a task retention ratio sum value; The quotient of the task retention ratio and value and the number of tasks is determined as the task association degree of the component module.

3. The method for determining a pruning module according to claim 1, wherein: The step of determining the input information retention degree of the component module according to the number of tasks and the output information includes: Obtaining the target sequence number of the component module in the current network layer and the number of images included in the input task; Determine a target component module in the current network layer whose sequence number is less than or equal to the target sequence number; Determining a first energy value of each target component module in a current network layer according to output information of each target component module, and determining a second energy value of a component module in a previous network layer according to output information of each component module in a previous network layer; The input information retention degree of the component module is determined according to the number of tasks, the first energy value, and the second energy value.

4. The method for determining a pruning module according to claim 3, wherein: The step of determining the first energy value of each target component module in the current network layer according to the output information of each target component module, and determining the second energy value of the component module in the previous network layer according to the output information of each component module in the previous network layer includes: Calculating the norm of output information of each target component module, and using the square of the norm as the first energy value of each target component module; The variance of the output information of each component module in the previous network layer is calculated to obtain the second energy value.

5. The method for determining a pruning module according to claim 3, wherein: The step of determining the input information retention degree of the component module according to the number of tasks, the first energy value, and the second energy value includes: Obtaining a ratio between each of the first energy values ​​and the second energy value to obtain an energy proportion corresponding to each of the target component modules; Summing up the energy proportions to obtain a sum of the energy proportions; A quotient of the energy proportion and value and the number of tasks is determined as the input information retention degree of the component module.

6. The method for determining a pruning module according to claim 1, wherein: The step of determining the importance index value of the component module according to the task relevance and the input information retention includes: Normalizing the task relevance and the input information retention; The sum of the normalized task relevance and the normalized input information retention is used as the importance index value of the component module; Alternatively, the product of the normalized task relevance and the normalized input information retention is used as the importance index value of the component module.

7. A device for determining a pruning module, characterized in that: The determination device of the pruning module includes a memory, a processor, and a pruning program of the network model stored in the memory and capable of running on the processor. When the processor executes the pruning program of the network model, the steps of the pruning method of the network model described in any one of claims 1-6 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a pruning program for a network model, and when the pruning program for the network model is executed by a processor, the steps of the pruning method for a network model according to any one of claims 1 to 6 are implemented.

Images