Neural network optimization methods, devices, electronic equipment, storage media and chips

Abstract

This disclosure provides a neural network optimization method, apparatus, electronic device, storage medium, and chip. The method includes: acquiring a neural network to be compressed; wherein the neural network to be compressed includes multiple network blocks to be compressed; in response to performing a compression operation on the multiple network blocks to be compressed to obtain multiple compressed network blocks, determining a target error for each compressed network block; wherein the target error is used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network blocks to be compressed with the compressed network blocks; and / or, the degree of difference between the network blocks to be compressed and the compressed network blocks; adjusting the parameters of each compressed network block according to the processing order determined based on the target error to obtain a target network block; and training the neural network to be compressed, which includes multiple target network blocks, to obtain a target neural network.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, and more specifically, to a neural network optimization method, apparatus, electronic device, storage medium, and chip. Background Technology

[0002] With the research and development of deep learning, neural networks have been widely applied in various technical fields, such as autonomous driving, public safety, and augmented reality. To ensure neural network performance, their size can be set relatively large, resulting in a large number of parameters. This leads to the consumption of significant storage and computing resources during neural network computation, posing a significant challenge to deploying neural networks on resource-constrained devices.

[0003] Therefore, it is particularly important to propose a neural network optimization method. Summary of the Invention

[0004] In view of this, the present disclosure provides at least one neural network optimization method, apparatus, electronic device, storage medium, and chip.

[0005] In a first aspect, this disclosure provides a neural network optimization method, including:

[0006] Obtain a neural network to be compressed; wherein the neural network to be compressed includes multiple network blocks to be compressed;

[0007] In response to performing a compression operation on the plurality of network blocks to be compressed to obtain a plurality of compressed network blocks, a target error is determined for each of the compressed network blocks; wherein the target error is used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; and / or, the degree of difference between the network block to be compressed and the compressed network block;

[0008] Following the processing order determined based on the target error, the parameters of each compressed network block are adjusted to obtain the target network block;

[0009] The target neural network is obtained by training a neural network to be compressed, which includes multiple target network blocks.

[0010] In the above method, a target error is determined for each compressed network block. This target error characterizes the degree of difference between the network block to be compressed and the compressed network block; and / or, characterizes the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block. That is, by using the target error of each compressed network block, the influence of each compressed network block on the neural network and / or the influence of the compression process on the network block to be compressed can be determined. Furthermore, based on this target error, the processing order of each network block to be compressed can be determined more accurately; for example, the network blocks to be compressed with a greater influence can be processed first, and the network blocks to be compressed with a smaller influence can be processed later.

[0011] By considering the impact of different network blocks on the inference results of the compressed neural network, a more accurate target network block can be obtained when adjusting the parameters of each compressed network block according to this processing order. Therefore, training the compressed neural network, which includes multiple target network blocks, yields a target neural network with better performance. Simultaneously, compression of the compressed neural network can be performed more quickly while maintaining its performance, thus improving the optimization efficiency of the compressed neural network.

[0012] In one possible implementation, the target error includes a compression error characterizing the degree of difference between the network block to be compressed and the compressed network block; determining the target error for each compressed network block includes:

[0013] Obtain the first sample data;

[0014] Using the network block to be compressed, feature extraction is performed on the first sample data to obtain the first output feature data corresponding to the first sample data; and

[0015] Using the compressed network block, feature extraction is performed on the first sample data to obtain the second output feature data corresponding to the first sample data;

[0016] The compression error of the compressed network block is determined using a first data pair consisting of the first output feature data and the second output feature data, and / or a second data pair consisting of the weight data of the network block to be compressed and the weight data of the compressed network block.

[0017] Here, features are extracted from the first sample data using the network block to be compressed and the compressed network block, respectively, to obtain first output feature data and second output feature data. Then, using the first data pair composed of the first output feature data and the second output feature data, and / or the second data pair composed of the weight data of the network block to be compressed and the weight data of the compressed network block, the compression error of the compressed network block can be determined more accurately, providing data support for subsequent determination of the processing order based on the target error.

[0018] In one possible implementation, the target error includes an inference error characterizing the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; determining the target error for each of the compressed network blocks includes:

[0019] Obtain the second sample data;

[0020] The second sample data is input into the neural network to be compressed to obtain the first detection result corresponding to the second sample data; and

[0021] In the neural network to be compressed, the network block to be compressed is replaced with the corresponding compressed network block to obtain the replaced neural network to be compressed; the second sample data is input into the replaced neural network to be compressed to obtain the second detection result corresponding to the second sample data;

[0022] The inference error of the compressed network block is determined using the first detection result and the second detection result.

[0023] Here, the second sample data is input into the neural network to be compressed to obtain the first detection result corresponding to the second sample data; and the second sample data is input into the replaced neural network to be compressed to obtain the second detection result corresponding to the second sample data. Using the first and second detection results, the inference error of the compressed network block can be determined more accurately, providing data support for subsequent determination of the processing order based on the target error.

[0024] In one possible implementation, adjusting the parameters of each compressed network block according to the processing order determined based on the target error to obtain the target network block includes:

[0025] According to the processing order of the target error from largest to smallest, the first network block to be processed is determined from the multiple compressed network blocks; and the first network block to be processed is determined as the current network block to be processed, and the neural network to be compressed is determined as the current neural network;

[0026] In the current neural network, the network block to be compressed corresponding to the current network block to be processed is replaced with the current network block to be processed to obtain an intermediate neural network;

[0027] Based on the intermediate neural network, the parameters of the current network block to be processed are adjusted to obtain the target network block corresponding to the current network block to be processed and the intermediate neural network including the target network block;

[0028] Based on the processing order, the next network block to be processed is determined from the multiple compressed network blocks, and the next network block to be processed is determined as the current network block to be processed. The intermediate neural network including the target network block is determined as the current neural network. The process returns to the step of replacing the network block to be compressed corresponding to the current network block to be processed in the current neural network with the current network block to be processed to obtain the intermediate neural network, until the target network block corresponding to each compressed network block is obtained.

[0029] In this embodiment of the disclosure, by processing the compressed network blocks with larger target errors in descending order, first adjusting the parameters of the compressed network blocks with larger target errors, and then adjusting the parameters of the compressed network blocks with smaller target errors, the target network blocks corresponding to each compressed network block can be obtained more accurately.

[0030] In one possible implementation, adjusting the parameters of each compressed network block according to the processing order determined based on the target error to obtain the target network block includes:

[0031] According to the processing order of the target error from largest to smallest, each of the multiple compressed network blocks is taken as a network block to be processed.

[0032] In the neural network to be compressed, the network block to be processed corresponding to the network block to be compressed is replaced with the network block to be processed to obtain an intermediate neural network;

[0033] Based on the intermediate neural network, the parameters of the network block to be processed are adjusted to obtain the target network block.

[0034] Considering that different network blocks have different degrees of influence on the inference results of the neural network and different robustness to compression operations, in order to more accurately determine the target network block, the parameters of each compressed network block can be adjusted in descending order of target error.

[0035] In one possible implementation, the step of adjusting the parameters of the network block to be processed based on the intermediate neural network to obtain the target network block includes:

[0036] Obtain third sample data;

[0037] The third sample data is input into the intermediate neural network to obtain the intermediate detection result corresponding to the third sample data;

[0038] Based on the intermediate detection results, the parameters of the network block to be processed included in the intermediate neural network are adjusted to obtain the intermediate network block;

[0039] The intermediate neural network including the intermediate network block is used as the updated intermediate neural network, and the process is returned to the step of obtaining the third sample data until the training cutoff condition is met. The training cutoff condition includes: the number of parameter adjustments is equal to the number threshold corresponding to the network block to be processed, and / or, until the target error of the intermediate network block is less than the error threshold.

[0040] The intermediate network block obtained after the last parameter adjustment is determined as the target network block.

[0041] Here, by setting a threshold for the number of iterations and / or an error threshold, the cutoff conditions for adjusting the parameters of the network block to be processed can be flexibly determined, thereby enabling the desired target network block to be obtained flexibly.

[0042] In one possible implementation, the number of iterations threshold corresponding to the network block to be processed is determined according to the following steps:

[0043] Based on the target error corresponding to the network block to be processed and the mapping relationship between the error and the number of times, the number of times threshold corresponding to the network block to be processed is determined, wherein the target error and the number of times threshold are positively correlated.

[0044] Here, based on the target error corresponding to the network block to be processed and the mapping relationship between error and number of times, the number of times threshold corresponding to each network block to be processed can be flexibly determined. Different network blocks to be processed can correspond to different number of times thresholds, which improves the efficiency and flexibility of adjusting the parameters of the network block to be processed.

[0045] In one possible implementation, obtaining the neural network to be compressed includes:

[0046] Obtain the original trained neural network;

[0047] Based on the set determining factors, and using at least one feature processing layer included in the original neural network, multiple network blocks to be compressed are determined.

[0048] The original neural network, which includes the plurality of network blocks to be compressed, is identified as the neural network to be compressed.

[0049] Here, after obtaining the trained original neural network, multiple network blocks to be compressed can be accurately determined based on at least one feature processing layer included in the original neural network, according to the set deterministic factors; so that the compression operation of the network blocks to be compressed can be performed subsequently to optimize the neural network to be compressed.

[0050] In one possible implementation, the determining factors include at least one of the following:

[0051] The amount of data for the parameters included in the feature processing layer;

[0052] The size of the input feature data corresponding to the feature processing layer;

[0053] The size of the output feature data corresponding to the feature processing layer;

[0054] The threshold number of feature processing layers included in the network block to be compressed.

[0055] Here, at least one determinant is set so that the network block to be compressed can be flexibly determined using the determinant.

[0056] Secondly, this disclosure provides an artificial intelligence (AI) chip, including: a processor and a memory; the memory is used to store network parameters of the target neural network; the target neural network is generated based on the neural network optimization method described in the first aspect or any embodiment.

[0057] The processor is configured to run the target neural network based on the network parameters read from the memory.

[0058] Here, by utilizing the neural network optimization method described in the first aspect or any embodiment, the neural network to be compressed is optimized to generate the target neural network. While ensuring the accuracy of the neural network, the number of network parameters is reduced, resulting in a smaller number of network parameters in the target neural network. Storing the network parameters of the target neural network in memory reduces storage resource consumption. Simultaneously, when the processor runs the target neural network, the smaller number of parameters used improves the processor's running efficiency.

[0059] The effects of the following devices, electronic equipment, etc., are described in the instructions above and will not be repeated here.

[0060] Thirdly, this disclosure provides a neural network optimization device, comprising:

[0061] An acquisition module is used to acquire a neural network to be compressed; wherein the neural network to be compressed includes multiple network blocks to be compressed;

[0062] A first determining module is configured to determine a target error for each compressed network block in response to performing a compression operation on the plurality of network blocks to be compressed to obtain a plurality of compressed network blocks; wherein the target error is used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; and / or the degree of difference between the network block to be compressed and the compressed network block.

[0063] An adjustment module is used to adjust the parameters of each compressed network block according to the processing order determined based on the target error, so as to obtain the target network block;

[0064] The training module is used to train the neural network to be compressed, which includes multiple target network blocks, to obtain the target neural network.

[0065] Fourthly, this disclosure provides an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, they perform the steps of the neural network optimization method as described in the first aspect or any embodiment above, or include the chip as described in the second aspect above.

[0066] Fifthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the neural network optimization method as described in the first aspect or any of the embodiments above.

[0067] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0068] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0069] Figure 1 A flowchart illustrating a neural network optimization method provided in an embodiment of this disclosure is shown.

[0070] Figure 2The diagram illustrates various compression operations in a neural network optimization method provided by an embodiment of this disclosure.

[0071] Figure 3 The diagram illustrates a flowchart of determining compression error in a neural network optimization method provided in an embodiment of this disclosure.

[0072] Figure 4 The diagram illustrates a flowchart of determining inference error in a neural network optimization method provided in an embodiment of this disclosure.

[0073] Figure 5 A schematic diagram of the architecture of an artificial intelligence (AI) chip provided in an embodiment of this disclosure is shown;

[0074] Figure 6 This illustration shows a schematic diagram of the architecture of a neural network optimization device provided in an embodiment of the present disclosure;

[0075] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0077] With the research and development of deep learning, neural networks have been widely applied in various technical fields, such as autonomous driving, public safety, and augmented reality. To ensure neural network performance, their size can be set relatively large, resulting in a large number of parameters. This leads to the consumption of significant storage and computing resources during neural network computation, posing a significant challenge to deploying neural networks on resource-constrained devices.

[0078] Generally, neural networks can be compressed to optimize them. This can be done through methods such as overall compression or incremental compression. Overall compression typically treats the neural network as a whole, performing pruning, quantization, and other compression operations on its overall structure to obtain a compressed neural network. This compressed network is then trained to optimize its parameters, resulting in an optimized neural network. However, overall compression requires manipulating a large number of parameters, and the overall compression operation has a significant impact on the performance of the neural network, potentially causing a substantial decrease in performance and resulting in poor compression effectiveness.

[0079] Progressive compression typically follows the data flow sequence in a neural network, sequentially compressing and adjusting the parameters of multiple feature processing layers to obtain compressed feature processing layers. The neural network, including these compressed feature processing layers, is then trained to obtain an optimized neural network. However, this progressive compression method results in a long compression time and low compression efficiency.

[0080] To alleviate the above problems, this disclosure provides a neural network optimization method, apparatus, electronic device, storage medium, and chip.

[0081] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0082] To facilitate understanding of the embodiments of this disclosure, a neural network optimization method disclosed in this disclosure will first be described in detail. The execution subject of the neural network optimization method provided in this disclosure is generally a computer device with certain computing capabilities, such as a terminal device or a server. In some possible implementations, the neural network optimization method can be implemented by a processor calling computer-readable instructions stored in memory.

[0083] See Figure 1 The diagram shown is a flowchart of a neural network optimization method provided in this embodiment of the present disclosure. The method includes: S101-S104, wherein:

[0084] S101, Obtain the neural network to be compressed; wherein, the neural network to be compressed includes multiple network blocks to be compressed.

[0085] S102, in response to performing a compression operation on multiple network blocks to be compressed to obtain multiple compressed network blocks, a target error for each compressed network block is determined; wherein, the target error is used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; and / or, the degree of difference between the network block to be compressed and the compressed network block.

[0086] S103, according to the processing order determined based on the target error, adjust the parameters of each compressed network block to obtain the target network block.

[0087] S104, train the neural network to be compressed, which includes multiple target network blocks, to obtain the target neural network.

[0088] In the above method, a target error is determined for each compressed network block. This target error characterizes the degree of difference between the network block to be compressed and the compressed network block; and / or, characterizes the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block. That is, by using the target error of each compressed network block, the influence of each compressed network block on the neural network and / or the influence of the compression process on the network block to be compressed can be determined. Furthermore, based on this target error, the processing order of each network block to be compressed can be determined more accurately; for example, the network blocks to be compressed with a greater influence can be processed first, and the network blocks to be compressed with a smaller influence can be processed later.

[0089] By considering the impact of different network blocks on the inference results of the compressed neural network, a more accurate target network block can be obtained when adjusting the parameters of each compressed network block according to this processing order. Therefore, training the compressed neural network, which includes multiple target network blocks, yields a target neural network with better performance. Simultaneously, compression of the compressed neural network can be performed more quickly while maintaining its performance, thus improving the optimization efficiency of the compressed neural network.

[0090] The following provides a detailed explanation of S101-S104.

[0091] Regarding S101:

[0092] Here, the neural network to be compressed can be a pre-trained, converged neural network. The network structure of this neural network can be configured according to requirements, which will not be detailed here. The neural network to be compressed includes multiple network blocks, each including at least one feature processing layer. Feature processing layers can be, for example, convolutional feature processing layers, pooling feature processing layers, activation feature processing layers, etc.

[0093] In implementation, the purpose of the neural network to be compressed can be set as needed. For example, the neural network to be compressed may include a neural network for face recognition, a neural network for image semantic segmentation, a neural network for vehicle tracking, and so on.

[0094] In one possible implementation, obtaining the neural network to be compressed may include:

[0095] Step A1: Obtain the original trained neural network.

[0096] Step A2: Based on the set determination factors, determine multiple network blocks to be compressed according to at least one feature processing layer included in the original neural network.

[0097] Step A3: The original neural network, which includes multiple network blocks to be compressed, is identified as the neural network to be compressed.

[0098] In implementation, the trained original neural network, which includes multiple feature processing layers, can be obtained first. Then, based on the set deterministic factors and at least one feature processing layer in the original neural network, multiple network blocks to be compressed are determined. For example, a single feature processing layer or multiple adjacent feature processing layers can be used to form a network block to be compressed. The original neural network containing these multiple network blocks to be compressed is then identified as the neural network to be compressed.

[0099] Here, after obtaining the trained original neural network, multiple network blocks to be compressed can be accurately determined based on at least one feature processing layer included in the original neural network, according to the set deterministic factors; so that the compression operation of the network blocks to be compressed can be performed subsequently to optimize the neural network to be compressed.

[0100] In one possible implementation, the determining factors include at least one of the following:

[0101] Factor 1: The amount of data in the parameters included in the feature processing layer.

[0102] Factor 2: The size of the input feature data corresponding to the feature processing layer.

[0103] Factor 3: The size of the output feature data corresponding to the feature processing layer.

[0104] Factor 4: Threshold for the number of feature processing layers included in the network block to be compressed.

[0105] In implementation, after obtaining the trained original neural network, multiple network blocks to be compressed can be determined based on at least one feature processing layer included in the original neural network, according to the set deterministic factors. These deterministic factors can be set according to the requirements of the neural network; this is merely an illustrative example.

[0106] For example, when factor one is identified, the amount of data for the parameters in each feature processing layer of the original neural network can be determined. If the amount of data corresponding to a feature processing layer is greater than a set parameter threshold, that feature processing layer can be identified as needing compression. Therefore, this feature processing layer can be used to construct a network block to be compressed. That is, the feature processing layer can be used as the network block to be compressed, or the feature processing layer and other adjacent feature processing layers can be used to construct the network block to be compressed.

[0107] For example, when factor two is included, multiple adjacent feature processing layers with the same size of the corresponding input feature data can be combined to obtain the network block to be compressed. Similarly, when factor three is included, multiple adjacent feature processing layers with the same size of the corresponding output feature data can be combined to obtain the network block to be compressed.

[0108] For example, when determining factors including factor four, when determining the network block to be compressed, the number of feature processing layers included in the network block to be compressed can be controlled to be less than a certain threshold. Alternatively, the number of convolutional feature processing layers included in the network block to be compressed can also be controlled to be less than a certain threshold.

[0109] The number of factors can be multiple. For example, when the factors include factor two and factor three, multiple adjacent feature processing layers with the same size of corresponding input feature data and the same size of corresponding output feature data can be combined to obtain the network block to be compressed.

[0110] For example, when the factors include factor two, factor three, and factor four, multiple adjacent feature processing layers with the same size of corresponding input feature data, the same size of corresponding output feature data, and a number of feature processing layers less than the quantity threshold can be combined to obtain the network block to be compressed.

[0111] Here, at least one determinant is set so that the network block to be compressed can be flexibly determined using the determinant.

[0112] Regarding S102:

[0113] After obtaining the neural network to be compressed, compression operations are performed on the multiple network blocks comprising the neural network to be compressed, resulting in multiple compressed network blocks. These compression operations may include pruning, sparsity reduction, quantization, etc. See also... Figure 2 The diagrams shown illustrate the various compression operations, in which... Figure 2 This includes: example data of feature data before and after pruning operation shown in a, example data of feature data before and after sparsification operation shown in b, and example data of feature data before and after quantization operation shown in c.

[0114] During implementation, the compression operations used for each network block to be compressed can be the same or different. Each network block can have one or more compression operations. For example, it can use only pruning operations, or only quantization operations, or pruning and sparsification operations, or even a combination of pruning, sparsification, and quantization operations. The compression operation used for each network block can be selected based on actual needs.

[0115] Since compression is a lossy compression operation, compressing the network block to be compressed will affect its performance, and consequently, the performance of the neural network to be compressed. Therefore, after obtaining multiple compressed network blocks, a target error can be determined for each compressed network block. The target error can include: a compression error characterizing the degree of difference between the network block to be compressed and the compressed network block, and / or an inference error characterizing the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block. The degree of difference between the network block to be compressed and the compressed network block can include: the degree of difference between the weight data of the network block to be compressed and the weight data of the network block to be compressed, and / or the degree of difference between the output feature data of the network block to be compressed and the output feature data of the network block to be compressed.

[0116] In practice, after the input feature data is determined, the target error corresponding to the compressed network block can be determined based on the output feature data of the compressed network block and the output feature data of the corresponding network block to be compressed. Alternatively, the target error corresponding to the compressed network block can be determined based on the detection results of the neural network including the compressed network block and the detection results of the neural network including the network block to be compressed.

[0117] In one possible implementation, the target error includes a compression error characterizing the degree of difference between the network block to be compressed and the compressed network block; see also Figure 3 As shown, determining the target error for each compressed network block can include:

[0118] S301, Obtain the first sample data.

[0119] S302, using the network block to be compressed, perform feature extraction on the first sample data to obtain the first output feature data corresponding to the first sample data.

[0120] S303: Using the compressed network block, feature extraction is performed on the first sample data to obtain the second output feature data corresponding to the first sample data.

[0121] S304, using a first data pair consisting of the first output feature data and the second output feature data, and / or a second data pair consisting of the weight data of the network block to be compressed and the weight data of the compressed network block, to determine the compression error of the compressed network block.

[0122] The first sample data can be the feature data output by the feature processing layer located before the network block to be compressed in the neural network. Alternatively, it can be the acquired video data, image data, point cloud data, etc.

[0123] After obtaining the first sample data, features are extracted from the first sample data using the network block to be compressed, resulting in the first output feature data corresponding to the first sample data. Then, features are extracted from the first sample data using the compressed network block corresponding to the network block to be compressed, resulting in the second output feature data corresponding to the first sample data.

[0124] In practice, the root mean squared error (RMSE) method, the Kullback-Leibler divergence (KLD) method, etc., can be used to determine the compression error of the compressed network block based on the first data pair consisting of the first output feature data and the second output feature data, and / or the second data pair consisting of the weight data of the network block to be compressed and the weight data of the compressed network block.

[0125] For example, the RMSE method can be used to perform calculations on the first data pair consisting of the first output feature data and the second output feature data to obtain the first calculation result; the first calculation result is determined as the compression error corresponding to the compressed network block.

[0126] Alternatively, the RMSE method can be used to calculate a first intermediate result on the first data pair consisting of the first output feature data and the second output feature data; and the RMSE method can be used to calculate a second intermediate result on the second data pair consisting of the weight data of the network block to be compressed and the weight data of the compressed network block; then the first intermediate result and the second intermediate result can be fused to obtain a second calculation result; the second calculation result is determined as the compression error corresponding to the compressed network block. For example, the fusion method may include: determining the average value, weighted summation, etc.

[0127] Alternatively, the first data pair consisting of the first output feature data and the second output feature data can be calculated using the KL divergence method to obtain a first intermediate result; and the second data pair consisting of the weight data of the network block to be compressed and the weight data of the compressed network block can be calculated using the KL divergence method to obtain a second intermediate result; then the first intermediate result and the second intermediate result can be fused to obtain a third calculation result; and the third calculation result can be determined as the compression error corresponding to the compressed network block.

[0128] Here, features are extracted from the first sample data using the network block to be compressed and the compressed network block, respectively, to obtain first output feature data and second output feature data. Then, using the first data pair composed of the first output feature data and the second output feature data, and / or the second data pair composed of the weight data of the network block to be compressed and the weight data of the compressed network block, the compression error of the compressed network block can be determined more accurately, providing data support for subsequent determination of the processing order based on the target error.

[0129] In one possible implementation, the target error includes an inference error characterizing the degree of change in the inference results of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; see also Figure 4 As shown, determining the target error for each compressed network block can include:

[0130] S401, Obtain the second sample data.

[0131] S402, the second sample data is input into the neural network to be compressed to obtain the first detection result corresponding to the second sample data.

[0132] S403, replace the network block to be compressed in the neural network to be compressed with the corresponding compressed network block to obtain the replaced neural network to be compressed; input the second sample data into the replaced neural network to be compressed to obtain the second detection result corresponding to the second sample data.

[0133] S404. Using the first and second detection results, determine the inference error of the compressed network block.

[0134] The second sample data can include image data, video data, point cloud data, etc., input to the neural network to be compressed. After obtaining the second sample data, it can be input into the neural network to obtain the first detection result corresponding to the second sample data. Additionally, the network block to be compressed can be replaced with a compressed network block in the neural network to obtain a replaced neural network; the second sample data is then input into the replaced neural network to obtain the second detection result corresponding to the second sample data. The first and second detection results are then used to determine the inference error of the compressed network block.

[0135] For example, if the neural network to be compressed is a classification network, the first detection result and the second detection result may include the prediction accuracy; therefore, the first prediction accuracy indicated by the first detection result and the second prediction accuracy indicated by the second detection result can be subtracted to obtain the inference error of the compressed network block.

[0136] If the neural network to be compressed is a detection network, the first and second detection results can include the detection box information of the object to be detected. Therefore, the overlap rate between the first detection box indicated by the first detection result and the second detection box indicated by the second detection result can be determined. For example, this overlap rate can be the Intersection over Union (IOU), and then this overlap rate can be determined as the inference error corresponding to the compressed network block. Alternatively, the deviation between the vertex information of the four vertices of the first detection box indicated by the first detection result and the vertex information of the four vertices of the second detection box indicated by the second detection result can be determined, and this deviation value can be determined as the inference error corresponding to the compressed network block.

[0137] There are multiple ways to determine inference error. The specific method can be determined based on the type of neural network, detection results, etc. This is just an example.

[0138] Here, the second sample data is input into the neural network to be compressed to obtain the first detection result corresponding to the second sample data; and the second sample data is input into the replaced neural network to be compressed to obtain the second detection result corresponding to the second sample data. Using the first and second detection results, the inference error of the compressed network block can be determined more accurately, providing data support for subsequent determination of the processing order based on the target error.

[0139] Regarding S103:

[0140] After determining the target error for each compressed network block, the processing order can be determined based on the target errors of each compressed network block. For example, the processing order of the compressed network blocks can be determined according to the target errors from largest to smallest. When the target error includes inference error and compression error, the inference error and compression error can be summed, averaged, or weighted summed, and then the resulting sum can be used to determine the processing order. Then, according to the determined processing order, the parameters of each compressed network block can be adjusted to obtain the target network block.

[0141] In one possible implementation, the parameters of each compressed network block are adjusted according to the processing order determined based on the target error to obtain the target network block, including:

[0142] According to the processing order of target error from largest to smallest, the first network block to be processed is determined from multiple compressed network blocks; and the first network block to be processed is determined as the current network block to be processed, and the neural network to be compressed is determined as the current neural network;

[0143] Replace the network block to be compressed corresponding to the current network block to be processed in the current neural network with the current network block to be processed to obtain the intermediate neural network;

[0144] Based on the intermediate neural network, the parameters of the current network block to be processed are adjusted to obtain the target network block corresponding to the current network block to be processed, and the intermediate neural network including the target network block.

[0145] Based on the processing order, the next network block to be processed is determined from multiple compressed network blocks, and the next network block to be processed is determined as the current network block to be processed. The intermediate neural network including the target network block is determined as the current neural network. The process returns to the step of replacing the network block to be compressed corresponding to the current network block to be processed in the current neural network with the current network block to be processed, and obtaining the intermediate neural network, until the target network block corresponding to each compressed network block is obtained.

[0146] For example, if the compressed network block includes compressed network block one, compressed network block two, and compressed network block three, and the processing order is: compressed network block two corresponding to compressed network block two, compressed network block one corresponding to compressed network block one, and compressed network block three corresponding to compressed network block three, then the parameter adjustment process for each compressed network block can include:

[0147] Step 1: Replace the second network block to be compressed in the neural network to be compressed with the second network block after compression to obtain the intermediate neural network 1; Based on the intermediate neural network 1, adjust the parameters of the second network block after compression to obtain the target network block 2 corresponding to the second network block after compression, and the intermediate neural network 1 including the target network block 2.

[0148] Step 2: Replace the first network block to be compressed in the intermediate neural network 1, which includes the target network block 2, with the compressed network block 1 to obtain the intermediate neural network 2; based on the intermediate neural network 2, adjust the parameters of the compressed network block 1 to obtain the target network block 1 corresponding to the compressed network block 1 and the intermediate neural network 2 including the target network block 1.

[0149] Step 3: Replace the third network block to be compressed in the intermediate neural network 2, which includes the target network block 1, with the compressed network block 3 to obtain the intermediate neural network 3; based on the intermediate neural network 3, adjust the parameters of the compressed network block 3 to obtain the target network block 3 corresponding to the compressed network block 3, and the intermediate neural network 3 including the target network block 3.

[0150] Furthermore, the intermediate neural network 3 (i.e., the neural network to be compressed, which includes multiple target network blocks) can be trained by fine-tuning the parameters of each feature processing layer included in the intermediate neural network 3 to obtain the target neural network.

[0151] In this embodiment of the disclosure, by processing the compressed network blocks with larger target errors in descending order, first adjusting the parameters of the compressed network blocks with larger target errors, and then adjusting the parameters of the compressed network blocks with smaller target errors, the target network blocks corresponding to each compressed network block can be obtained more accurately.

[0152] Another possible implementation involves adjusting the parameters of each compressed network block according to the processing order determined based on the target error to obtain the target network block, which may include:

[0153] Step B1: According to the processing order of target error from largest to smallest, each compressed network block in the multiple compressed network blocks is taken as the network block to be processed.

[0154] Step B2: Replace the network block to be compressed with the network block to be processed in the neural network to be compressed, and obtain the intermediate neural network.

[0155] Step B3: Based on the intermediate neural network, adjust the parameters of the network block to be processed to obtain the target network block.

[0156] For example, if the compressed network block includes compressed network block one, compressed network block two, and compressed network block three, and the processing order is: compressed network block two corresponding to compressed network block two, compressed network block one corresponding to compressed network block one, and compressed network block three corresponding to compressed network block three, then the parameter adjustment process for each compressed network block can include:

[0157] Step 1: Replace the second network block to be compressed in the neural network to be compressed with the second compressed network block to obtain the intermediate neural network 1; based on the intermediate neural network 1, adjust the parameters of the second compressed network block to obtain the target network block 2 corresponding to the second compressed network block.

[0158] Step 2: Replace the first network block to be compressed in the neural network to be compressed with the first compressed network block to obtain the intermediate neural network 2; based on the intermediate neural network 2, adjust the parameters of the first compressed network block to obtain the target network block 1 corresponding to the first compressed network block.

[0159] Step 3: Replace the third network block to be compressed in the neural network to be compressed with the third network block after compression to obtain the intermediate neural network 3; based on the intermediate neural network 3, adjust the parameters of the third network block after compression to obtain the target network block 3 corresponding to the third network block after compression.

[0160] Considering that different network blocks have different degrees of influence on the inference results of the neural network and different robustness to compression operations, in order to more accurately determine the target network block, the parameters of each compressed network block can be adjusted in descending order of target error.

[0161] In one possible implementation, step B3, adjusting the parameters of the network block to be processed based on the intermediate neural network to obtain the target network block, may include:

[0162] Step B31: Obtain the third sample data.

[0163] Step B32: Input the third sample data into the intermediate neural network to obtain the intermediate detection result corresponding to the third sample data.

[0164] Step B33: Based on the intermediate detection results, adjust the parameters of the network blocks to be processed included in the intermediate neural network to obtain the intermediate network blocks.

[0165] Step B34: The intermediate neural network including the intermediate network block is used as the updated intermediate neural network, and the process is returned to the step of obtaining the third sample data until the training cutoff condition is met. The training cutoff condition includes: the number of parameter adjustments is equal to the number threshold corresponding to the network block to be processed, and / or, until the target error of the intermediate network block is less than the error threshold.

[0166] Step B35: The intermediate network block obtained after the last parameter adjustment is determined as the target network block.

[0167] For each network block to be processed, the intermediate neural network including the network block to be processed can be trained once or multiple times. During each training, only the parameters of the network block to be processed are adjusted. After one or more adjustments, the target network block corresponding to the network block to be processed is obtained.

[0168] In implementation, a third sample data point can be acquired first, which serves as the input data for the neural network to be compressed. This third sample data can be the same as or different from the second sample data. The third sample data is then input into an intermediate neural network to obtain intermediate detection results. Based on these intermediate detection results, the parameters of the network blocks to be processed within the intermediate neural network are adjusted to obtain the intermediate network blocks.

[0169] If the target error corresponding to the intermediate network block is greater than or equal to the error threshold, and / or the number of times the parameters of the network block to be processed are adjusted is less than the number threshold corresponding to the network block to be processed, the intermediate neural network including the intermediate network block will be used as the updated intermediate neural network and the process will return to step B31.

[0170] If the target error corresponding to the intermediate network block is less than the error threshold, and / or the number of times the parameters of the network block to be processed are adjusted is equal to the number of times the network block to be processed is adjusted, the intermediate network block obtained after the last parameter adjustment is determined as the target network block.

[0171] Here, by setting a threshold for the number of iterations and / or an error threshold, the cutoff conditions for adjusting the parameters of the network block to be processed can be flexibly determined, thereby enabling the desired target network block to be obtained flexibly.

[0172] In one possible implementation, the number of times threshold corresponding to the network block to be processed is determined according to the following steps: based on the target error corresponding to the network block to be processed and the mapping relationship between the error and the number of times, the number of times threshold corresponding to the network block to be processed is determined; wherein, the target error and the number of times threshold are positively correlated.

[0173] Considering that the larger the target error of the network block to be processed, the more times the network block needs to be adjusted in terms of parameters, the threshold number of times the network block to be processed can be determined based on the target error of the network block to be processed and the mapping relationship between error and number of times.

[0174] For example, the mapping relationship between error and number of attempts can be shown in Table 1 below. Then, by combining Table 1 and the target error corresponding to the network block to be processed, the number of attempts threshold for the network block to be processed can be determined. For example, if the target error of network block 1 to be processed is 0.75, then the number of attempts threshold for network block 1 to be processed is determined to be 8.

[0175] Table 1. Mapping relationship between error and number of attempts.

[0176] Error x 1>＝x>0.9 0.9>＝x>0.8 0.8>＝x>0.7 …… 0.1>＝x>0 Number of times threshold y 10 9 8 …… 1

[0177] For example, you can also set the relationship between error and number of times, such as y = kx, y = kx 2 The expression is defined as follows: y = 5x; where y is the frequency threshold, x is the error, and k is the correlation coefficient, with k being a positive number. The specific value of k can be determined based on the actual situation. For example, when the mapping relationship is y = 5x, the target error corresponding to the network block to be processed can be substituted into the relational expression y = 5x to obtain the calculation result; then, based on the calculation result, the frequency threshold corresponding to the network block to be processed can be determined. For example, if the calculation result is an integer, then the calculation result is determined as the frequency threshold; if the calculation result is not an integer, then the calculation result can be rounded down, and the rounded result is determined as the frequency threshold. The rounding operation can be either rounding up or rounding down.

[0178] Here, based on the target error corresponding to the network block to be processed and the mapping relationship between error and number of times, the number of times threshold corresponding to each network block to be processed can be flexibly determined. Different network blocks to be processed can correspond to different number of times thresholds, which improves the efficiency and flexibility of adjusting the parameters of the network block to be processed.

[0179] Regarding S104:

[0180] In practice, after obtaining the target network blocks corresponding to multiple compressed network blocks, each network block to be compressed can be replaced with its corresponding target network block to obtain a neural network to be compressed that includes multiple target network blocks. Then, the neural network to be compressed that includes multiple target network blocks is trained, and the parameters of at least one feature processing layer in the neural network to be compressed that includes multiple target network blocks are adjusted to obtain the target neural network.

[0181] For example, training samples can be obtained and input into a neural network to be compressed, which includes multiple target network blocks, to obtain detection results. Based on the detection results and the annotation results corresponding to the training samples, the parameters of the neural network to be compressed, which includes multiple target network blocks, can be fine-tuned to obtain the target neural network.

[0182] In practice, the target neural network can be applied to face recognition scenarios. The application process of the target neural network can include: acquiring the image to be detected; using the target neural network to detect the image to be detected, and obtaining the face detection result corresponding to the image to be detected.

[0183] The image to be detected can be an image acquired by an image acquisition device. The image to be detected is input into the target neural network to perform detection on the image and obtain the face detection result corresponding to the image to be detected. For example, the face detection result can be face identification information (such as name, employee number, etc.) and face matching information (such as match or non-match).

[0184] In specific implementation, the target neural network can be applied to road recognition scenarios. The application process of the target neural network can include: acquiring road images collected by the driving device during driving; using the target neural network to perform target detection on the road images to obtain the object detection results corresponding to the road images; and controlling the driving state of the driving device based on the object detection results corresponding to the road images.

[0185] For example, the driving device can be an autonomous vehicle, a vehicle equipped with an Advanced Driving Assistance System (ADAS), or a robot. The road images can be image data collected in real time by the driving device during its operation.

[0186] By utilizing the generated target neural network to detect objects in road images, corresponding object detection results are generated. For example, the object detection results can include the location and orientation information of each target object in the road image. The target object can be any object to be detected, such as motor vehicles, non-motorized vehicles, pedestrians, animals, road signs, etc. Furthermore, the driving state of the driving device can be controlled based on the object detection results corresponding to the road image.

[0187] When controlling the driving device, the system can control its acceleration, deceleration, stopping, steering, braking, and obstacle avoidance. For example, obstacle avoidance may include bypassing an object or changing the driving route. Alternatively, it can play voice prompts to remind the driver of the driving status of the driving device.

[0188] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0189] Based on the same concept, this disclosure also provides an artificial intelligence (AI) chip, see [link to relevant documentation]. Figure 5 The diagram shown is an architectural schematic of an artificial intelligence (AI) chip provided in this embodiment of the present disclosure, including a processor 501 and a memory 502. Specifically:

[0190] The memory 502 is used to store the network parameters of the target neural network; the target neural network is generated based on the neural network optimization method described in the above embodiments.

[0191] Processor 501 is used to run the target neural network based on network parameters read from memory.

[0192] After obtaining the target neural network, its network parameters can be stored in memory. The processor reads the network parameters from memory and runs the target neural network based on these parameters. The process of obtaining the target neural network can be referred to the detailed descriptions of S101 to S104 above, and will not be repeated here.

[0193] Here, by utilizing the neural network optimization method described in the first aspect or any embodiment, the neural network to be compressed is optimized to generate the target neural network. While ensuring the accuracy of the neural network, the number of network parameters is reduced, resulting in a smaller number of network parameters in the target neural network. Storing the network parameters of the target neural network in memory reduces storage resource consumption. Simultaneously, when the processor runs the target neural network, the smaller number of parameters used improves the processor's running efficiency.

[0194] Based on the same concept, this disclosure also provides a neural network optimization device, see [link to relevant documentation]. Figure 6 The diagram shown is a schematic representation of the architecture of a neural network optimization device provided in this embodiment of the present disclosure, including an acquisition module 601, a first determination module 602, an adjustment module 603, and a training module 604. Specifically:

[0195] The acquisition module 601 is used to acquire the neural network to be compressed; wherein the neural network to be compressed includes multiple network blocks to be compressed;

[0196] The first determining module 602 is configured to determine a target error for each compressed network block in response to performing a compression operation on the plurality of network blocks to be compressed to obtain a plurality of compressed network blocks; wherein the target error is used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; and / or the degree of difference between the network block to be compressed and the compressed network block.

[0197] The adjustment module 603 is used to adjust the parameters of each compressed network block according to the processing order determined based on the target error, so as to obtain the target network block;

[0198] Training module 604 is used to train a neural network to be compressed, which includes multiple target network blocks, to obtain the target neural network.

[0199] In one possible implementation, the target error includes a compression error characterizing the degree of difference between the network block to be compressed and the compressed network block; the first determining module 602, when determining the target error for each compressed network block, is used to:

[0200] Obtain the first sample data;

[0201] Using the network block to be compressed, feature extraction is performed on the first sample data to obtain the first output feature data corresponding to the first sample data; and

[0202] Using the compressed network block, feature extraction is performed on the first sample data to obtain the second output feature data corresponding to the first sample data;

[0203] The compression error of the compressed network block is determined using a first data pair consisting of the first output feature data and the second output feature data, and / or a second data pair consisting of the weight data of the network block to be compressed and the weight data of the compressed network block.

[0204] In one possible implementation, the target error includes an inference error characterizing the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; the first determining module 602, when determining the target error for each compressed network block, is used to:

[0205] Obtain the second sample data;

[0206] The second sample data is input into the neural network to be compressed to obtain the first detection result corresponding to the second sample data; and

[0207] In the neural network to be compressed, the network block to be compressed is replaced with the corresponding compressed network block to obtain the replaced neural network to be compressed; the second sample data is input into the replaced neural network to be compressed to obtain the second detection result corresponding to the second sample data;

[0208] The inference error of the compressed network block is determined using the first detection result and the second detection result.

[0209] In one possible implementation, the adjustment module 603, when adjusting the parameters of each compressed network block according to the processing order determined based on the target error to obtain the target network block, is used to:

[0210] According to the processing order of the target error from largest to smallest, the first network block to be processed is determined from the multiple compressed network blocks; and the first network block to be processed is determined as the current network block to be processed, and the neural network to be compressed is determined as the current neural network;

[0211] In the current neural network, the network block to be compressed corresponding to the current network block to be processed is replaced with the current network block to be processed to obtain an intermediate neural network;

[0212] Based on the intermediate neural network, the parameters of the current network block to be processed are adjusted to obtain the target network block corresponding to the current network block to be processed and the intermediate neural network including the target network block;

[0213] Based on the processing order, the next network block to be processed is determined from the multiple compressed network blocks, and the next network block to be processed is determined as the current network block to be processed. The intermediate neural network including the target network block is determined as the current neural network. The process returns to the step of replacing the network block to be compressed corresponding to the current network block to be processed in the current neural network with the current network block to be processed to obtain the intermediate neural network, until the target network block corresponding to each compressed network block is obtained.

[0214] In one possible implementation, the adjustment module 603, when adjusting the parameters of each compressed network block according to the processing order determined based on the target error to obtain the target network block, is used to:

[0215] According to the processing order of the target error from largest to smallest, each of the multiple compressed network blocks is taken as a network block to be processed.

[0216] In the neural network to be compressed, the network block to be processed corresponding to the network block to be compressed is replaced with the network block to be processed to obtain an intermediate neural network;

[0217] Based on the intermediate neural network, the parameters of the network block to be processed are adjusted to obtain the target network block.

[0218] In one possible implementation, the adjustment module 603, when adjusting the parameters of the network block to be processed based on the intermediate neural network to obtain the target network block, is used to:

[0219] Obtain third sample data;

[0220] The third sample data is input into the intermediate neural network to obtain the intermediate detection result corresponding to the third sample data;

[0221] Based on the intermediate detection results, the parameters of the network block to be processed included in the intermediate neural network are adjusted to obtain the intermediate network block;

[0222] The intermediate neural network including the intermediate network block is used as the updated intermediate neural network, and the process is returned to the step of obtaining the third sample data until the training cutoff condition is met. The training cutoff condition includes: the number of parameter adjustments is equal to the number threshold corresponding to the network block to be processed, and / or, until the target error of the intermediate network block is less than the error threshold.

[0223] The intermediate network block obtained after the last parameter adjustment is determined as the target network block.

[0224] In one possible implementation, the apparatus further includes a second determining module 605, which is configured to determine a threshold number of iterations corresponding to the network block to be processed according to the following steps:

[0225] Based on the target error corresponding to the network block to be processed and the mapping relationship between the error and the number of times, the number of times threshold corresponding to the network block to be processed is determined, wherein the error and the number of times are positively correlated.

[0226] In one possible implementation, the acquisition module 601, when acquiring the neural network to be compressed, is used to:

[0227] Obtain the original trained neural network;

[0228] Based on the set determining factors, and using at least one feature processing layer included in the original neural network, multiple network blocks to be compressed are determined.

[0229] The original neural network, which includes the plurality of network blocks to be compressed, is identified as the neural network to be compressed.

[0230] In one possible implementation, the determining factors include at least one of the following:

[0231] The amount of data for the parameters included in the feature processing layer;

[0232] The size of the input feature data corresponding to the feature processing layer;

[0233] The size of the output feature data corresponding to the feature processing layer;

[0234] The threshold number of feature processing layers included in the network block to be compressed.

[0235] In some embodiments, the functions or templates of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0236] Based on the same technical concept, embodiments of this disclosure also provide an electronic device. (Refer to...) Figure 7 The diagram shows the structure of an electronic device provided in this embodiment, including a processor 701, a memory 702, and a bus 703. The memory 702 stores execution instructions and includes a main memory 7021 and an external memory 7022. The main memory 7021, also called internal memory, is used to temporarily store computational data in the processor 701 and data exchanged with external memory 7022 such as a hard disk. The processor 701 exchanges data with the external memory 7022 through the main memory 7021. When the electronic device 700 is running, the processor 701 and the memory 702 communicate through the bus 703, causing the processor 701 to execute the following instructions:

[0237] Obtain a neural network to be compressed; wherein the neural network to be compressed includes multiple network blocks to be compressed;

[0238] In response to performing a compression operation on the plurality of network blocks to be compressed to obtain a plurality of compressed network blocks, a target error is determined for each of the compressed network blocks; wherein the target error is used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; and / or, the degree of difference between the network block to be compressed and the compressed network block;

[0239] Following the processing order determined based on the target error, the parameters of each compressed network block are adjusted to obtain the target network block;

[0240] The target neural network is obtained by training a neural network to be compressed, which includes multiple target network blocks.

[0241] The specific processing flow of processor 701 can be referred to the description in the above method embodiments, and will not be repeated here. Alternatively,

[0242] The electronic device provided in this disclosure may further include the chip provided in this disclosure.

[0243] Furthermore, this disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the neural network optimization method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0244] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the neural network optimization method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0245] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0246] This disclosure relates to the field of augmented reality (AR). It involves acquiring image information of target objects in a real-world environment and then using various visual algorithms to detect or identify the relevant features, states, and attributes of these objects, thereby achieving an AR effect that combines virtual and real elements to suit specific applications. For example, target objects may include human features such as faces, limbs, gestures, and movements; objects such as signs and markers; or venues such as sand tables, display areas, or displayed items. Visual algorithms may include visual localization, SLAM, 3D reconstruction, image registration, background segmentation, keypoint extraction and tracking of objects, and pose or depth detection. Specific applications can include interactive scenarios related to real-world scenes or objects, such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays, as well as human-related special effects processing, such as makeup enhancement, limb enhancement, special effects displays, and virtual model displays. Convolutional neural networks (CNNs) can be used to detect or identify the relevant features, states, and attributes of target objects. The aforementioned CNNs are network models trained using deep learning frameworks.

[0247] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0248] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0249] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0250] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0251] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A neural network optimization method, characterized in that, include: Obtain a neural network to be compressed; wherein the neural network to be compressed includes multiple network blocks to be compressed; In response to performing a compression operation on the plurality of network blocks to be compressed to obtain a plurality of compressed network blocks, a target error is determined for each of the compressed network blocks; wherein the target error is used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; and / or, the degree of difference between the network block to be compressed and the compressed network block; Following the processing order determined based on the target error, the parameters of each compressed network block are adjusted to obtain the target network block; A target neural network is obtained by training a neural network to be compressed, which includes multiple target network blocks, and the target neural network is stored. The target neural network is applied to a face recognition scenario or a road recognition scenario.

2. The method according to claim 1, characterized in that, The target error includes a compression error used to characterize the degree of difference between the network block to be compressed and the compressed network block; Determining the target error for each of the compressed network blocks includes: Obtain the first sample data; Using the network block to be compressed, feature extraction is performed on the first sample data to obtain the first output feature data corresponding to the first sample data; and Using the compressed network block, feature extraction is performed on the first sample data to obtain the second output feature data corresponding to the first sample data; The compression error of the compressed network block is determined using a first data pair consisting of the first output feature data and the second output feature data, and / or a second data pair consisting of the weight data of the network block to be compressed and the weight data of the compressed network block.

3. The method according to claim 1, characterized in that, The target error includes an inference error used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; Determining the target error for each of the compressed network blocks includes: Obtain the second sample data; The second sample data is input into the neural network to be compressed to obtain the first detection result corresponding to the second sample data; and In the neural network to be compressed, the network block to be compressed is replaced with the corresponding compressed network block to obtain the replaced neural network to be compressed; the second sample data is input into the replaced neural network to be compressed to obtain the second detection result corresponding to the second sample data; The inference error of the compressed network block is determined using the first detection result and the second detection result.

4. The method according to any one of claims 1 to 3, characterized in that, The step of adjusting the parameters of each compressed network block according to the processing order determined based on the target error to obtain the target network block includes: According to the processing order of the target error from largest to smallest, the first network block to be processed is determined from the multiple compressed network blocks; and the first network block to be processed is determined as the current network block to be processed, and the neural network to be compressed is determined as the current neural network; In the current neural network, the network block to be compressed corresponding to the current network block to be processed is replaced with the current network block to be processed to obtain an intermediate neural network; Based on the intermediate neural network, the parameters of the current network block to be processed are adjusted to obtain the target network block corresponding to the current network block to be processed and the intermediate neural network including the target network block; Based on the processing order, the next network block to be processed is determined from the multiple compressed network blocks, and the next network block to be processed is determined as the current network block to be processed. The intermediate neural network including the target network block is determined as the current neural network. The process returns to the step of replacing the network block to be compressed corresponding to the current network block to be processed in the current neural network with the current network block to be processed to obtain the intermediate neural network, until the target network block corresponding to each compressed network block is obtained.

5. The method according to any one of claims 1 to 3, characterized in that, The step of adjusting the parameters of each compressed network block according to the processing order determined based on the target error to obtain the target network block includes: According to the processing order of the target error from largest to smallest, each of the multiple compressed network blocks is taken as a network block to be processed. In the neural network to be compressed, the network block to be processed corresponding to the network block to be compressed is replaced with the network block to be processed to obtain an intermediate neural network; Based on the intermediate neural network, the parameters of the network block to be processed are adjusted to obtain the target network block.

6. The method according to claim 5, characterized in that, The step of adjusting the parameters of the network block to be processed based on the intermediate neural network to obtain the target network block includes: Obtain third sample data; The third sample data is input into the intermediate neural network to obtain the intermediate detection result corresponding to the third sample data; Based on the intermediate detection results, the parameters of the network block to be processed included in the intermediate neural network are adjusted to obtain the intermediate network block; The intermediate neural network including the intermediate network block is used as the updated intermediate neural network, and the process is returned to the step of obtaining the third sample data until the training cutoff condition is met. The training cutoff condition includes: the number of parameter adjustments is equal to the number threshold corresponding to the network block to be processed, and / or, until the target error of the intermediate network block is less than the error threshold. The intermediate network block obtained after the last parameter adjustment is determined as the target network block.

7. The method according to claim 6, characterized in that, The threshold number of iterations corresponding to the network block to be processed is determined according to the following steps: Based on the target error corresponding to the network block to be processed and the mapping relationship between the error and the number of times, the number of times threshold corresponding to the network block to be processed is determined, wherein the target error and the number of times threshold are positively correlated.

8. The method according to any one of claims 1 to 3, characterized in that, The process of obtaining the neural network to be compressed includes: Obtain the original trained neural network; Based on the set determining factors, and using at least one feature processing layer included in the original neural network, multiple network blocks to be compressed are determined. The original neural network, which includes the plurality of network blocks to be compressed, is identified as the neural network to be compressed.

9. The method according to claim 8, characterized in that, The determining factors include at least one of the following: The amount of data for the parameters included in the feature processing layer; The size of the input feature data corresponding to the feature processing layer; The size of the output feature data corresponding to the feature processing layer; The threshold number of feature processing layers included in the network block to be compressed.

10. An artificial intelligence (AI) chip, characterized in that, The chip includes a processor and a memory; the memory is used to store the network parameters of the target neural network; the target neural network is generated based on the neural network optimization method according to any one of claims 1 to 9. The processor is configured to run the target neural network based on the network parameters read from the memory.

11. A neural network optimization device, characterized in that, include: An acquisition module is used to acquire a neural network to be compressed; wherein the neural network to be compressed includes multiple network blocks to be compressed; A first determining module is configured to determine a target error for each compressed network block in response to performing a compression operation on the plurality of network blocks to be compressed to obtain a plurality of compressed network blocks; wherein the target error is used to characterize the degree of change in the inference result of the neural network to be compressed after replacing the network block to be compressed with the compressed network block; and / or the degree of difference between the network block to be compressed and the compressed network block. An adjustment module is used to adjust the parameters of each compressed network block according to the processing order determined based on the target error, so as to obtain the target network block; The training module is used to train a neural network to be compressed, which includes multiple target network blocks, to obtain a target neural network, and to store the target neural network, wherein the target neural network is applied to a face recognition scenario or a road recognition scenario.

12. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the steps of the neural network optimization method as described in any one of claims 1 to 9; or includes a chip as described in claim 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the neural network optimization method as described in any one of claims 1 to 9.

Images