Weight dynamic optimization methods, devices and electronic equipment

Abstract

This application provides a method, apparatus, and electronic device for dynamic weight optimization. In this embodiment, during each training cycle of knowledge distillation, the weight parameters in the first neural network are first sparsified using the currently determined sparsity to obtain a second neural network. Then, knowledge distillation is performed using each training sample and the second neural network. This coupling of knowledge distillation and sparsification during training forms an end-to-end training framework for dynamic co-optimization of knowledge distillation and sparsification. This not only compensates for the accuracy loss caused by sparsification, minimizing accuracy loss while achieving model compression, but also avoids the performance degradation caused by training sparsification and knowledge distillation in stages.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to methods, apparatus and electronic devices for dynamic weight optimization. Background Technology

[0002] Neural networks, also known as artificial neural networks (ANNs), are mathematical models that mimic the behavioral characteristics of animal neural networks to perform distributed parallel information processing. Neural networks are typically deployed in electronic devices and can be used for image processing tasks such as image classification, object detection, image segmentation, and pose estimation. With the continuous development of artificial intelligence technology, the scale of neural networks is constantly increasing, leading to a significant increase in storage and computational overhead. However, due to the limited memory and computing power of electronic devices, researchers have proposed using model compression methods to compress neural network models and reduce their storage and computational costs.

[0003] Currently, in practical applications, sparsification methods such as weight sparsification are commonly used to compress neural network models, reducing storage and computational overhead. Weight sparsification refers to reducing the number of network parameters by setting some weights to zero, thus compressing the model. However, these sparsification methods alter the network structure, potentially leading to decreased model accuracy and impacting image processing accuracy. For example, in image classification, sparsification may prevent the effective extraction of crucial image features, resulting in lower classification accuracy. Therefore, minimizing accuracy loss while compressing neural networks using sparsification methods has become a pressing technical challenge. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus and electronic device for dynamic weight optimization, so as to minimize the loss of accuracy while compressing neural networks using sparsification methods.

[0005] This application provides a dynamic weight optimization method, which is applied to at least image processing scenarios, including image classification, object detection, image segmentation, and pose estimation scenarios. The method includes:

[0006] In the current training cycle, the sparsity used to sparsify the weight parameters of the current first neural network is determined; the sparsity determined in different training cycles is different, and if the current training cycle is the first training cycle, the first neural network is a pre-trained neural network; if the current training cycle is not the first training cycle, the first neural network is a network obtained by knowledge distillation of the pre-trained neural network in the previous training cycles.

[0007] Based on the sparsity, the weight parameters in the current first neural network are sparsified to obtain the second neural network; the network loss value is determined using each training sample and the second neural network.

[0008] Based on the gradient of each weight parameter in the first neural network according to the network loss value, the weight parameters in the first neural network are optimized to obtain the third neural network;

[0009] If the current training period is not the last training period, the third neural network is used as the first neural network in the next training period, and when the next training period is the current training period, the step of determining the sparsity of the weight parameters of the current first neural network is returned.

[0010] If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsified using a preset sparsity to obtain the target neural network. The target neural network is used to perform image processing based on the input image data to be processed in the image processing scenario.

[0011] This application embodiment also provides a weight dynamic optimization device, which is applied to at least image processing scenarios, including image classification scenarios, object detection scenarios, image segmentation scenarios, and pose estimation scenarios. The device includes:

[0012] The determination module is used to determine the sparsity of the weight parameters of the current first neural network in the current training cycle. The sparsity determined in different training cycles is different. If the current training cycle is the first training cycle, the first neural network is a pre-trained neural network. If the current training cycle is not the first training cycle, the first neural network is a network obtained by knowledge distillation of the pre-trained neural network in the previous training cycles.

[0013] A sparse module is used to sparsify the weight parameters in the current first neural network based on the sparsity to obtain a second neural network; and to determine the network loss value using each training sample and the second neural network.

[0014] An optimization module is used to optimize the weight parameters in the first neural network based on the gradient of each weight parameter in the first neural network, so as to obtain a third neural network.

[0015] The training module is used to, if the current training period is not the last training period, use the third neural network as the first neural network in the next training period, and return the step of determining the sparsity of the weight parameters of the current first neural network when the next training period is the current training period.

[0016] If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsified using a preset sparsity to obtain the target neural network. The target neural network is used to perform image processing based on the input image data to be processed in the image processing scenario.

[0017] This application also provides an electronic device, which includes:

[0018] Processor; and

[0019] A computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method described above.

[0020] This application also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the above method.

[0021] As can be seen from the above technical solutions, in the embodiments of this application, by first using the currently determined sparsity to sparsify the weight parameters in the current first neural network to obtain the second neural network in each training cycle of knowledge distillation, and then using each training sample and the second neural network to perform knowledge distillation, an end-to-end training framework for dynamic collaborative optimization of knowledge distillation and sparsification is formed by coupling knowledge distillation and sparsification during the training process. This not only makes up for the accuracy loss caused by sparsification, so as to minimize the accuracy loss while achieving model compression, but also avoids the performance degradation problem caused by training sparsification and knowledge distillation in stages.

[0022] Furthermore, in this embodiment, the neural network is not sparsified all at once according to the final sparsity, but iteratively sparsified gradually in each training cycle. By gradually increasing the difference in output features (i.e., knowledge difference) between the student network (i.e., the second neural network) and the teacher network (i.e., the pre-trained neural network) during knowledge distillation, the learning difficulty of the student network can be avoided due to excessively high initial sparsity leading to large differences in output features, thus effectively reducing the learning difficulty of the student network. In addition, this embodiment also optimizes the neural network before sparsification (i.e., the first neural network) by utilizing the network loss value. This avoids the problem of decreased neural network accuracy due to the non-participation of some sparsified weight parameters in the optimization, thereby enhancing the learning ability of the student network. The performance of the trained student network (i.e., the target neural network) is basically no different from that of the pre-trained neural network (i.e., the teacher network), thus achieving the goal of minimizing accuracy loss while compressing the neural network using sparsification methods. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this application, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of the method flow provided in the embodiments of this application.

[0026] Figure 3 This is a schematic diagram of another method flow provided for an embodiment of this application.

[0027] Figure 4 This is a schematic diagram illustrating the training implementation provided in an embodiment of this application.

[0028] Figure 5 This is a schematic diagram of the device structure provided in the embodiments of this application.

[0029] Figure 6 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0031] The system architecture used in the embodiments of this application will be described below:

[0032] See Figure 1, Figure 1 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 1 As shown, the system architecture 100 can be used to implement the method provided in the embodiments of this application. The system architecture 100 may include a training device 101, an application device 102, a user device 103, a data acquisition device 104, and a database 105.

[0033] The acquisition device 104 can be used to acquire image data as training samples for model training and store the training samples in the database 105 for later use. For example, taking image classification as an example, the training samples may include sample images and their corresponding ground truth class labels, where the ground truth class labels can be manually pre-labeled. Optionally, the acquisition device 104 in this embodiment can be a camera or other similar device.

[0034] It should be noted that, in this embodiment, the training samples stored in the database 105 may include not only image data obtained by the acquisition device 104, but also image data obtained from publicly available datasets on the Internet. This embodiment does not specifically limit this.

[0035] Based on this, training device 101 can be used to train a model according to the method provided in the embodiments of this application, based on the training samples stored in database 105, to obtain a target neural network. Then, the target neural network can be deployed in application device 102. Optionally, in this embodiment, the target neural network can be deployed in the computing module 1021 of application device 102.

[0036] In this embodiment, as an example, the target neural network can be a neural network suitable for the current application scenario. For example, for image classification scenarios, the target neural network can include CNN (Convolutional Neural Network), VGGNet (Visual Geometry Group Network), etc. For object detection scenarios, the target neural network can include YOLO (You Only Look Once) network, SSD (Single Shot Multibox Detector), etc., etc., and will not be listed one by one here.

[0037] The user can transmit the image data to be processed to the application device 102 through the user device 103. After receiving the image data to be processed through the I / O (Input / Output) interface 1022, the application device 102 can use the target neural network to perform image processing based on the image data to be processed input by the user device 103 to obtain the image processing result, and return the image processing result to the user device 103 through the I / O interface 1022.

[0038] For example, taking image classification as an example, a user can transmit image data containing objects to be classified (such as cats, dogs, etc.) to an application device 102 through a user device 103. After receiving the image data containing the objects to be classified through an I / O interface 1022, the application device 102 will use a target neural network to perform image classification processing on the image data to obtain the image classification result (such as the object category to which the object to be classified belongs), and return the image classification result to the user device 103 through the I / O interface 1022.

[0039] It should be noted that in this embodiment, the training device 101 and the application device 102 can be the same device or different devices, and there is no specific limitation here.

[0040] This concludes the process. Figure 1 The system architecture shown is described below. Figure 2 The method flow provided in the embodiments of this application is described.

[0041] See Figure 2 , Figure 2 This is a flowchart illustrating a method provided in an embodiment of this application. The method can be applied to at least image processing scenarios, optionally including image classification, object detection, image segmentation, and pose estimation scenarios, etc., without specific limitations. In this embodiment, the method can be executed by an electronic device (such as the training device described above). For example, as an embodiment, the electronic device can be a server, laptop, smart terminal, etc., without specific limitations.

[0042] like Figure 2 As shown, the process may include the following steps:

[0043] Step 201: In the current training cycle, determine the sparsity used to sparsify the weight parameters of the current first neural network.

[0044] In this embodiment, by combining sparsification, knowledge distillation is performed on the pre-trained neural network to obtain the target neural network for image processing. Specifically, in each training cycle, this embodiment first determines the sparsity used to sparsify the weight parameters of the current first neural network, so as to sparsify the weight parameters of the current first neural network to obtain the second neural network, and then performs knowledge distillation on the second neural network.

[0045] In this embodiment, the sparsity determined under different training cycles is different. As an example, the above-mentioned determination of the sparsity used to sparsify the weight parameters of the current first neural network under the current training cycle can be implemented in a specific way, for example, by determining the sparsity used to sparsify the weight parameters of the current first neural network based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient; wherein, the sparsity adjustment coefficient is determined based on the preset sparsity and the total number of training cycles.

[0046] As can be seen, in this embodiment, the weight parameters of the current first neural network are not directly sparsified according to a fixed preset sparsity in each training cycle. Instead, the sparsity corresponding to the current training cycle is determined by the cycle identifier and sparsity adjustment coefficient. That is, iterative sparsity (i.e., dynamic sparsity) is gradually performed in each training cycle. This iterative sparsity method can gradually increase the difference in output features, i.e., knowledge difference, between the student network (i.e., the second neural network) and the teacher network (i.e., the pre-trained neural network) in knowledge distillation in each training cycle. Compared with directly sparsifying the weight parameters of the current first neural network according to a fixed preset sparsity, which leads to a large difference in output features and thus increases the learning difficulty of the student network, this method can effectively reduce the learning difficulty of the student network and improve the performance of the subsequent target neural network (i.e., the trained neural network).

[0047] As for how to determine the sparsity used to sparsify the weight parameters of the current first neural network based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient, and how to determine the sparsity adjustment coefficient based on the preset sparsity and the total number of training cycles, examples will be given below, and will not be elaborated here.

[0048] In this embodiment, if the current training period is the first training period, the first neural network is a pre-trained neural network, that is, a copy of the pre-trained neural network can be used as the first neural network. If the current training period is not the first training period, the first neural network is a network obtained by knowledge distillation of the pre-trained neural network in the historical training period.

[0049] Here, the pre-trained neural network can be considered as the teacher network in knowledge distillation. It should be noted that the weight parameters of the pre-trained neural network remain fixed during the knowledge distillation process. Optionally, this embodiment does not specifically limit how the pre-trained neural network is obtained; it can be flexibly set based on actual application needs. For example, the pre-trained neural network can be trained based on a pre-prepared training dataset. As for how specifically the pre-trained neural network is trained based on a pre-prepared training dataset, this embodiment does not specifically limit it.

[0050] Step 202: Based on the above sparsity, the weight parameters in the current first neural network are sparsified to obtain the second neural network; the network loss value is determined using each training sample and the second neural network.

[0051] In this embodiment, after determining the sparsity for sparsifying the weight parameters of the current first neural network, the weight parameters in the current first neural network are sparsified based on this sparsity to obtain a second neural network. Then, the network loss value is determined using each training sample and the second neural network for subsequent network optimization. This embodiment, by coupling knowledge distillation and sparsification in each training cycle, forms an end-to-end training framework for dynamic collaborative optimization of knowledge distillation and sparsification. This not only compensates for the accuracy loss caused by sparsification, minimizing accuracy loss while achieving model compression, but also avoids the performance degradation caused by training sparsification and knowledge distillation in stages.

[0052] Optionally, the training dataset used in each training cycle in this embodiment can be the same or different, and there is no specific limitation here.

[0053] As for how to sparsify the weight parameters in the first neural network based on the above sparsity, and how to determine the network loss value using each training sample and the second neural network, examples will be given below, and will not be elaborated here.

[0054] Step 203: Based on the gradient of each weight parameter in the first neural network according to the above network loss value, optimize the weight parameters in the first neural network to obtain the third neural network; if the current training period is not the last training period, the third neural network is used as the first neural network in the next training period, and when the next training period is the current training period, return to the step of determining the sparsity used to sparsify the weight parameters of the current first neural network; if the current training period is the last training period, the weight parameters of the third neural network are sparsified using the preset sparsity to obtain the target neural network.

[0055] In this embodiment, after obtaining the network loss value used to optimize each weight parameter in the first neural network, the weight parameters in the first neural network are optimized based on the gradient of each weight parameter in the first neural network using the network loss value, thus obtaining the third neural network.

[0056] Next, check if the current training cycle is the last training cycle; if the current training cycle is not the last training cycle, then use the third neural network as the first neural network in the next training cycle, and return to the above steps of determining the sparsity of the weight parameters used to sparsify the current first neural network when the next training cycle is the current training cycle.

[0057] If the current training cycle is the last training cycle, it indicates that training has ended. At this time, the weight parameters of the third neural network can be sparsified using a preset sparsity to obtain the target neural network. This target neural network can be used to perform image processing based on the input image data in image processing scenarios. As described above, the image processing here may include, but is not limited to, image classification, object detection, image segmentation, pose estimation, etc., and this embodiment is not specifically limited to these.

[0058] As for how to optimize the weight parameters in the first neural network based on the gradient of each weight parameter in the first neural network based on the above network loss value, the following will provide an example description, which will not be elaborated here.

[0059] This concludes the process. Figure 2 The process is shown below.

[0060] pass Figure 2 As can be seen from the process shown, in this embodiment of the application, by first using the currently determined sparsity to sparsify the weight parameters in the current first neural network to obtain the second neural network in each training cycle of knowledge distillation, and then using each training sample and the second neural network to perform knowledge distillation, an end-to-end training framework for dynamic collaborative optimization of knowledge distillation and sparsity is formed by coupling knowledge distillation and sparsity during the training process. This not only makes up for the accuracy loss caused by sparsity, so as to minimize the accuracy loss while achieving model compression, but also avoids the performance degradation problem caused by training sparsity and knowledge distillation in stages.

[0061] Furthermore, in this embodiment, the neural network is not sparsified all at once according to the final sparsity, but iteratively sparsified gradually in each training cycle. By gradually increasing the difference in output features (i.e., knowledge difference) between the student network (i.e., the second neural network) and the teacher network (i.e., the pre-trained neural network) during knowledge distillation, the learning difficulty of the student network can be avoided due to excessively high initial sparsity leading to large differences in output features, thus effectively reducing the learning difficulty of the student network. In addition, this embodiment also optimizes the neural network before sparsification (i.e., the first neural network) by utilizing the network loss value. This avoids the problem of decreased neural network accuracy due to the non-participation of some sparsified weight parameters in the optimization, thereby enhancing the learning ability of the student network. The performance of the trained student network (i.e., the target neural network) is basically no different from that of the pre-trained neural network (i.e., the teacher network), thus achieving the goal of minimizing accuracy loss while compressing the neural network using sparsification methods.

[0062] The following describes how, in step 201 above, the sparsity used to sparsify the weight parameters of the current first neural network is determined based on the period identifier and sparsity adjustment coefficient of the current training period:

[0063] In this embodiment, as one example, the sparsity used to sparsify the weight parameters of the current first neural network is determined based on the period identifier of the current training period and the sparsity adjustment coefficient. In a specific implementation, for example, the period identifier of the current training period and the sparsity adjustment coefficient can be input as input parameters into the sparsity calculation formula shown below to obtain the corresponding sparsity:

[0064] .

[0065] in, Indicates sparsity. This represents the sparsity adjustment factor. The period identifier that indicates the current training period.

[0066] Optionally, the period identifier here can be the number of periods. For example, if the current training period is the first training period (i.e., the first training period), then the period identifier of the current training period is 1; if the current training period is the second training period, then the period identifier of the current training period is 2, and so on.

[0067] In this embodiment, as one example, the above-mentioned determination of the sparsity adjustment coefficient relies on the preset sparsity and the total number of training epochs. In a specific implementation, for example, the preset sparsity and the total number of training epochs can be input as input parameters into the following coefficient calculation formula to obtain the sparsity adjustment coefficient:

[0068] .

[0069] in, Indicates the preset sparsity. This represents the total number of training cycles.

[0070] Optionally, the preset sparsity and the total number of training cycles can be flexibly set based on actual application needs. For example, the preset sparsity can be 0.5 or 0.6, and the total number of training cycles can be 10 or 20. This embodiment does not specifically limit these values.

[0071] The following describes the sparsification process performed on the weight parameters in the first neural network in step 202 based on the aforementioned sparsity:

[0072] In this embodiment, the sparsification of the weight parameters in the first neural network based on the aforementioned sparsity can be implemented in many ways:

[0073] For example, as an implementation, firstly, the weight parameters in the first neural network are sorted in order to obtain a weight sequence. This order could be, for example, from largest to smallest.

[0074] Next, the last N weight parameters in the weight sequence are set to specified values, while the current values ​​of the remaining weight parameters are kept unchanged. These specified values ​​can be, for example, 0.

[0075] In this embodiment, the aforementioned N can be determined based on the total number of weight parameters in the weight sequence and the sparsity determined in the current training period. Specifically, as an example, N can be determined as the product of the total number of weight parameters in the weight sequence and the sparsity determined in the current training period.

[0076] For example, assuming the total number of weight parameters in the first neural network is 100 and the sparsity determined in the current training cycle is 0.2, then N = 100 * 0.2 = 20.

[0077] For example, as another embodiment, the weight parameters in the first neural network can be subjected to structured sparsity processing based on sparsity. Specifically, in implementation, each weight parameter in the first neural network is represented in the form of a weight matrix. Based on this, for each row of weight parameters in the weight matrix, the last N weight parameters in that row are set to specified values, while the current values ​​of the remaining weight parameters in that row remain unchanged. Then, for each column of weight parameters in the weight matrix, the last N weight parameters in that column are set to specified values, while the current values ​​of the remaining weight parameters in that column remain unchanged.

[0078] For example, as another embodiment, semi-structured sparsity processing can be performed on the weight parameters in the first neural network based on sparsity. Specifically, in implementation, for each row of weight parameters in the weight matrix, the last N weight parameters in that row can be set to specified values, while keeping the current values ​​of the remaining weight parameters in that row unchanged.

[0079] As for how to use the preset sparsity to sparsify the weight parameters of the third neural network, it is similar to the sparsification of the weight parameters in the current first neural network based on the above sparsity. Please refer to the relevant description above, which will not be repeated here.

[0080] The following describes step 202 above, which involves determining the network loss value using the training samples and the second neural network:

[0081] As an example, see Figure 3 As shown, the above-mentioned method of determining the network loss value using each training sample and the second neural network may include the following steps in its specific implementation:

[0082] Step 301: Based on each training sample and the second neural network, determine the feature-based distillation loss value; wherein the feature-based distillation loss value is used to indicate the difference in output features of the same layer between the second neural network and the pre-trained neural network.

[0083] In this embodiment, as one example, the step of determining the feature-based distillation loss value based on each training sample and the second neural network can be implemented as follows: First, each training sample is input into the second neural network to obtain the first sample features corresponding to each training sample output by a specified intermediate layer of the second neural network; then, each training sample is input into a pre-trained neural network to obtain the second sample features corresponding to each training sample output by a specified intermediate layer of the pre-trained neural network; then, the first sample features and second sample features corresponding to each training sample are used as input parameters to a preset first loss function to obtain a first loss value, and the feature-based distillation loss value is determined based on the first loss value.

[0084] In this embodiment, the intermediate layer refers to the layer located between the input layer and the output layer in the neural network, which can be used to perform processing such as feature extraction on the input data. The specified intermediate layer in this embodiment can be flexibly set based on actual application needs; for example, it can be any intermediate layer in the neural network, and there is no specific limitation here.

[0085] Optionally, this embodiment does not specifically limit the first loss function, and it can be flexibly set according to actual needs. For example, it can be the KL (Kullback-Leibler) divergence loss function or the mean squared error loss function.

[0086] In this embodiment, as one example, the above-mentioned determination of the feature-based distillation loss value based on the first loss value can be implemented in a specific way, for example, by determining the first loss value as the feature-based distillation loss value.

[0087] Step 302: Using each training sample and the second neural network, determine the soft label distillation loss value; wherein the soft label distillation loss value is used to indicate the difference in output results between the second neural network and the pre-trained neural network.

[0088] In this embodiment, as an example, the soft label distillation loss value is determined using each training sample and the second neural network in this step. In specific implementation, it can be as follows: first, each training sample is input into the second neural network to obtain the first output result corresponding to each training sample output by the output layer of the second neural network; then, each training sample is input into a pre-trained neural network to obtain the second output result corresponding to each training sample output by the output layer of the pre-trained neural network; then, the first output result and the second output result corresponding to each training sample are used as input parameters to a preset second loss function to obtain the second loss value, and the soft label distillation loss value is determined based on the second loss value.

[0089] Optionally, this embodiment does not specifically limit the second loss function, and it can be flexibly set according to actual needs, such as the cross-entropy loss function or the L1 norm loss function.

[0090] In this embodiment, as one example, the above-mentioned determination of the soft label distillation loss value based on the second loss value can be implemented in a specific way, for example, by determining the second loss value as the soft label distillation loss value.

[0091] Step 303: Determine the network loss value based on the feature-based distillation loss value and the soft-label distillation loss value.

[0092] In this embodiment, the determination of the network loss value based on the feature-based distillation loss value and the soft-label distillation loss value in this step can be implemented in many ways. For example, as one embodiment, the average of the feature-based distillation loss value and the soft-label distillation loss value can be used as the network loss value. As another embodiment, the weighted average of the feature-based distillation loss value and the soft-label distillation loss value can be used as the network loss value.

[0093] This completes the work. Figure 3 The method flow shown is described.

[0094] The following describes the optimization of the weight parameters in the first neural network based on the gradients of the aforementioned network loss values ​​in step 203:

[0095] In this embodiment, as one example, the optimization of the weight parameters in the first neural network based on the gradient of each weight parameter in the first neural network according to the above-mentioned network loss value may, in specific implementation, include:

[0096] First, for each weight parameter, the weight parameter and its input are fed into a preset gradient calculation function to obtain the gradient of the network loss value for that weight parameter.

[0097] Optionally, this embodiment does not specifically limit the gradient calculation function, and it can be flexibly set according to actual application needs. For example, it can be: gradient ,in, Denotes the first loss function. Denotes the first loss function. Represents the weight parameters. express Compared to The partial derivative results, express Compared to The partial derivative results.

[0098] Then, the gradient of the network loss value with respect to the weight parameter is used as an input parameter to the weight optimization function to obtain the optimized weight parameter.

[0099] Optionally, this embodiment does not specifically limit the weight optimization function, and it can be flexibly set according to actual application needs. For example, it can be: optimized weight parameter = unoptimized weight parameter - a * gradient of network loss value at that weight parameter. This indicates the optimization parameters, which can be flexibly set based on actual needs.

[0100] To facilitate understanding of the specific implementation process of the above-mentioned dynamic weight optimization method, specific examples are provided below.

[0101] As an example, the specific implementation process of the weight dynamic optimization method provided in this embodiment is as follows:

[0102] Step 1: Train a pre-trained neural network using a pre-prepared training dataset.

[0103] Step 2: In the current training cycle, based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient, determine the sparsity used to sparsify the weight parameters of the current first neural network, and perform sparsification processing on the weight parameters in the current first neural network based on the sparsity to obtain the second neural network.

[0104] Step 3: Determine the network loss value using the training samples and the second neural network in the current training cycle, and use the network loss value for backpropagation to optimize the weight parameters in the first neural network to obtain the third neural network.

[0105] For example, see Figure 4 As shown, each training sample is input into the first neural network to obtain the first sample features corresponding to each training sample output by the specified intermediate layer of the second neural network, and the first output result corresponding to each training sample output by the output layer of the second neural network.

[0106] Each training sample is input into the first neural network and the pre-trained neural network to obtain the second sample features corresponding to each training sample output by the specified intermediate layer of the second neural network, and the second output results corresponding to each training sample output by the output layer of the pre-trained neural network.

[0107] Subsequently, feature-based distillation loss values ​​are obtained based on the first and second sample features corresponding to each training sample, and soft-label distillation loss values ​​are obtained based on the first and second output results corresponding to each training sample. The network loss value is then determined based on these feature-based and soft-label distillation loss values, and backpropagation is performed using these network loss values ​​to optimize the weight parameters in the first neural network.

[0108] Optionally, in this embodiment, the gradient of the network loss value with respect to each weight parameter can be directly estimated using the Straight-Through Estimator (STE) method, so as to optimize each weight parameter based on the gradient of the network loss value with respect to each weight parameter. As to how the gradient of the network loss value with respect to each weight parameter is directly estimated using the STE method, this embodiment does not specifically limit the specific method.

[0109] Step 4: If the current training period is not the last training period, the third neural network is used as the first neural network in the next training period, and when the next training period is the current training period, the process returns to Step 2 above; if the current training period is the last training period, the weight parameters of the third neural network are sparsified using a preset sparsity to obtain the target neural network, which is used for image processing.

[0110] This concludes the description of the method provided in the embodiments of this application. The apparatus provided in the embodiments of this application will now be described:

[0111] As an example, this embodiment also provides a dynamic weight optimization device. For example, see... Figure 5 , Figure 5 This is a schematic diagram of a weight dynamic optimization device provided in an embodiment of this application. The device corresponds to… Figure 2The process is shown below. Figure 5 As shown, the weight dynamic optimization device 500 can be applied to at least image processing scenarios and can be executed by an electronic device. The weight dynamic optimization device 500 includes: a determination module 501, a sparsity module 502, an optimization module 503, and a training module 504.

[0112] The determining module 501 is used to determine the sparsity of the weight parameters of the current first neural network in the current training cycle. The sparsity determined in different training cycles is different. If the current training cycle is the first training cycle, the first neural network is a pre-trained neural network. If the current training cycle is not the first training cycle, the first neural network is a network obtained by knowledge distillation of the pre-trained neural network in the historical training cycles.

[0113] The sparsity module 502 is used to sparsify the weight parameters in the current first neural network based on the sparsity to obtain a second neural network; and to determine the network loss value using each training sample and the second neural network.

[0114] Optimization module 503 is used to optimize the weight parameters in the first neural network based on the gradient of each weight parameter in the first neural network based on the network loss value, so as to obtain a third neural network;

[0115] Training module 504 is used to take the third neural network as the first neural network in the next training cycle if the current training cycle is not the last training cycle, and return the step of determining the sparsity of the weight parameters of the current first neural network when the next training cycle is the current training cycle.

[0116] If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsified using a preset sparsity to obtain the target neural network. The target neural network is used to perform image processing based on the input image data to be processed in the image processing scenario.

[0117] As one embodiment, determining the sparsity for sparsifying the weight parameters of the current first neural network includes:

[0118] The sparsity is determined based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient.

[0119] The sparsity adjustment coefficient is determined based on the preset sparsity and the total number of training cycles.

[0120] As an example, the sparsification process of the weight parameters in the current first neural network based on the sparsity includes:

[0121] The weight parameters in the first neural network are sorted in order to obtain the weight sequence;

[0122] The last N weight parameters in the weight sequence are set to specified values, while the current values ​​of the remaining weight parameters remain unchanged; wherein, N depends on the total number of weight parameters in the weight sequence and the sparsity.

[0123] As one embodiment, determining the network loss value using each training sample and the second neural network includes:

[0124] Based on each training sample and the second neural network, a feature-based distillation loss value is determined; wherein the feature-based distillation loss value is used to indicate the difference in output features of the same layer in the second neural network and the pre-trained neural network;

[0125] Using each training sample and the second neural network, a soft-label distillation loss value is determined; wherein, the soft-label distillation loss value is used to indicate the difference in output results between the second neural network and the pre-trained neural network;

[0126] The network loss value is determined based on the feature-based distillation loss value and the soft-label distillation loss value.

[0127] As one embodiment, determining the feature-based distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes:

[0128] Each training sample is input into the second neural network to obtain the first sample features corresponding to each training sample output by a specified intermediate layer of the second neural network.

[0129] Each training sample is input into the pre-trained neural network to obtain the second sample features corresponding to each training sample output by the specified intermediate layer of the pre-trained neural network;

[0130] The first sample feature and the second sample feature corresponding to each training sample are used as input parameters to a preset first loss function to obtain a first loss value, and the feature-based distillation loss value is determined based on the first loss value.

[0131] As one embodiment, determining the soft-label distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes:

[0132] Each training sample is input into the second neural network to obtain the first output result corresponding to each training sample output by the output layer of the second neural network;

[0133] Each training sample is input into the pre-trained neural network to obtain the second output result corresponding to each training sample output by the output layer of the pre-trained neural network;

[0134] The first and second output results corresponding to each training sample are used as input parameters to a preset second loss function to obtain a second loss value, and the soft label distillation loss value is determined based on the second loss value.

[0135] This concludes the process. Figure 5 Structural description of the device shown.

[0136] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0137] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0138] Please see Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided as an exemplary embodiment of this application. The electronic device includes a processor and a computer-readable storage medium; the computer-readable storage medium stores a plurality of computer program instructions, which, when executed by the processor, implement the method disclosed in the above example of this application. Depending on the actual function of the electronic device, other hardware may also be included, which will not be elaborated further.

[0139] Based on the same concept as the above method, this application also provides a computer-readable storage medium storing a plurality of computer program instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.

[0140] For example, the aforementioned computer-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, messages, etc. For instance, computer-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0141] The above are merely preferred embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A dynamic weight optimization method, characterized in that, The method is applied to at least image processing scenarios, including image classification, object detection, image segmentation, and pose estimation scenarios. The method includes: In the current training cycle, the sparsity is determined based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient; wherein, the sparsity adjustment coefficient is determined based on the preset sparsity and the total number of training cycles; the sparsity determined in different training cycles is different, and if the current training cycle is the first training cycle, then the first neural network is a pre-trained neural network; if the current training cycle is not the first training cycle, then the first neural network is a network obtained by knowledge distillation of the pre-trained neural network in the historical training cycles; Based on the sparsity, the weight parameters in the current first neural network are sparsified to obtain the second neural network; the network loss value is determined using each training sample and the second neural network. Based on the gradient of each weight parameter in the first neural network according to the network loss value, the weight parameters in the first neural network are optimized to obtain the third neural network; If the current training period is not the last training period, the third neural network is used as the first neural network in the next training period, and when the next training period is the current training period, the step of determining the sparsity of the weight parameters of the current first neural network is returned. If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsified using a preset sparsity to obtain the target neural network. The target neural network is used to perform image processing based on the input image data to be processed in the image processing scenario.

2. The method according to claim 1, characterized in that, The sparsification process for the weight parameters in the current first neural network based on the sparsity includes: The weight parameters in the first neural network are sorted in order to obtain the weight sequence; The last N weight parameters in the weight sequence are set to specified values, while the current values ​​of the remaining weight parameters are kept unchanged; wherein, N depends on the total number of weight parameters in the weight sequence and the sparsity.

3. The method according to claim 1, characterized in that, The step of determining the network loss value using each training sample and the second neural network includes: Based on each training sample and the second neural network, a feature-based distillation loss value is determined; wherein the feature-based distillation loss value is used to indicate the difference in output features of the same layer in the second neural network and the pre-trained neural network; Using each training sample and the second neural network, a soft-label distillation loss value is determined; wherein, the soft-label distillation loss value is used to indicate the difference in output results between the second neural network and the pre-trained neural network; The network loss value is determined based on the feature-based distillation loss value and the soft-label distillation loss value.

4. The method according to claim 3, characterized in that, The step of determining the feature-based distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Each training sample is input into the second neural network to obtain the first sample features corresponding to each training sample output by a specified intermediate layer of the second neural network. Each training sample is input into the pre-trained neural network to obtain the second sample features corresponding to each training sample output by the specified intermediate layer of the pre-trained neural network; The first sample feature and the second sample feature corresponding to each training sample are used as input parameters to a preset first loss function to obtain a first loss value, and the feature-based distillation loss value is determined based on the first loss value.

5. The method according to claim 3, characterized in that, The step of determining the soft-label distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Each training sample is input into the second neural network to obtain the first output result corresponding to each training sample output by the output layer of the second neural network; Each training sample is input into the pre-trained neural network to obtain the second output result corresponding to each training sample output by the output layer of the pre-trained neural network; The first and second output results corresponding to each training sample are used as input parameters to a preset second loss function to obtain a second loss value, and the soft label distillation loss value is determined based on the second loss value.

6. A weighted dynamic optimization device, characterized in that, The apparatus is applied to at least image processing scenarios, including at least image classification, object detection, image segmentation, and pose estimation scenarios. The apparatus includes: The determination module is used to determine the sparsity in the current training cycle based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient; wherein, the sparsity adjustment coefficient is determined based on the preset sparsity and the total number of training cycles; the sparsity determined in different training cycles is different, and if the current training cycle is the first training cycle, then the first neural network is a pre-trained neural network; if the current training cycle is not the first training cycle, then the first neural network is a network obtained by knowledge distillation of the pre-trained neural network in the historical training cycles; A sparse module is used to sparsify the weight parameters in the current first neural network based on the sparsity to obtain a second neural network; and to determine the network loss value using each training sample and the second neural network. An optimization module is used to optimize the weight parameters in the first neural network based on the gradient of each weight parameter in the first neural network, thereby obtaining a third neural network. The training module is used to, if the current training period is not the last training period, use the third neural network as the first neural network in the next training period, and return the step of determining the sparsity of the weight parameters of the current first neural network when the next training period is the current training period. If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsified using a preset sparsity to obtain the target neural network. The target neural network is used to perform image processing based on the input image data to be processed in the image processing scenario.

7. The apparatus according to claim 6, characterized in that, The sparsification process for the weight parameters in the current first neural network based on the sparsity includes: The weight parameters in the first neural network are sorted in order to obtain the weight sequence; The last N weight parameters in the weight sequence are set to specified values, while the current values ​​of the remaining weight parameters remain unchanged; wherein, N depends on the total number of weight parameters in the weight sequence and the sparsity. And / or, The step of determining the network loss value using each training sample and the second neural network includes: Based on each training sample and the second neural network, a feature-based distillation loss value is determined; wherein the feature-based distillation loss value is used to indicate the difference in output features of the same layer in the second neural network and the pre-trained neural network; Using each training sample and the second neural network, a soft-label distillation loss value is determined; wherein, the soft-label distillation loss value is used to indicate the difference in output results between the second neural network and the pre-trained neural network; The network loss value is determined based on the feature-based distillation loss value and the soft-label distillation loss value; And / or, The step of determining the feature-based distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Each training sample is input into the second neural network to obtain the first sample features corresponding to each training sample output by a specified intermediate layer of the second neural network. Each training sample is input into the pre-trained neural network to obtain the second sample features corresponding to each training sample output by the specified intermediate layer of the pre-trained neural network; The first sample feature and the second sample feature corresponding to each training sample are used as input parameters to a preset first loss function to obtain a first loss value, and the feature-based distillation loss value is determined based on the first loss value. And / or, The step of determining the soft-label distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Each training sample is input into the second neural network to obtain the first output result corresponding to each training sample output by the output layer of the second neural network; Each training sample is input into the pre-trained neural network to obtain the second output result corresponding to each training sample output by the output layer of the pre-trained neural network; The first and second output results corresponding to each training sample are used as input parameters to a preset second loss function to obtain a second loss value, and the soft label distillation loss value is determined based on the second loss value.

8. An electronic device, characterized in that, The electronic device includes: Processor; and A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 5.

Images