Weight dynamic optimization method and device and electronic equipment

Abstract

The embodiment of the invention provides a weight dynamic optimization method and device and electronic equipment. In the embodiment of the invention, in each training period of knowledge distillation, the weight parameter in the current first neural network is sparsened by using the currently determined sparseness to obtain the second neural network, and then the knowledge distillation is performed by using each training sample and the second neural network. In this way, an end-to-end training framework of knowledge distillation and rarefaction dynamic collaborative optimization is formed by coupling knowledge distillation and rarefaction in the training process, precision loss caused by rarefaction can be made up, and the precision loss is reduced to the maximum extent on the basis of achieving model compression; and the problem of performance reduction caused by staged training of rarefaction and knowledge distillation can be avoided.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and electronic device for dynamic weight optimization. Background Art

[0002] A neural network, also known as an artificial neural network (ANN), is a mathematical model that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. Neural networks are often deployed in electronic devices and can be used for image processing such as image classification, object detection, image segmentation, and pose estimation. With the continuous advancement of artificial intelligence technology, the scale of neural networks has continued to increase, resulting in a significant increase in their storage and computational overhead. However, due to the limited memory and computing power of electronic devices, researchers have proposed using model compression methods to reduce the storage and computational overhead of neural networks.

[0003] Currently, in practical applications, sparsification methods such as weight sparsification are usually used to compress neural network models in order to reduce the storage and computational overhead of the neural network. Weight sparsification here can refer to resetting some weights in the neural network to zero, thereby reducing the number of network parameters to achieve the purpose of compressing the model. However, the use of the above-mentioned sparsification methods will cause changes in the network structure, which may lead to a decrease in model accuracy and affect the accuracy of image processing. For example, in image classification scenarios, sparsification may cause some key image feature information to be unable to be effectively extracted, resulting in a decrease in image classification accuracy. Therefore, how to minimize the loss of accuracy while compressing the neural network using sparsification methods has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In view of this, the present application provides a weight dynamic optimization method, device and electronic device to minimize the accuracy loss while using a sparsification method to compress the neural network.

[0005] The present application provides a method for dynamic weight optimization, which is applied at least to image processing scenarios, including at least image classification scenarios, target detection scenarios, image segmentation scenarios, and pose estimation scenarios. The method includes: In a current training cycle, determining a sparsity for sparsifying weight parameters of a current first neural network; the sparsity determined in different training cycles is different, and if the current training cycle is the first training cycle, the first neural network is a pre-trained neural network; if the current training cycle is not the first training cycle, the first neural network is a network obtained by performing knowledge distillation on the pre-trained neural network in a previous training cycle; Based on the sparsity, the weight parameters in the current first neural network are subjected to sparse processing to obtain a second neural network; and a network loss value is determined using each training sample and the second neural network; Optimizing the weight parameters in the first neural network based on the gradient of each weight parameter of the network loss value in the first neural network to obtain a third neural network; If the current training cycle is not the last training cycle, using the third neural network as the first neural network of the next training cycle, and returning to the step of determining the sparsity for sparsifying the weight parameters of the current first neural network when the next training cycle is the current training cycle; If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsely processed using a preset sparsity to obtain a target neural network, which is used to perform image processing based on the input image data to be processed in the image processing scenario.

[0006] The present application also provides a weight dynamic optimization device, which is applied to at least an image processing scenario, including at least an image classification scenario, a target detection scenario, an image segmentation scenario, and a pose estimation scenario. The device includes: a determination module for determining, in a current training cycle, a sparsity for sparsifying weight parameters of a current first neural network; the sparsity determined in different training cycles being different, and if the current training cycle is the first training cycle, the first neural network being a pre-trained neural network; and if the current training cycle is not the first training cycle, the first neural network being a network obtained by performing knowledge distillation on the pre-trained neural network in a previous training cycle; A sparse module is used to perform sparse processing on the weight parameters in the current first neural network based on the sparsity to obtain a second neural network; and determine a network loss value using each training sample and the second neural network; an optimization module, configured to optimize the weight parameters in the first neural network based on the gradient of each weight parameter of the network loss value in the first neural network to obtain a third neural network; a training module configured to use the third neural network as the first neural network of a next training cycle if the current training cycle is not the last training cycle, and return to the step of determining a sparsity for sparsifying the weight parameters of the current first neural network when the next training cycle is the current training cycle; If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsely processed using a preset sparsity to obtain a target neural network, which is used to perform image processing based on the input image data to be processed in the image processing scenario.

[0007] An embodiment of the present application further provides an electronic device, comprising: processor; and A computer-readable storage medium stores computer program instructions, which, when executed by a processor, cause the processor to execute the steps of the above method.

[0008] An embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor is caused to execute the steps in the above method.

[0009] It can be seen from the above technical solution that in the embodiment of the present application, in each training cycle of knowledge distillation, the weight parameters in the current first neural network are first sparsely used using the currently determined sparsity to obtain a second neural network, and then the training samples and the second neural network are used for knowledge distillation. In this way, by coupling knowledge distillation and sparsification during the training process, an end-to-end training framework for dynamic collaborative optimization of knowledge distillation and sparsification is formed, which can not only make up for the accuracy loss caused by sparsification, so as to minimize the accuracy loss on the basis of achieving model compression, but also avoid the performance degradation problem caused by the staged training of sparsification and knowledge distillation.

[0010] Furthermore, in the embodiment of the present application, the neural network is not sparsified at one time according to the final sparsity, but is iteratively sparsified step by step in each training cycle. In this way, by gradually increasing the output feature difference, that is, the knowledge difference, between the student network, i.e., the second neural network, and the teacher network, i.e., the pre-trained neural network in knowledge distillation, it is possible to avoid the situation where the difficulty of learning the student network increases suddenly due to excessive output feature difference caused by the initial sparsity being too high, thereby effectively reducing the difficulty of learning the student network. Moreover, this embodiment also optimizes the neural network before sparsification, i.e., the first neural network, by utilizing the network loss value. In this way, it is possible to avoid the problem of decreased accuracy of the neural network due to the fact that some of the weight parameters that have been sparsified do not participate in the optimization, thereby enhancing the learning ability of the student network, so that the performance of the trained student network, i.e., the target neural network, is basically not reduced compared to the pre-trained neural network, i.e., the teacher network, thereby achieving the goal of minimizing accuracy loss while compressing the neural network using the sparsification method. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1 A schematic diagram of the system architecture provided in an embodiment of the present application.

[0013] Figure 2 A schematic diagram of the method flow provided in an embodiment of the present application.

[0014] Figure 3 A flowchart of another method provided in an embodiment of the present application.

[0015] Figure 4 A schematic diagram of the training implementation provided in an embodiment of the present application.

[0016] Figure 5 A schematic diagram of the device structure provided in an embodiment of the present application.

[0017] Figure 6 A schematic diagram of the electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0019] The following describes the system architecture used in the embodiments of the present application: See also Figure 1 , Figure 1 This is a schematic diagram of the system architecture provided in the embodiment of this application. Figure 1 As shown, the system architecture 100 can be used to implement the method provided in the embodiment of the present application. The system architecture 100 may include a training device 101, an application device 102, a user device 103, a collection device 104, and a database 105.

[0020] The acquisition device 104 can be used to collect image data as training samples for model training and store these training samples in the database 105 for future use. For example, in image classification, the training samples may include sample images and their corresponding true category labels, where the true category labels may be manually pre-labeled. Optionally, the acquisition device 104 in this embodiment may be a camera or other device.

[0021] It should be noted that, in this embodiment, the training samples stored in the above-mentioned database 105 may include not only image data obtained by using the acquisition device 104, but also image data obtained from data sets publicly available on the Internet, etc., and this embodiment does not specifically limit this.

[0022] Based on this, the training device 101 can be used to perform model training based on the training samples stored in the database 105 according to the method provided in the embodiment of the present application to obtain a target neural network. Afterwards, the target neural network can be deployed on the application device 102. Optionally, in this embodiment, the target neural network can be deployed in the computing module 1021 of the application device 102.

[0023] In this embodiment, as an embodiment, the above-mentioned target neural network can be a neural network suitable for the current application scenario. For example, for image classification scenarios, the target neural network can include CNN (Convolutional Neural Network), VGGNet (Visual Geometry Group Network), etc. For target detection scenarios, the target neural network can include YOLO (You Only Look Once) network, SSD (Single Shot Multibox Detector), etc., and examples are not given here one by one.

[0024] The user can transmit the image data to be processed to the application device 102 through the user device 103. After that, after receiving the image data to be processed through the I / O (Input / Output) interface 1022, the application device 102 can use the target neural network to perform image processing based on the image data to be processed input by the user device 103 to obtain an image processing result, and return the image processing result to the user device 103 through the I / O interface 1022.

[0025] For example, taking image classification as an example, a user can transmit image data to be processed containing objects to be classified (such as cats, dogs, etc.) to the application device 102 through the user device 103. After the application device 102 receives the image data to be processed containing the objects to be classified through the I / O interface 1022, it will use the target neural network to perform image classification processing on the image data to obtain an image classification result (such as the object category to which the object to be classified belongs), and return the image classification result to the user device 103 through the I / O interface 1022.

[0026] It should be noted that, in this embodiment, the training device 101 and the application device 102 may be the same device or different devices, which is not specifically limited here.

[0027] This is completed Figure 1 The following is a description of the system architecture. Figure 2 The method flow provided in the embodiments of the present application is described.

[0028] See also Figure 2, Figure 2 This is a flowchart of a method provided in an embodiment of the present application. This method can be applied to at least image processing scenarios. Optionally, these image processing scenarios include at least image classification, object detection, image segmentation, and pose estimation, among others, though these are not specifically limited here. In this embodiment, the method can be executed by an electronic device (such as the training device described above). For example, as an example, the electronic device can be a server, a laptop, a smart terminal, or the like, though these are not specifically limited in this embodiment.

[0029] like Figure 2 As shown, the process may include the following steps: Step 201: In the current training cycle, determine the sparsity for sparsifying the weight parameters of the current first neural network.

[0030] In this embodiment, knowledge distillation is performed on a pre-trained neural network in conjunction with sparsification to obtain a target neural network for image processing. Specifically, during each training cycle, this embodiment first determines the sparsity used to sparsify the weight parameters of the current first neural network, then sparsifies the weight parameters of the current first neural network to obtain a second neural network, and then performs knowledge distillation on the second neural network.

[0031] In this embodiment, the sparsity determined in different training cycles is different. As an example, in the current training cycle, the sparsity used to sparsify the weight parameters of the current first neural network is determined. In a specific implementation, for example, the sparsity used to sparsify the weight parameters of the current first neural network is determined based on the cycle identifier of the current training cycle and a sparsity adjustment coefficient; wherein the sparsity adjustment coefficient is determined based on a preset sparsity and the total number of training cycles.

[0032] It can be seen that in this embodiment, in each training cycle, the weight parameters of the current first neural network are not directly sparsified according to a fixed preset sparsity, but the sparsity corresponding to the current training cycle is determined by relying on the cycle identifier of the current training cycle and the sparsity adjustment coefficient, that is, iterative sparsification (that is, dynamic sparsification) is gradually performed in each training cycle. This iterative sparsification method can gradually increase the output feature difference, that is, the knowledge difference, between the student network, that is, the second neural network, and the teacher network, that is, the pre-trained neural network in knowledge distillation in each training cycle. Compared with the situation where the weight parameters of the current first neural network are directly sparsified according to a fixed preset sparsity, which results in a large difference in output features and a sudden increase in the difficulty of student network learning, it can effectively reduce the difficulty of student network learning and improve the performance of the subsequent target neural network, that is, the trained neural network.

[0033] As for how to determine the sparsity for sparsifying the weight parameters of the current first neural network based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient, and how to determine the sparsity adjustment coefficient depending on the preset sparsity and the total number of training cycles, examples will be given below and will not be repeated here.

[0034] In this embodiment, as an embodiment, if the current training cycle is the first training cycle, the above-mentioned first neural network is a pre-trained neural network, that is, at this time a copy of the pre-trained neural network can be used as the first neural network; if the current training cycle is not the first training cycle, the above-mentioned first neural network is a network obtained by performing knowledge distillation on the pre-trained neural network in the historical training cycle.

[0035] Here, the pre-trained neural network can be considered to refer to the teacher network in the knowledge distillation process. It should be noted that the weight parameters of the pre-trained neural network remain fixed during the knowledge distillation process. Optionally, this embodiment does not specifically limit how to obtain the pre-trained neural network, and can be flexibly set based on actual application requirements. For example, the pre-trained neural network can be trained based on a pre-prepared training dataset. As for the specific method of training the pre-trained neural network based on the pre-prepared training dataset, this embodiment does not specifically limit it.

[0036] Step 202: Based on the sparsity, the weight parameters in the current first neural network are subjected to sparse processing to obtain a second neural network; and the network loss value is determined using each training sample and the second neural network.

[0037] In this embodiment, after determining the sparsity used to sparsify the weight parameters of the current first neural network, the weight parameters in the current first neural network are sparsified based on the sparsity to obtain a second neural network; thereafter, the network loss value is determined using each training sample and the second neural network for subsequent network optimization. In this embodiment, by coupling knowledge distillation and sparsification in each training cycle, an end-to-end training framework for dynamic collaborative optimization of knowledge distillation and sparsification is formed. This framework can not only compensate for the accuracy loss caused by sparsification to minimize the accuracy loss while achieving model compression, but also avoid the performance degradation caused by training sparsification and knowledge distillation in stages.

[0038] Optionally, the training data sets used in each training cycle in this embodiment may be the same or different, which is not specifically limited here.

[0039] As for how to perform sparse processing on the weight parameters in the current first neural network based on the above sparsity, and how to use each training sample and the second neural network to determine the network loss value, examples will be given below and will not be repeated here.

[0040] Step 203: Based on the gradient of each weight parameter of the network loss value in the first neural network, the weight parameters in the first neural network are optimized to obtain a third neural network; if the current training cycle is not the last training cycle, the third neural network is used as the first neural network in the next training cycle, and when the next training cycle is the current training cycle, the process returns to the step of determining the sparsity for sparsifying the weight parameters of the current first neural network; if the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsified using the preset sparsity to obtain a target neural network.

[0041] In this embodiment, after obtaining the network loss value used to optimize the weight parameters in the first neural network, the weight parameters in the first neural network are optimized based on the gradient of the network loss value of the weight parameters in the first neural network to obtain the third neural network.

[0042] Afterwards, it is checked whether the current training cycle is the last training cycle; if the current training cycle is not the last training cycle, the third neural network is used as the first neural network of the next training cycle, and when the next training cycle is the current training cycle, the process returns to the step of determining the sparsity for sparsifying the weight parameters of the current first neural network.

[0043] If the current training cycle is the last training cycle, training is complete. The weight parameters of the third neural network can be sparsified using a preset sparsity to obtain a target neural network. This target neural network can be used to perform image processing based on the input image data in an image processing scenario. As previously described, the image processing here may include, but is not limited to, image classification, object detection, image segmentation, pose estimation, and the like, and is not specifically limited in this embodiment.

[0044] As for how to optimize the weight parameters in the first neural network based on the gradients of the weight parameters of the first neural network based on the above-mentioned network loss value in this step, an example will be given below and it will not be described here.

[0045] So far, completed Figure 2 The process shown.

[0046] pass Figure 2It can be seen from the shown process that in the embodiment of the present application, by first using the currently determined sparsity to sparse the weight parameters in the current first neural network to obtain a second neural network in each training cycle of knowledge distillation, and then using each training sample and the second neural network to perform knowledge distillation, an end-to-end training framework for dynamic collaborative optimization of knowledge distillation and sparsification is formed by coupling knowledge distillation and sparsification during the training process. This can not only make up for the accuracy loss caused by sparsification, so as to minimize the accuracy loss on the basis of achieving model compression, but also avoid the performance degradation problem caused by training sparsification and knowledge distillation in stages.

[0047] Furthermore, in the embodiment of the present application, the neural network is not sparsified at one time according to the final sparsity, but is iteratively sparsified step by step in each training cycle. In this way, by gradually increasing the output feature difference, that is, the knowledge difference, between the student network, i.e., the second neural network, and the teacher network, i.e., the pre-trained neural network in knowledge distillation, it is possible to avoid the situation where the difficulty of learning the student network increases suddenly due to excessive output feature difference caused by the initial sparsity being too high, thereby effectively reducing the difficulty of learning the student network. Moreover, this embodiment also optimizes the neural network before sparsification, i.e., the first neural network, by utilizing the network loss value. In this way, it is possible to avoid the problem of decreased accuracy of the neural network due to the fact that some of the weight parameters that have been sparsified do not participate in the optimization, thereby enhancing the learning ability of the student network, so that the performance of the trained student network, i.e., the target neural network, is basically not reduced compared to the pre-trained neural network, i.e., the teacher network, thereby achieving the goal of minimizing accuracy loss while compressing the neural network using the sparsification method.

[0048] The following describes the determination of the sparsity for sparsifying the weight parameters of the current first neural network based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient in step 201: In this embodiment, as an embodiment, the above-mentioned determination of the sparsity for sparsifying the weight parameters of the current first neural network based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient can be specifically implemented as follows: the cycle identifier of the current training cycle and the sparsity adjustment coefficient are input as input parameters into the sparsity calculation formula shown below to obtain the corresponding sparsity: .

[0049] in, represents sparsity, represents the sparsity adjustment coefficient, Indicates the cycle identifier of the current training cycle.

[0050] Optionally, the cycle identifier here can be, for example, the cycle number. For example, if the current training cycle is the first training cycle (that is, the first training cycle), the cycle identifier of the current training cycle is 1; if the current training cycle is the second training cycle, the cycle identifier of the current training cycle is 2, and so on.

[0051] In this embodiment, as an embodiment, the sparsity adjustment coefficient is determined based on the preset sparsity and the total number of training cycles. In a specific implementation, for example, the preset sparsity and the total number of training cycles are input as input parameters into the following coefficient calculation formula to obtain the sparsity adjustment coefficient: .

[0052] in, Indicates the preset sparsity, Represents the total number of training epochs.

[0053] Optionally, the preset sparsity and total number of training cycles here can be flexibly set based on actual application requirements. For example, the preset sparsity can be 0.5 or 0.6, and the total number of training cycles can be 10 or 20, etc. This embodiment does not specifically limit this.

[0054] The following describes the sparsification process of the weight parameters in the current first neural network based on the sparsity in step 202: In this embodiment, the sparsification processing of the weight parameters in the current first neural network based on the sparsity can be implemented in many ways: For example, as an embodiment, first, the weight parameters in the first neural network are sorted in order to obtain a weight sequence, where the order can be, for example, from large to small.

[0055] Afterwards, the N weight parameters ranked last in the weight sequence are set to the specified values, and the current values ​​of the remaining weight parameters are maintained unchanged. The specified value here can be 0, for example.

[0056] In this embodiment, N may be determined based on the total number of weight parameters in the weight sequence and the sparsity determined in the current training cycle. Specifically, as an embodiment, N may be determined as the product of the total number of weight parameters in the weight sequence and the sparsity determined in the current training cycle.

[0057] For example, assuming that the total number of weight parameters in the first neural network is 100 and the sparsity determined in the current training cycle is 0.2, then N=100*0.2=20.

[0058] For another example, as another embodiment, structured sparse processing can be performed on each weight parameter in the first neural network based on sparsity. Specifically, during implementation, each weight parameter in the first neural network is represented in the form of a weight matrix. Based on this, for each row of weight parameters in the weight matrix, the last N weight parameters in the row of weight parameters are set to specified values, and the current values ​​of the remaining weight parameters in the row of weight parameters are maintained unchanged. Thereafter, for each column of weight parameters in the weight matrix, the last N weight parameters in the column of weight parameters are set to specified values, and the current values ​​of the remaining weight parameters in the column of weight parameters are maintained unchanged.

[0059] For another example, semi-structured sparse processing can be performed on each weight parameter in the first neural network based on sparsity. Specifically, during implementation, for each row of weight parameters in the weight matrix, the last N weight parameters in the row can be set to specified values, while the current values ​​of the remaining weight parameters in the row are maintained unchanged.

[0060] As for how to use the preset sparsity to perform sparse processing on the weight parameters of the third neural network, it is similar to the sparse processing of the weight parameters in the current first neural network based on the above sparsity. Please refer to the above related description and will not be repeated here.

[0061] The following describes the determination of the network loss value using each training sample and the second neural network in step 202: As an example, see Figure 3 As shown, the above-mentioned determination of the network loss value using each training sample and the second neural network may include the following steps in specific implementation, for example: Step 301: Determine a feature-based distillation loss value based on each training sample and the second neural network; wherein the feature-based distillation loss value is used to indicate the difference in output features of the same layer between the second neural network and the pre-trained neural network.

[0062] In this embodiment, as an embodiment, in this step, a feature-based distillation loss value is determined based on each training sample and the second neural network. In a specific implementation, for example, it can be: first input each training sample into the second neural network to obtain the first sample feature corresponding to each training sample output by the specified intermediate layer of the second neural network; and, input each training sample into the pre-trained neural network to obtain the second sample feature corresponding to each training sample output by the specified intermediate layer of the pre-trained neural network; then, the first sample feature and the second sample feature corresponding to each training sample are input as input parameters to a preset first loss function to obtain a first loss value, and based on the first loss value, a feature-based distillation loss value is determined.

[0063] In this embodiment, the intermediate layer refers to a layer between the input layer and the output layer in a neural network, and can be used to perform feature extraction and other processing on the input data. The designated intermediate layer in this embodiment can be flexibly set based on actual application requirements, for example, it can be any intermediate layer in the neural network, and is not specifically limited here.

[0064] Optionally, this embodiment does not specifically limit the first loss function, and it can be flexibly set based on actual needs, for example, it can be a KL (Kullback-Leibler) divergence loss function or a mean square error loss function.

[0065] In this embodiment, as an embodiment, the above-mentioned determination of the feature-based distillation loss value based on the first loss value may be specifically implemented as follows: the first loss value may be determined as the feature-based distillation loss value.

[0066] Step 302: Determine a soft label distillation loss value using each training sample and the second neural network; wherein the soft label distillation loss value is used to indicate the difference in output results between the second neural network and the pre-trained neural network.

[0067] In this embodiment, as an embodiment, in this step, each training sample and the second neural network are used to determine the soft label distillation loss value. In the specific implementation, for example, it can be: first input each training sample into the second neural network to obtain the first output result corresponding to each training sample output by the output layer of the second neural network; and input each training sample into the pre-trained neural network to obtain the second output result corresponding to each training sample output by the output layer of the pre-trained neural network; then, the first output result and the second output result corresponding to each training sample are input as input parameters into the preset second loss function to obtain the second loss value, and the soft label distillation loss value is determined based on the second loss value.

[0068] Optionally, this embodiment does not specifically limit the second loss function, and it can be flexibly set based on actual needs, for example, it can be a cross entropy loss function or an L1 norm loss function.

[0069] In this embodiment, as an embodiment, the soft label distillation loss value is determined based on the second loss value. In a specific implementation, for example, the second loss value may be determined as the soft label distillation loss value.

[0070] Step 303: Determine a network loss value based on the feature-based distillation loss value and the soft label distillation loss value.

[0071] In this embodiment, there are many ways to implement the network loss value in this step based on the feature-based distillation loss value and the soft label distillation loss value. For example, as one embodiment, the average of the feature-based distillation loss value and the soft label distillation loss value can be used as the network loss value. For another example, as another embodiment, the weighted average of the feature-based distillation loss value and the soft label distillation loss value can be used as the network loss value.

[0072] This completes the Figure 3 Description of the method flow shown.

[0073] The following describes the optimization of the weight parameters in the first neural network based on the gradients of the weight parameters in the first neural network based on the network loss value in step 203: In this embodiment, as an embodiment, the above-mentioned optimization of the weight parameters in the first neural network based on the gradient of each weight parameter in the first neural network based on the above-mentioned network loss value may include, for example, the following when specifically implemented: First, for each weight parameter, the weight parameter and are input as input parameters into the preset gradient calculation function to obtain the gradient of the network loss value at the weight parameter.

[0074] Optionally, this embodiment does not specifically limit the gradient calculation function, and can be flexibly set based on actual application requirements, for example, it can be: gradient ,in, represents the first loss function, represents the first loss function, represents the weight parameter, express Relative to The partial derivative results of express Relative to The partial derivative results of .

[0075] Afterwards, the gradient of the network loss value at the weight parameter is input as an input parameter to the weight optimization function to obtain the optimized weight parameter.

[0076] Optionally, this embodiment does not specifically limit the weight optimization function, and can also be flexibly set based on actual application requirements. For example, it can be: the optimized weight parameter = the weight parameter before optimization - a*the gradient of the network loss value at the weight parameter, where Indicates optimization parameters, which can be flexibly set based on actual needs.

[0077] In order to facilitate understanding of the specific implementation process of the above-mentioned weight dynamic optimization method, a specific embodiment is described below by way of example.

[0078] As an embodiment, the specific implementation process of the weight dynamic optimization method provided in this embodiment is as follows: Step 1: Train a pre-trained neural network using a pre-prepared training dataset.

[0079] Step 2: In the current training cycle, based on the cycle identifier of the current training cycle and the sparsity adjustment coefficient, determine the sparsity for sparsifying the weight parameters of the current first neural network, and based on the sparsity, perform sparsification processing on the weight parameters in the current first neural network to obtain the second neural network.

[0080] Step 3: Determine the network loss value using the training samples in the current training cycle and the second neural network, and use the network loss value to perform back propagation to optimize the weight parameters in the first neural network to obtain the third neural network.

[0081] Specifically, see Figure 4 As shown, each training sample is input into the first neural network to obtain the first sample features corresponding to each training sample output by the designated intermediate layer of the second neural network, and the first output results corresponding to each training sample output by the output layer of the second neural network.

[0082] Each training sample is input into the first neural network and the pre-trained neural network to obtain a second sample feature corresponding to each training sample output by the designated intermediate layer of the second neural network, and a second output result corresponding to each training sample output by the output layer of the pre-trained neural network.

[0083] Then, a feature-based distillation loss value is obtained based on the first sample feature and the second sample feature corresponding to each training sample, and a soft-label distillation loss value is obtained based on the first output result and the second output result corresponding to each training sample. A network loss value is determined based on the feature-based distillation loss value and the soft-label distillation loss value, and backpropagation is performed using the network loss value to optimize the weight parameters in the first neural network.

[0084] Optionally, in this embodiment, a straight-through estimator (STE) method can be used to directly estimate the gradient of the network loss value at each weight parameter, thereby optimizing each weight parameter based on the gradient of the network loss value at each weight parameter. This embodiment does not specifically limit how to directly estimate the gradient of the network loss value at each weight parameter using the STE method.

[0085] Step 4: If the current training cycle is not the last training cycle, the third neural network is used as the first neural network of the next training cycle, and when the next training cycle is the current training cycle, the process returns to step 2 above. If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsely processed using a preset sparsity to obtain a target neural network, which is used for image processing.

[0086] This completes the description of the method provided in the embodiment of the present application. The following describes the device provided in the embodiment of the present application: As an embodiment, this embodiment also provides a weight dynamic optimization device. For example, see Figure 5 , Figure 5 This is a structural diagram of a weight dynamic optimization device provided in an embodiment of the present application. Figure 2 As shown in the process. Figure 5 As shown, the weight dynamic optimization device 500 can be applied to at least an image processing scenario, which can be executed by an electronic device. The weight dynamic optimization device 500 includes: a determination module 501, a sparse module 502, an optimization module 503, and a training module 504; Determination module 501 is configured to determine, in a current training cycle, a sparsity for sparsifying the weight parameters of the current first neural network; the sparsity determined in different training cycles is different, and if the current training cycle is the first training cycle, the first neural network is a pre-trained neural network; if the current training cycle is not the first training cycle, the first neural network is a network obtained by performing knowledge distillation on the pre-trained neural network in a previous training cycle; A sparse module 502 is configured to perform sparse processing on the weight parameters in the current first neural network based on the sparsity to obtain a second neural network; and determine a network loss value using each training sample and the second neural network; An optimization module 503 is configured to optimize the weight parameters in the first neural network based on the gradient of each weight parameter of the network loss value in the first neural network to obtain a third neural network; A training module 504 is configured to use the third neural network as the first neural network of the next training cycle if the current training cycle is not the last training cycle, and return to the step of determining the sparsity for sparsifying the weight parameters of the current first neural network when the next training cycle is the current training cycle; If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsely processed using a preset sparsity to obtain a target neural network, which is used to perform image processing based on the input image data to be processed in the image processing scenario.

[0087] As an embodiment, determining the sparsity for sparsifying the weight parameters of the current first neural network includes: Determining the sparsity based on a cycle identifier of a current training cycle and a sparsity adjustment coefficient; The sparsity adjustment coefficient is determined based on the preset sparsity and the total number of training cycles.

[0088] As an embodiment, the performing sparse processing on the weight parameters in the current first neural network based on the sparsity includes: Sorting the weight parameters in the first neural network in order to obtain a weight sequence; The N weight parameters that are ranked last in the weight sequence are set to specified values, and the current values ​​of the remaining weight parameters are maintained unchanged; wherein N is determined based on the total number of weight parameters in the weight sequence and the sparsity.

[0089] As an embodiment, determining the network loss value using each training sample and the second neural network includes: Determining a feature-based distillation loss value based on each training sample and the second neural network, wherein the feature-based distillation loss value is used to indicate a difference in output features of the same layer in the second neural network and the pre-trained neural network; Determining a soft label distillation loss value using each training sample and the second neural network, wherein the soft label distillation loss value is used to indicate a difference in output results between the second neural network and the pre-trained neural network; The network loss value is determined based on the feature-based distillation loss value and the soft label distillation loss value.

[0090] As an embodiment, determining the feature-based distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Inputting each training sample into the second neural network to obtain a first sample feature corresponding to each training sample output by a designated intermediate layer of the second neural network; Inputting each training sample into the pre-trained neural network to obtain a second sample feature corresponding to each training sample output by the designated intermediate layer of the pre-trained neural network; The first sample feature and the second sample feature corresponding to each training sample are input as input parameters to a preset first loss function to obtain a first loss value, and the feature-based distillation loss value is determined based on the first loss value.

[0091] In one embodiment, determining the soft label distillation loss value using the training samples, the second neural network, and the pre-trained neural network includes: Inputting each training sample into the second neural network to obtain a first output result corresponding to each training sample output by the output layer of the second neural network; Inputting each training sample into the pre-trained neural network to obtain a second output result corresponding to each training sample output by an output layer of the pre-trained neural network; The first output result and the second output result corresponding to each training sample are input as input parameters to a preset second loss function to obtain a second loss value, and the soft label distillation loss value is determined based on the second loss value.

[0092] So far, completed Figure 5 Structural description of the device shown.

[0093] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0094] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application scheme. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0095] See Figure 6 , is a schematic diagram of the hardware structure of an electronic device provided in an exemplary embodiment of the present application. The electronic device includes a processor and a computer-readable storage medium; the computer-readable storage medium stores a number of computer program instructions, which, when executed by the processor, implement the method disclosed in the above example of the present application. The electronic device may also include other hardware depending on its actual function, which will not be described in detail here.

[0096] Based on the same application concept as the above method, an embodiment of the present application also provides a computer-readable storage medium, on which a number of computer program instructions are stored. When the computer program instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.

[0097] Illustratively, the computer-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, messages, and the like. For example, the computer-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0098] The above are merely preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A weight dynamic optimization method, characterized in that: The method is at least applied to an image processing scenario, which includes at least an image classification scenario, a target detection scenario, an image segmentation scenario, and a posture estimation scenario, and the method includes: In a current training cycle, determining a sparsity for sparsifying weight parameters of a current first neural network; the sparsity determined in different training cycles is different, and if the current training cycle is the first training cycle, the first neural network is a pre-trained neural network; if the current training cycle is not the first training cycle, the first neural network is a network obtained by performing knowledge distillation on the pre-trained neural network in a previous training cycle; Based on the sparsity, the weight parameters in the current first neural network are subjected to sparse processing to obtain a second neural network; and a network loss value is determined using each training sample and the second neural network; Optimizing the weight parameters in the first neural network based on the gradient of each weight parameter of the network loss value in the first neural network to obtain a third neural network; If the current training cycle is not the last training cycle, using the third neural network as the first neural network of the next training cycle, and returning to the step of determining the sparsity for sparsifying the weight parameters of the current first neural network when the next training cycle is the current training cycle; If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsely processed using a preset sparsity to obtain a target neural network, which is used to perform image processing based on the input image data to be processed in the image processing scenario.

2. The method according to claim 1, characterized in that Determining the sparsity for sparsifying the weight parameters of the current first neural network includes: Determining the sparsity based on a cycle identifier of a current training cycle and a sparsity adjustment coefficient; The sparsity adjustment coefficient is determined based on the preset sparsity and the total number of training cycles.

3. The method according to claim 1, characterized in that The performing sparse processing on the weight parameters in the current first neural network based on the sparsity includes: Sorting the weight parameters in the first neural network in order to obtain a weight sequence; The N weight parameters that are ranked last in the weight sequence are set to specified values, and the current values ​​of the remaining weight parameters are maintained unchanged; wherein N is determined based on the total number of weight parameters in the weight sequence and the sparsity.

4. The method according to claim 1, wherein Determining the network loss value by using each training sample and the second neural network includes: Determining a feature-based distillation loss value based on each training sample and the second neural network, wherein the feature-based distillation loss value is used to indicate a difference in output features of the same layer in the second neural network and the pre-trained neural network; Determining a soft label distillation loss value using each training sample and the second neural network, wherein the soft label distillation loss value is used to indicate a difference in output results between the second neural network and the pre-trained neural network; The network loss value is determined based on the feature-based distillation loss value and the soft label distillation loss value.

5. The method according to claim 4, characterized in that Determining the feature-based distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Inputting each training sample into the second neural network to obtain a first sample feature corresponding to each training sample output by a designated intermediate layer of the second neural network; Inputting each training sample into the pre-trained neural network to obtain a second sample feature corresponding to each training sample output by the designated intermediate layer of the pre-trained neural network; The first sample feature and the second sample feature corresponding to each training sample are input as input parameters to a preset first loss function to obtain a first loss value, and the feature-based distillation loss value is determined based on the first loss value.

6. The method according to claim 4, characterized in that Determining the soft label distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Inputting each training sample into the second neural network to obtain a first output result corresponding to each training sample output by the output layer of the second neural network; Inputting each training sample into the pre-trained neural network to obtain a second output result corresponding to each training sample output by an output layer of the pre-trained neural network; The first output result and the second output result corresponding to each training sample are input as input parameters to a preset second loss function to obtain a second loss value, and the soft label distillation loss value is determined based on the second loss value.

7. A weight dynamic optimization device, characterized in that: The device is at least applied to an image processing scenario, which includes at least an image classification scenario, a target detection scenario, an image segmentation scenario, and a posture estimation scenario, and includes: a determination module for determining, in a current training cycle, a sparsity for sparsifying weight parameters of a current first neural network; the sparsity determined in different training cycles being different, and if the current training cycle is the first training cycle, the first neural network being a pre-trained neural network; and if the current training cycle is not the first training cycle, the first neural network being a network obtained by performing knowledge distillation on the pre-trained neural network in a previous training cycle; A sparse module is used to perform sparse processing on the weight parameters in the current first neural network based on the sparsity to obtain a second neural network; and determine a network loss value using each training sample and the second neural network; an optimization module, configured to optimize the weight parameters in the first neural network based on the gradient of each weight parameter of the network loss value in the first neural network to obtain a third neural network; a training module configured to use the third neural network as the first neural network of a next training cycle if the current training cycle is not the last training cycle, and return to the step of determining a sparsity for sparsifying the weight parameters of the current first neural network when the next training cycle is the current training cycle; If the current training cycle is the last training cycle, the weight parameters of the third neural network are sparsely processed using a preset sparsity to obtain a target neural network, which is used to perform image processing based on the input image data to be processed in the image processing scenario.

8. The device according to claim 7, characterized in that Determining the sparsity for sparsifying the weight parameters of the current first neural network includes: Determining the sparsity based on a cycle identifier of a current training cycle and a sparsity adjustment coefficient; Wherein, the sparsity adjustment coefficient is determined based on the preset sparsity and the total number of training cycles; and / or, The performing sparse processing on the weight parameters in the current first neural network based on the sparsity includes: Sorting the weight parameters in the first neural network in order to obtain a weight sequence; Setting the last N weight parameters in the weight sequence to specified values ​​and maintaining the current values ​​of the remaining weight parameters unchanged; wherein N depends on the total number of weight parameters in the weight sequence and the sparsity; and / or, Determining the network loss value by using each training sample and the second neural network includes: Determining a feature-based distillation loss value based on each training sample and the second neural network, wherein the feature-based distillation loss value is used to indicate a difference in output features of the same layer in the second neural network and the pre-trained neural network; Determining a soft label distillation loss value using each training sample and the second neural network, wherein the soft label distillation loss value is used to indicate a difference in output results between the second neural network and the pre-trained neural network; Determining the network loss value based on the feature-based distillation loss value and the soft label distillation loss value; and / or, Determining the feature-based distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Inputting each training sample into the second neural network to obtain a first sample feature corresponding to each training sample output by a designated intermediate layer of the second neural network; Inputting each training sample into the pre-trained neural network to obtain a second sample feature corresponding to each training sample output by the designated intermediate layer of the pre-trained neural network; Inputting the first sample feature and the second sample feature corresponding to each training sample as input parameters into a preset first loss function to obtain a first loss value, and determining the feature-based distillation loss value based on the first loss value; and / or, Determining the soft label distillation loss value using each training sample, the second neural network, and the pre-trained neural network includes: Inputting each training sample into the second neural network to obtain a first output result corresponding to each training sample output by the output layer of the second neural network; Inputting each training sample into the pre-trained neural network to obtain a second output result corresponding to each training sample output by an output layer of the pre-trained neural network; The first output result and the second output result corresponding to each training sample are input as input parameters to a preset second loss function to obtain a second loss value, and the soft label distillation loss value is determined based on the second loss value.

9. An electronic device, characterized in that: The electronic device includes: processor; and A computer-readable storage medium having computer program instructions stored therein, wherein the computer program instructions, when executed by the processor, cause the processor to perform the steps of any one of the methods of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the method according to any one of claims 1 to 6.

Images