A CNN-Resnet model training method and device applied to a tension control system and a medium

Abstract

The application provides a CNN-Resnet model training method applied to a tension control system, sample data is input into a teacher model which is trained, first soft label data is obtained through multiple convolution operations; the sample data is input into a student model of a teacher-student model, second soft label data is obtained through multiple convolutions, and teacher-student loss is calculated according to the first soft label data and the second soft label data; student model loss is calculated according to the difference between the second soft label data and corresponding hard label data in the sample data; the teacher-student loss and the student model loss are used as total loss of knowledge distillation, the student model is subjected to knowledge distillation, and training of the student model is completed. The size of the CNN-Resnet model can be compressed without affecting the model accuracy, and a small model CNN-Resnet network which can be used for the tension control system is obtained.

Description

Technical Field

[0001] This invention relates to the field of tension control systems, and in particular to a CNN-ResNet model training method applied to tension control systems. Background Technology

[0002] Tension control systems are often an integration of tension sensors and tension controllers. They are mainly used in equipment in metallurgy, papermaking, film production, dyeing and finishing, weaving, plastics, and wire manufacturing. They are automatic control systems that achieve constant tension or tapered tension control. Their main function is to achieve synchronization between rollers and uniform control of winding and unwinding.

[0003] The standard definition of a tension control system is an algorithm used in mechanical pendulum control to stabilize the position of the pendulum at a fixed point. In tension control algorithms, due to the temporal nature of the correction system, CNN-LSTM recurrent sequence neural networks are typically used for high-level feature extraction of tension environmental variables. However, existing CNN-LSTM models are too large, consuming significant computational resources and resulting in wasted resources. Summary of the Invention

[0004] Based on this, the purpose of this invention is to provide a CNN-ResNet model training method for tension control systems. By introducing a teacher-student model and training the teacher-student model through knowledge distillation, the size of the CNN-ResNet model can be compressed without affecting the model accuracy, resulting in a small CNN-ResNet network that can be used in tension control systems.

[0005] This invention is achieved through the following technical solution:

[0006] On one hand, this invention provides a CNN-ResNet model training method for tension control systems, comprising:

[0007] The sample data is input into a trained teacher model, and the first soft label data is obtained through multiple convolution operations.

[0008] The sample data is input into the student model of the teacher-student model, and the second soft label data is obtained through multiple convolutions. The teacher-student loss is calculated based on the first soft label data and the second soft label data.

[0009] The student model loss is calculated based on the difference between the second soft-label data and the corresponding hard-label data in the sample data;

[0010] The student model is then subjected to knowledge distillation based on the teacher-student loss and the student model loss as the total loss for knowledge distillation, thus completing the training of the student model.

[0011] Furthermore, the formula for calculating the total loss of the knowledge distillation is optimized as follows:

[0012]

[0013] in This is the sum of the teacher-student loss and the student model loss. The average weight of the Student model. This represents the average weight of the Teacher model.

[0014] Furthermore, the total loss of the knowledge distillation also includes an adjustable parameter A, and the specific formula for calculating the total loss is optimized as follows:

[0015]

[0016] in This is the sum of the teacher-student loss and the student model loss. The average weight of the Student model. Let A be the mean weight of the Teacher model, and A be an adjustable parameter.

[0017] For different error correction-related variable factors of the tension system, the value of the adjustable parameter can be adjusted so that the same teacher-student network model can be adapted to model training with different variable factors.

[0018] Furthermore, before obtaining the first soft label data through multiple convolution operations, the following steps are also included:

[0019] Global pooling is performed on the data after multiple convolutions. This allows for global abstraction of the high-level features extracted by the upper convolutional layers.

[0020] Furthermore, the sample data includes any one of the following: belt offset, belt speed, and angle.

[0021] On the other hand, this application also provides a CNN-ResNet model training device for use in tension control systems, which includes:

[0022] Teacher layer module: used to input sample data into a trained teacher model and obtain the first soft label data through multiple convolution operations;

[0023] Student layer module: used to input the sample data into the student model of the teacher-student model, obtain the second soft label data through multiple convolutions, and calculate the teacher-student loss based on the first soft label data and the second soft label data;

[0024] The student loss calculation module is used to calculate the student model loss based on the difference between the second soft-label data and the corresponding hard-label data in the sample data.

[0025] Knowledge distillation module: Used to perform knowledge distillation on the student model based on the teacher-student loss and the student model loss as the total loss for knowledge distillation, thereby completing the training of the student model.

[0026] Furthermore, the teacher layer module also includes:

[0027] Pooling submodule: Used to perform global pooling on data after multiple convolutions.

[0028] On the other hand, this application also provides a storage medium storing a computer program that, when executed by a processor, implements the steps of a CNN-ResNet model training method for a tension control system as described in any of the preceding claims.

[0029] This invention discloses a CNN-ResNet model training method applied to tension control systems. In the tension control system, a teacher-student model is introduced. Knowledge distillation is used to train the teacher-student model, utilizing the knowledge learned by the larger model to guide the training of the smaller model. This allows the smaller model to achieve performance comparable to the larger model, but with a significantly reduced number of parameters, thus compressing and accelerating the CNN-ResNet model. Furthermore, the total loss function of knowledge distillation is optimized. It fully considers the situation where a large difference in weights between the student and teacher models in the tension system model can lead to difficulty in model convergence during training. Therefore, the difference in the mean weights of the student and teacher models is considered in the total loss function, enabling model training to converge. An adjustable parameter is further introduced to handle different correction-related variable factors in the tension system. By adjusting the value of this adjustable parameter, the same teacher-student network model can be adapted to model training with different variable factors.

[0030] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0031] Figure 1 A flowchart of a CNN-ResNet model training method applied to a tension control system provided in this application;

[0032] Figure 2 A structural block diagram of a CNN-ResNet model training device for use in a tension control system is provided in this application;

[0033] Figure 3 A structural diagram of an exemplary teacher model provided for this application.

[0034] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings of the embodiments of this application. Detailed Implementation

[0035] Please see Figure 1 and Figure 2 , Figure 1 A flowchart of a CNN-ResNet model training method applied to a tension control system provided in this application; Figure 2 This application provides a structural block diagram of a CNN-ResNet model training device for use in a tension control system; this application also provides an implementation method for training a CNN-ResNet model for use in a tension control system, comprising:

[0036] S10: Input the sample data into a trained teacher model and obtain the first soft label data through multiple convolution operations. Step S10 is executed by the teacher layer module 10.

[0037] The CNN-ResNet model trained in this application is mainly used in tension control systems. Therefore, the sample data consists of relevant variable factors across multiple data channels, including offset, velocity, and angle. The training set is collected from these data channels. Figure 3 The teacher model consists of a convolutional neural network, designed based on the relationship between variable factors in the input path and the tension control results. It includes an input layer, convolutional layers, multiple pooling layers, and four densely connected residual modules. Image data is received through the input layer, followed by a 7x7 convolutional layer. After the first convolutional layer, pooling layers are used to extract more abstract features. Following the first pooling layer are four densely connected residual modules, each enhancing information flow between layers by connecting them pairwise. The first densely connected residual module has seven 3x3 convolutional layers. The other three have similar structures, with 12, 24, and 16 3x3 convolutional layers respectively. A global pooling layer then abstracts the high-level features extracted by the upper convolutional layers globally. Its function is similar to the global abstraction of features learned autonomously by fully connected layers; however, it cannot extract local details. Nevertheless, the global pooling layer saves a significant amount of training parameters, thus reducing the model size. Finally, the Softmax function of the output layer outputs a probability matrix for each category. The category index corresponding to the number with the highest probability is the final calculated category result.

[0038] Taking the offset of the path as an example, the sample data of the offset is input into the teacher model for training. The teacher model is continuously trained according to the loss function of the teacher model to obtain the trained teacher model. Then, multiple sample data are selected from the test set. The data is input into the teacher model to obtain the teacher model's first soft label for each sample. .

[0039] S20: The sample data is input into the student model of the teacher-student model, and the second soft-label data is obtained through multiple convolutions. The teacher-student loss is then calculated based on the first soft-label data and the second soft-label data. Step S20 is executed by the student layer module 20.

[0040] Our student models are typically shallow ResNets; here we use ResNet-18.

[0041] The basic meaning of ResNet18 is that the basic architecture of the network is ResNet, and the network depth is 18 layers. However, the network depth here refers to the weight layers of the network, that is, including pooling, activation, and linear layers, but not the batch normalization layer and pooling layer. The student model includes, in order:

[0042] 7 The convolutional layer has 7 convolutional kernels with a kernel size of 77, a stride of 2, padding of 3, and 64 output channels.

[0043] The pooling layer has a kernel size of 33, a stride of 2, and padding of 1. The final output data size is 64. 56 56, which means that this pooling does not change the number of data channels, but halves the data size.

[0044] 3 Three convolutional layers, each with a kernel size of 33, a stride of 1, and padding of 1. The final output data from the first two convolutional layers is 64 bytes. 54 54 means that this layer does not change the data size or the number of channels.

[0045] 3 The third convolutional layer passes through another convolutional layer with a kernel size of 11, followed by downsampling. The final output data is 128. 28 28. This means doubling the number of output channels and halving the size of all output data.

[0046] Then, pass it through another convolutional layer with a kernel size of 11 and downsample. The resulting output is 256. 14 14. This means doubling the number of output channels and halving the size of all output data.

[0047] Similarly, it is passed through a convolutional layer with a kernel size of 11 and downsampled again. The final output is 512. 7 7. This means doubling the number of output channels and halving the size of all output data. The input data is then processed by the student model described above to obtain the prediction results.

[0048] S30: Calculate the student model loss based on the difference between the second soft-label data and the corresponding hard-label data in the sample data. Step S30 is executed by the student loss calculation module 30.

[0049] Taking the offset of the path as an example, the same samples as those input into the teacher model are selected from the sample data. The data is then input into the student model to obtain the student model's second soft label for each sample. The difference between the corresponding samples of the second soft label and the first soft label is calculated to obtain the teacher-student model loss. This teacher-student model loss guides the student model to learn the prior knowledge of the pre-trained teacher model. Similarly, the loss between the student model and the true value, i.e., the student model loss, is also calculated based on the difference between the true offset of the sample data and the second soft label data.

[0050] S40: Using the teacher-student loss and student model loss as the total loss for knowledge distillation, perform knowledge distillation on the student model to complete the training of the student model. Step S40 is executed by the knowledge distillation module 40.

[0051] The student model is optimized using teacher-student loss and student model loss, continuously transferring knowledge from the teacher model to the student model to improve its performance and generalization ability. Training the student model using second soft-labeled data and the teacher model's first soft-labeled data allows it to better learn the teacher model's knowledge. Fine-tuning the student model with sample data allows it to better adapt to the characteristics of the sample data. Through knowledge distillation, a student model with high accuracy and a suitable size is obtained, which can be used for offset control in tension control systems.

[0052] In another preferred embodiment, the inventors discovered that during knowledge distillation, some weights in the student model differ significantly from those in the teacher model, thus affecting the accuracy of the student model during training and making it difficult for the model to converge. Based on this, the inventors further optimized the loss function for knowledge distillation. The optimized loss function for knowledge distillation is as follows:

[0053]

[0054] in The loss function for existing knowledge distillation, The average weight of the Student model. Let A be the mean weight of the Teacher model, and A be an adjustable parameter.

[0055] This application provides an example of how to obtain a small CNN-ResNet model for offset control applied in a tension control system, using the offset of the band as an example. However, in the tension control system, there are also multiple related variable factors in the band, such as velocity and angle. For each variable factor, the convergence is not consistent. Therefore, an adjustable parameter A is introduced. For different variable factors, the convergence condition of the CNN-ResNet model is changed accordingly, thereby improving the universality of this training method.

[0056] This invention discloses a CNN-ResNet model training method applied to tension control systems. In the tension control system, a teacher-student model is introduced. Knowledge distillation is used to train the teacher-student model, utilizing the knowledge learned by the larger model to guide the training of the smaller model. This allows the smaller model to achieve performance comparable to the larger model, but with a significantly reduced number of parameters, thus compressing and accelerating the CNN-ResNet model. Furthermore, the total loss function of knowledge distillation is optimized. It fully considers the situation where a large difference in weights between the student and teacher models in the tension system model can lead to difficulty in model convergence during training. Therefore, the difference in the mean weights of the student and teacher models is considered in the total loss function, enabling model training to converge. An adjustable parameter is further introduced to handle different correction-related variable factors in the tension system. By adjusting the value of this adjustable parameter, the same teacher-student network model can be adapted to model training with different variable factors.

[0057] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a CNN-ResNet model training method for tension control systems as described in any of the above embodiments.

[0058] This invention can take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0059] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A CNN-ResNet model training method applied to tension control systems, characterized in that, include: The sample data is input into a trained teacher model, and the first soft label data is obtained through multiple convolution operations; wherein, the teacher model is a convolutional neural network; The sample data is input into the student model of the teacher-student model, and the second soft label data is obtained through multiple convolutions. The teacher-student loss is calculated based on the first soft label data and the second soft label data. The sample data is any one of the following: path offset, path velocity, and angle. The student model is a ResNet model. The student model loss is calculated based on the difference between the second soft-label data and the corresponding hard-label data in the sample data; The student model is then subjected to knowledge distillation using the teacher-student loss, student model loss, mean weights of the student model, and mean weights of the teacher model as the total loss for knowledge distillation, thus completing the training of the student model. The formula for calculating the total loss of knowledge distillation is optimized as follows: in This is the sum of the teacher-student loss and the student model loss. The average weight of the Student model. This represents the average weight of the Teacher model.

2. The CNN-ResNet model training method for tension control systems according to claim 1, characterized in that, The total loss of the knowledge distillation also includes an adjustable parameter A, and the optimized formula for calculating the total loss is as follows: in This is the sum of the teacher-student loss and the student model loss. The average weight of the Student model. Let A be the mean weight of the Teacher model, and A be an adjustable parameter.

3. The CNN-ResNet model training method for tension control systems according to claim 2, characterized in that, Before obtaining the first soft label data through multiple convolution operations, the following steps are also included: Perform global pooling on the data after multiple convolutions.

4. A CNN-ResNet model training device for use in tension control systems, characterized in that, include: Teacher layer module: used to input sample data into a trained teacher model and obtain the first soft label data through multiple convolution operations; wherein, the teacher model is a convolutional neural network; Student layer module: used to input the sample data into the student model of the teacher-student model, obtain the second soft label data through multiple convolutions, and calculate the teacher-student loss based on the first soft label data and the second soft label data; Student loss calculation module: used to calculate the student model loss based on the difference between the second soft-label data and the corresponding hard-label data in the sample data; wherein, the sample data is any one of: path offset, path velocity, and angle; the student model is a ResNet model; The knowledge distillation module is used to perform knowledge distillation on the student model based on the teacher-student loss, the student model loss, the average weight of the student model, and the average weight of the teacher model as the total loss for knowledge distillation, thereby completing the training of the student model. The formula for calculating the total loss of knowledge distillation is optimized as follows: in This is the sum of the teacher-student loss and the student model loss. The average weight of the Student model. This represents the average weight of the Teacher model.

5. A CNN-ResNet model training device for tension control systems according to claim 4, characterized in that, The teacher layer module includes: Pooling submodule: Used to perform global pooling on data after multiple convolutions.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of a CNN-Resnet model training method for tension control systems as described in any one of claims 1 to 3.

Images