A knowledge distillation hot-start training method and system for lightweight networks

Abstract

The present invention provides a knowledge distillation hot start training method and system for lightweight networks, which includes obtaining at least one randomly initialized lightweight network model; taking a low-resolution image and a corresponding high-resolution image as a pair of training samples, and obtaining a training sample set based on multiple training samples; inputting the low-resolution image in the training sample set into the lightweight network model, training the lightweight network model, and obtaining a first pre-trained network model; loading the weights of the first pre-trained network model into the lightweight network model, and keeping the various training parameters unchanged, performing one or more rounds of training on the lightweight network model based on the training sample set, and obtaining a second pre-trained network model; loading the weights of the second pre-trained network model into the lightweight network model, and training the lightweight network model based on the supervision of a teacher network, and obtaining a third pre-trained network model.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a knowledge distillation hot start training method and system for lightweight networks. Background Art

[0002] Image super-resolution technology generates high-resolution images from corresponding low-resolution images. Lightweight image super-resolution models have high computational efficiency and a small number of parameters, making them widely used in image super-resolution. However, they also suffer from issues such as insufficient image super-resolution performance. Researchers typically use methods such as network pruning, network quantization, and knowledge distillation to improve the performance of lightweight image super-resolution models based on the supervision of a teacher network. However, differences in the representation spaces between the teacher and student networks hinder the training of the student network, resulting in poor training results.

[0003] Therefore, in order to solve the above problems, it is hoped to propose a knowledge distillation hot start training method and system for lightweight networks. Summary of the Invention

[0004] One or more embodiments of the present specification provide a knowledge distillation hot start training method for lightweight networks. The knowledge distillation hot start training method for lightweight networks includes: obtaining at least one randomly initialized lightweight network model; taking a low-resolution image and a corresponding high-resolution image as a pair of training samples, and obtaining a training sample set based on multiple training samples; inputting the low-resolution image in the training sample set into the lightweight network model, training the lightweight network model, obtaining a first pre-trained network model, and adjusting the parameters of the lightweight network model based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold; loading the weights of the first pre-trained network model into the lightweight network model, and keeping the training parameters. The number remains unchanged, and based on the training sample set, the lightweight network model is trained for one or more rounds to obtain a second pre-trained network model, and the parameters of the lightweight network model are adjusted based on the L1 loss function until the gap between the output data and the original high-resolution image data is smaller than a preset threshold; the weights of the second pre-trained network model are loaded onto the lightweight network model, and the lightweight network model is trained based on the supervision of the teacher network to obtain a third pre-trained network model, and the parameters of the lightweight network model are adjusted based on the hybrid loss function until the gap between the output data and the original high-resolution image data and the high-resolution image data output by the teacher network is smaller than a preset threshold.

[0005] One or more embodiments of the present specification provide a knowledge distillation hot start training system for lightweight networks, the system comprising: a first acquisition module for acquiring at least one randomly initialized lightweight network model; a second acquisition module for taking a low-resolution image and a corresponding high-resolution image as a pair of training samples, and acquiring a training sample set based on multiple training samples; a first training module for inputting a low-resolution image in the training sample set into the lightweight network model, training the lightweight network model, and obtaining a first pre-trained network model; adjusting the parameters of the lightweight network model based on an L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold; a second training module for adjusting the weights of the first pre-trained network model. Loading the weights of the second pre-trained network model to the lightweight network model, and keeping the various training parameters unchanged, based on the training sample set, performing one or more rounds of training on the lightweight network model to obtain a second pre-trained network model, and adjusting the parameters of the lightweight network model based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold; a third training module is used to load the weights of the second pre-trained network model to the lightweight network model, train the lightweight network model based on the supervision of the teacher network, obtain a third pre-trained network model, and adjust the parameters of the lightweight network model based on the hybrid loss function until the difference between the output data and the original high-resolution image data and the high-resolution image data output by the teacher network model is less than a preset threshold.

[0006] One or more embodiments of this specification provide a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes any one of the above methods.

[0007] In some embodiments of this specification, the processor performs hot-start training and knowledge distillation training on an initial lightweight network model to obtain a higher-performance lightweight network model. This approach allows the lightweight network model to maintain lower computational and data requirements, resulting in higher performance. Hot-start training can reduce model heterogeneity, allowing the student model to better receive supervision from the teacher model, improving training effectiveness and stability, and broadening the application scope of the training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein:

[0009] Figure 11 is a module diagram of a knowledge distillation hot-start training system for lightweight networks according to some embodiments of this specification;

[0010] Figure 2 is an exemplary flow chart of a knowledge distillation hot start training method for a lightweight network according to some embodiments of this specification;

[0011] Figure 3 is an exemplary schematic diagram of a first pre-trained network model according to some embodiments of this specification;

[0012] Figure 4 is an exemplary schematic diagram of a second pre-trained network model according to some embodiments of this specification;

[0013] Figure 5 This is an exemplary schematic diagram of the third pre-trained network model according to some embodiments of this specification. DETAILED DESCRIPTION

[0014] To more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly describes the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this specification. Those skilled in the art can apply this specification to other similar scenarios based on these drawings without inventive effort. Unless otherwise apparent from the context or otherwise noted, the same reference numerals in the figures represent the same structure or operation.

[0015] It should be understood that the terms "system," "device," "unit," and / or "module" used herein are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0016] As used in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not refer to the singular but also include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0017] Flowcharts are used throughout this specification to illustrate the operations performed by systems according to embodiments of this specification. It should be understood that preceding or following operations do not necessarily need to be performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0018] Figure 1 This is a module diagram of a knowledge distillation hot start training system for lightweight networks according to some embodiments of this specification.

[0019] In some embodiments, the knowledge distillation hot-start training system 100 for lightweight networks may include a first acquisition module 110 , a second acquisition module 120 , a first training module 130 , a second training module 140 and a third training module 150 .

[0020] The first acquisition module 110 can be used to acquire at least one randomly initialized lightweight network model. For more details on acquiring lightweight network models, please refer to Figure 2 and its related descriptions.

[0021] The second acquisition module 120 can be used to use the low-resolution image and the corresponding high-resolution image as a pair of training samples, and acquire a training sample set based on the multiple training samples. For more details on acquiring the training sample set, please refer to Figure 2 and its related descriptions.

[0022] The first training module 130 can be used to input the low-resolution images in the training sample set into the lightweight network model, train the lightweight network model, and obtain a first pre-trained network model; adjust the parameters of the lightweight network model based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold. For more details about the first pre-trained network model, please refer to Figure 2 、 Figure 3 and its related descriptions.

[0023] The second training module 140 can be used to load the weights of the first pre-trained network model into the lightweight network model, while keeping the training parameters unchanged. Based on the training sample set, the lightweight network model is trained for one or more rounds to obtain a second pre-trained network model. The parameters of the lightweight network model are adjusted based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold. For more details about the second pre-trained network model, please refer to Figure 2 、 Figure 4 and its related descriptions.

[0024] In some embodiments, the processor can calculate the gap between the high-resolution image output by the second pre-trained network model and the original high-resolution image based on the L1 loss function; adjust the various parameters of the second pre-trained network model based on the gap, and then train again; stop training until the gap is lower than the preset threshold, and obtain a trained second pre-trained network model.

[0025] The third training module 150 can be used to load the weights of the second pre-trained network model into the lightweight network model, train the lightweight network model based on the supervision of the teacher network, obtain a third pre-trained network model, and adjust the parameters of the lightweight network model based on the hybrid loss function until the difference between the output data and the original high-resolution image data and the high-resolution image data output by the teacher network model is less than a preset threshold. For more details about the third pre-trained network model, please refer to Figure 2 、 Figure 5 and its related descriptions.

[0026] It should be understood that Figure 1 The illustrated system and its modules may be implemented in various ways.

[0027] It should be noted that the above description of a knowledge distillation hot start training system for lightweight networks and its modules is for convenience only and does not limit this specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principles of the system, they may arbitrarily combine the modules or form subsystems to connect with other modules without deviating from the principles. In some embodiments, Figure 1 The first acquisition module 110, second acquisition module 120, first training module 130, second training module 140, and third training module 150 disclosed herein may be different modules within a system, or a single module may implement the functions of two or more of the aforementioned modules. For example, the modules may share a storage module, or each module may have its own storage module. Such variations are within the scope of protection of this specification.

[0028] Figure 2 This is an exemplary flow chart of a knowledge distillation hot start training method for a lightweight network according to some embodiments of this specification. Figure 2 As shown, the process 200 includes the following steps. In some embodiments, the process 200 can be executed by a processor.

[0029] In step 210 , the processor obtains at least one randomly initialized lightweight network model.

[0030] A lightweight network model is one that is relatively small in size and number of parameters, yet computationally efficient. For example, lightweight neural network models can be created using techniques such as distillation, pruning, quantization, and weight sharing.

[0031] In some embodiments, the processor may perform random initialization on the lightweight network model to obtain an initial lightweight network model. For example, the processor may initialize various parameters of the model to random numbers within a closed interval, such as a random number within the interval [0, 1], to facilitate subsequent training of the lightweight network model.

[0032] In step 220 , the low-resolution image and the corresponding high-resolution image are used as a pair of training samples, and a training sample set is obtained based on the plurality of training samples.

[0033] Training samples are samples used to train lightweight network models. For example, low-resolution images in the samples can be input into the lightweight network model, and the corresponding high-resolution images can be used as labels for training, thereby achieving model optimization.

[0034] In some embodiments, the training samples can be determined based on historical data. For example, the processor can use a pair of images with higher scores in the historical data as a pair of training samples. The scores can be determined by expert evaluation.

[0035] In some embodiments, the processor may combine multiple pairs of training samples into a training sample set for training the model.

[0036] In step 230, the low-resolution images in the training sample set are input into the lightweight network model, the lightweight network model is trained, and a first pre-trained network model is obtained; the parameters of the lightweight network model are adjusted based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold.

[0037] The first pre-trained network model is a stable model obtained after one round of training of the lightweight network model, and has stable parameters and weights.

[0038] The L1 loss function is the mean absolute error, which refers to the average distance between the model's estimated value and the true value. The L1 loss function can be expressed as: Where p is a pixel, P refers to the image block; N is the number of pixels p in the block, and x(p) and y(p) represent the image data output by the model and the original high-resolution image data, respectively.

[0039] In some embodiments, the processor may adjust the parameters of the lightweight network model based on the L1 loss function. For example, the processor may use the average of the sample model parameters in the historical data as the parameters of the second pre-trained network model. The sample model may be a model with high model similarity and a gap value output by the L1 loss function that is less than a preset threshold.

[0040] In some embodiments, the processor can calculate the gap value between the high-resolution image output by the second pre-trained network model and the original high-resolution image based on the L1 loss function; adjust the various parameters of the second pre-trained network model based on the gap value, and then train again; stop training until the gap value is lower than a preset threshold, and obtain a trained second pre-trained network model.

[0041] In some embodiments, the processor uses the L1 loss function to train the model, which allows the model to obtain a higher initial learning rate (5e-4), thereby obtaining the target model faster.

[0042] More details about the first pre-trained model can be found in Figure 3 and its related descriptions.

[0043] Step 240: Load the weights of the first pre-trained network model into the lightweight network model, and keep the training parameters unchanged. Based on the training sample set, perform one or more rounds of training on the lightweight network model to obtain a second pre-trained network model. Adjust the parameters of the lightweight network model based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold.

[0044] The second pre-trained network model is a model with a more general structure and less heterogeneity.

[0045] In some embodiments, the processor may load the weights and parameters of the first pre-trained network model into a lightweight network model and perform one or more rounds of training to obtain a second pre-trained network model.

[0046] More details about the second pre-trained model can be found in Figure 4 and its related descriptions.

[0047] Step 250: Load the weights of the second pre-trained network model into the lightweight network model, train the lightweight network model based on the supervision of the teacher network, obtain a third pre-trained network model, and adjust the parameters of the lightweight network model based on the hybrid loss function until the difference between the output data and the original high-resolution image data and the high-resolution image data output by the teacher network is less than a preset threshold.

[0048] The teacher network is a network model with a large amount of computation and parameters, but has better performance. The teacher network can be used to supervise the learning of other networks.

[0049] In some embodiments, the processor may use the network with the largest number of parameters and the largest amount of computation in the second pre-trained model as the teacher network.

[0050] The third pre-trained network model is a model with higher resolution performance and the amount of computation and parameters remains unchanged relative to the initial lightweight network model.

[0051] In some embodiments, the processor can load the weights and parameters of the second pre-trained network model into a lightweight network model, use the model as a training model, and input the low-resolution images in the training samples into the training model and the teacher model at the same time for supervised training to obtain a third pre-trained network model.

[0052] The hybrid loss function is a loss function used for training the third pre-trained network model. The hybrid loss function may include a loss function between the output data of the third pre-trained network model and the output data of the teacher network model, and a loss function between the output data of the third pre-trained network model and the original high-resolution image data.

[0053] In some embodiments, the processor uses a hybrid loss function to train the model, which allows the model to have a smaller initial learning rate (1e-4), thereby allowing the model to output better results during training.

[0054] In some embodiments, the loss function between the output data of the third pre-trained network model and the output data of the teacher network model can be formally expressed as: Among them, y t (p) represents the pixel value output by the teacher network model.

[0055] In some embodiments, the hybrid loss function can be formally expressed as: L(P) = (1-αLl1P+αLKD(P)

[0056] α is a hyperparameter that balances the two loss functions. Its value ranges from 0 to 1. A larger value of α results in greater supervision from the teacher model. The specific value of α can be set by an expert.

[0057] More details about the third pre-trained network model can be found in Figure 5 and its related descriptions.

[0058] In some embodiments of this specification, the processor obtains a third pre-trained network model by performing hot-start training (second training) and knowledge distillation training (third training) on ​​a randomly initialized lightweight network model. This third pre-trained network model can output more accurate high-resolution images while maintaining the same number of parameters and computational complexity. Hot-start training can also effectively reduce model heterogeneity, broadening the application scope of the training method and improving model performance in a more stable manner.

[0059] It should be noted that the above description of process 200 is for illustration and purpose only and does not limit the scope of application of this specification. Those skilled in the art may make various modifications and variations to process 200 under the guidance of this specification. However, such modifications and variations are still within the scope of this specification.

[0060] Figure 3 This is an exemplary schematic diagram of a first pre-trained network model according to some embodiments of this specification.

[0061] The first pre-trained network model 320 is used to improve the stability of the initial lightweight network model. The first pre-trained network model can be of various types. For example, the first pre-trained network model can include a neural network model, a graph neural network model, etc.

[0062] In some embodiments, the input of the first pre-trained network model 320 may include the low-resolution image 310 , and the output of the first pre-trained network model may include the first reconstructed high-resolution image 330 .

[0063] In some embodiments, the first pre-trained network model can be obtained by training multiple labeled first training samples. For example, multiple labeled first training samples can be input into the initial first pre-trained network model, and an L1 loss function can be constructed using the labels and the results of the initial first pre-trained network model. Based on the L1 loss function, the parameters of the initial first pre-trained network model are iteratively updated using gradient descent or other methods. When preset conditions are met, model training is completed, resulting in a trained first pre-trained network model. The preset conditions may include convergence of the L1 loss function, the number of iterations reaching a threshold, etc.

[0064] In some embodiments, the first training sample may include at least a sample low-resolution image. The label may be the corresponding original high-resolution image in the sample. The label may be obtained based on the sample data.

[0065] In some embodiments of this specification, the processor obtains a first pre-trained network model by training a randomly initialized lightweight network model. In this way, a model with more stable weights and parameters can be obtained, which facilitates improving the stability of subsequent training.

[0066] Figure 4 is an exemplary schematic diagram of a second pre-trained network model according to some embodiments of this specification.

[0067] The second pre-trained network model 420 is used to reduce the structural heterogeneity of the first pre-trained network model. The second pre-trained network model can be of various types. For example, the second pre-trained network model can include a neural network model, a graph neural network model, etc.

[0068] In some embodiments, the input of the second pre-trained network model 410 may include the low-resolution image 310 , and the output of the second pre-trained network model may include the second reconstructed high-resolution image 420 .

[0069] In some embodiments, the second pre-trained network model can be obtained by training multiple labeled second training samples. For example, the weights and parameters of the trained first pre-trained network model can be loaded into a lightweight network model to obtain an initial second pre-trained network model, and multiple labeled second training samples can be input into the initial second pre-trained network model. The L1 loss function is constructed by the labels and the results of the initial second pre-trained network model, and the parameters of the initial second pre-trained network model are iteratively updated based on the L1 loss function by gradient descent or other methods. When the preset conditions are met, the model training is completed and the trained second pre-trained network model is obtained. The preset conditions can be that the L1 loss function converges, the number of iterations reaches a threshold, etc.

[0070] In some embodiments, the second training sample may include at least a sample low-resolution image. The label may be the corresponding original high-resolution image in the sample. The label may be determined based on the sample data.

[0071] In some embodiments of this specification, the processor loads the weights and parameters of the first pre-trained network model into the initial lightweight network model for training to obtain a second pre-trained network model. In this way, the structural heterogeneity between the models can be effectively reduced and the optimization efficiency of the models can be improved.

[0072] Figure 5 This is an exemplary schematic diagram of the third pre-trained network model according to some embodiments of this specification.

[0073] The third pre-trained network model 520 is used to improve the performance of the lightweight network model. The third pre-trained network model can be of various types. For example, the third pre-trained network model can include a neural network model, a graph neural network model, etc.

[0074] In some embodiments, the input of the third pre-trained network model 520 may include the low-resolution image 310 , and the output of the third pre-trained network model may include the third reconstructed high-resolution image 530 .

[0075] In some embodiments, the third pre-trained network model can be obtained by training multiple labeled third training samples. For example, the weights and parameters of the trained second pre-trained network model can be loaded into a lightweight network model to obtain an initial third pre-trained network model, and multiple third training samples with first labels and second labels are input into the initial third pre-trained network model. A hybrid loss function is constructed by the results of the first label and the initial third pre-trained network model, and the results of the second label and the initial third pre-trained network model. The parameters of the initial third pre-trained network model are iteratively updated based on the hybrid loss function by gradient descent or other methods. When the preset conditions are met, the model training is completed, and the trained third pre-trained network model is obtained. The preset conditions may be that the hybrid loss function converges, the number of iterations reaches a threshold, etc.

[0076] In some embodiments, the third training sample may include at least a sample low-resolution image. The first label may be the corresponding original high-resolution image in the sample, and the second label may be the high-resolution image output by the teacher network model 510. The label may be determined based on the sample data and the teacher network output data.

[0077] More details about the hybrid loss function can be found in Figure 2 and its related descriptions.

[0078] In some embodiments of this specification, the processor loads the weights and parameters of the second pre-trained network model into the initial lightweight network model and trains it under the supervision of the teacher network to obtain a third pre-trained network model. In this way, the accuracy of the model can be greatly increased without increasing the number of model parameters or the amount of computation.

[0079] In some embodiments, in some embodiments, a computer-readable storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer can execute a knowledge distillation hot start training method for a lightweight network.

[0080] While the basic concepts have been described above, it will be apparent to those skilled in the art that the detailed disclosure is merely illustrative and does not limit this specification. Although not explicitly stated herein, various modifications, improvements, and revisions to this specification may be made by those skilled in the art. Such modifications, improvements, and revisions are suggested in this specification and remain within the spirit and scope of the exemplary embodiments of this specification.

[0081] This specification also uses specific terms to describe the embodiments of this specification. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "one embodiment," "an embodiment," or "an alternative embodiment" two or more times in different locations in this specification do not necessarily refer to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of this specification may be appropriately combined.

[0082] In addition, unless expressly stated in the claims, the order of the processing elements and sequences, the use of alphanumeric characters, or the use of other names described in this specification are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some of the invention embodiments currently considered useful through various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the spirit and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0083] Similarly, it should be noted that, in order to simplify the presentation of this specification and thus facilitate understanding of one or more embodiments of the invention, the foregoing descriptions of the embodiments of this specification sometimes combine multiple features into a single embodiment, figure, or description thereof. However, this disclosure method does not imply that the subject matter of this specification requires more features than those recited in the claims. In fact, an embodiment may have fewer features than all of the features of a single disclosed embodiment.

[0084] In some embodiments, numbers are used to describe the quantity of components and attributes. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise stated, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the description and claims are approximate values, which may change according to the required characteristics of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the settings of such numerical values ​​are as accurate as possible within the feasible range.

[0085] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, and documents, cited in this specification is hereby incorporated by reference in its entirety. This includes application history documents that are inconsistent with or conflict with the content of this specification, as well as documents (currently or subsequently attached to this specification) that limit the broadest scope of the claims of this specification. It should be noted that if the descriptions, definitions, and / or terminology used in the accompanying materials are inconsistent or conflicting with the content of this specification, the descriptions, definitions, and / or terminology used in this specification will control.

[0086] Finally, it should be understood that the embodiments described in this specification are intended only to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly described and illustrated in this specification.

Claims

1. A knowledge distillation hot start training method for lightweight networks, characterized in that: The method comprises: Obtain at least one randomly initialized lightweight network model; Taking a low-resolution image and a corresponding high-resolution image as a pair of training samples, obtaining a training sample set based on multiple training samples; Inputting the low-resolution images in the training sample set into the lightweight network model, training the lightweight network model, and adjusting the parameters of the lightweight network model based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold, thereby obtaining a first pre-trained network model; The weights of the first pre-trained network model are loaded onto the lightweight network model, and various training parameters are kept unchanged. Based on the training sample set, the lightweight network model is trained for one or more rounds, and the parameters of the lightweight network model are adjusted based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold, thereby obtaining a second pre-trained network model. The adjusting the parameters of the lightweight network model based on the L1 loss function includes: Calculate the difference between the high-resolution image output by the second pre-trained network model and the original high-resolution image based on the L1 loss function; Adjusting various parameters of the second pre-trained network model based on the gap and then performing training; When the gap is lower than the preset threshold, the training is stopped to obtain a trained second pre-trained network model; The weights of the second pre-trained network model are loaded into the lightweight network model, the lightweight network model is trained based on the supervision of the teacher network, and the parameters of the lightweight network model are adjusted based on the hybrid loss function until the difference between the output data and the original high-resolution image data and the high-resolution image data output by the teacher network model is less than a preset threshold, thereby obtaining a third pre-trained network model.

2. The method according to claim 1, characterized in that The L1 loss function is: in, is the L1 loss function, p is a pixel, P refers to the image block, N is the number of pixels p in the block, x(p) and y(p) represent the image data output by the model and the original high-resolution image data, respectively.

3. The method according to claim 1, characterized in that The hybrid loss function includes a loss function between the output data of the training network and the output data of the teacher network model, and a loss function between the output data of the training network model and the original high-resolution image data.

4. The method according to claim 3, characterized in that The loss function between the output data of the training network and the output data of the teacher network model is: Among them, y t (p) is the pixel value output by the teacher network model, is the loss function between the output data of the training network and the output data of the teacher network model, p is a pixel point, P refers to the image block, N is the number of pixels p in the block, is the image data output by the model.

5. The method according to claim 3, characterized in that The hybrid loss function is: in, is a hybrid loss function, α is a hyperparameter that balances the above two loss functions, is the L1 loss function, It is the loss function between the output data of the training network and the output data of the teacher network model.

6. A knowledge distillation hot start training system for lightweight networks, configured to execute the knowledge distillation hot start training method for lightweight networks according to any one of claims 1 to 5, characterized in that: The system comprises: A first acquisition module, configured to acquire at least one randomly initialized lightweight network model; A second acquisition module is configured to use the low-resolution image and the corresponding high-resolution image as a pair of training samples, and acquire a training sample set based on the plurality of training samples; A first training module is configured to input low-resolution images in a training sample set into the lightweight network model, train the lightweight network model, and obtain a first pre-trained network model; and adjust parameters of the lightweight network model based on an L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold; a second training module, configured to load the weights of the first pre-trained network model into the lightweight network model, keep various training parameters unchanged, perform one or more rounds of training on the lightweight network model based on the training sample set to obtain a second pre-trained network model, and adjust the parameters of the lightweight network model based on the L1 loss function until the difference between the output data and the original high-resolution image data is less than a preset threshold; The third training module is used to load the weights of the second pre-trained network model into the lightweight network model, train the lightweight network model based on the supervision of the teacher network, obtain a third pre-trained network model, and adjust the parameters of the lightweight network model based on the hybrid loss function until the difference between the output data and the original high-resolution image data and the high-resolution image data output by the teacher network model is less than a preset threshold.

7. The system according to claim 6, characterized in that The adjusting the parameters of the lightweight network model based on the L1 loss function includes: Calculate the difference between the high-resolution image output by the second pre-trained network model and the original high-resolution image based on the L1 loss function; Adjusting various parameters of the second pre-trained network model based on the gap and then performing training; When the gap is lower than the preset threshold, the training is stopped to obtain a trained second pre-trained network model.

8. A computer-readable storage medium storing computer instructions, wherein when a computer reads the computer instructions in the storage medium, the computer executes the method according to any one of claims 1 to 5.

Images