Transferring knowledge from teacher neural network to student neural network

Abstract

To train a student neural network so as to adopt the behavior of a teacher neural network.SOLUTION: A method 100 comprises: Step 110 of providing a set of training images x; Step 120 of producing, from at least one training image, one or more style-augmented versions x'; Step 130 of processing the training images and the augmented versions by a teacher neural network 1 on one hand, and by a student neural network 2 on the other hand; Step 140 of evaluating, using a predetermined loss function 3, to which extent outputs 1b, 2b or intermediate work products 1a, 2a produced by the student neural network from each image are in agreement with the outputs or intermediate work products produced by the teacher neural network from the same image; and Step 150 of optimizing parameters 2c that characterize the behavior of the student neural network towards the goal of improving the value of the loss function.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to training a student neural network with a smaller architecture based on the knowledge of a given already trained teacher network with a larger architecture. [Background technology]

[0002] background During neural network training, learned knowledge is stored in parameters that characterize the neural network's behavior. Therefore, the capacity for storing knowledge is proportional to the number of trainable parameters. So-called foundational models contain a large number of parameters and are trained with large amounts of data. Examples of such foundational models include the Contrastive Language-Image Pre-Training (CLIP) model, which learns correspondences between natural language and images, and the Segment Anything Model (SAM) model, which can segment any object from any image. Summary of the Invention [Problem to be solved by the invention]

[0003] Due to the large number of parameters, using a large foundation model for real-world applications can be computationally expensive and slow. In particular, the model must fit into large storage when stored and into RAM when used. For applications running on hardware platforms that do not allow the use of the full foundation model due to resource scarcity, a student model with a smaller architecture is trained to mimic the behavior of the large foundation model, called the teacher model. If a sufficiently large set of training examples is used, and the student model provides essentially equivalent output to the teacher model for each of these, the trained student model can be used as a fast and resource-efficient yet still sufficiently accurate approximation to the teacher model. [Means for solving the problem]

[0004] Disclosure of the Invention The present invention provides a method for training a student neural network to adopt the behavior of a given teacher neural network, which has been trained to perform a given operation on an input image.

[0005] During the course of the method, a set of training images x is provided that may, but need not, be labeled with a "ground truth" about the outputs that the student network should generate from these training images x.

[0006] From at least one training image x, one or more style-enhanced versions x' are generated. These style-enhanced versions x' have the same semantic content as the original training image x, i.e., in the context of this specification, the style-enhanced versions x' have the same meaning as the original training image x. The style-enhanced versions x' differ from the original training image x in their respective styles, i.e., the style-enhanced versions x' represent different ways of rendering the same semantic meaning of the original training image x.

[0007] A training image x and an augmented version x' are processed by a teacher neural network on the one hand and a student neural network on the other hand, causing the teacher neural network and the student neural network to generate respective outputs. While generating these outputs, the teacher neural network and the student neural network also generate intermediate work products. Particularly in networks organized into multiple layers or other building blocks, the output of one such layer or building block can be viewed as an intermediate work product that is passed to the next layer or building block.

[0008] A predetermined loss function is then used to evaluate the degree to which the output and / or intermediate work products generated by the student neural network from each image x, x' match the output and / or intermediate work products generated by the teacher neural network from the same images x, x'. Parameters characterizing the behavior of the student neural network are then optimized with the goal of improving the value of the loss function. These parameters may also include parameters related to the architecture of the student network itself. That is, the method is not only useful for training student neural networks of fixed architecture, but may also be used as a tool for Neural Architecture Search (NAS).

[0009] We find that expanding the training images x by restyling them significantly expands the diversity of images on which to base our assessment of whether the behavior of the student neural network corresponds as intended to that of the teacher neural network. If this manifold is large enough, we can expect the behavior of the student neural network to match that of the teacher neural network well enough, even for input images that the student neural network has not seen during training. Each time a new style change is applied to the training images x, the entire number of training images x is effectively added back to the final training data set.

[0010] Furthermore, data augmentation in this manner is independent of the specific tasks solved by the teacher and student neural networks, meaning that the method is not limited to neural networks used, for example, for image classification (i.e., assigning classification scores for one or more classes to an image as a whole), meaning that the student neural network can even solve tasks that are different from those solved by the teacher neural network.

[0011] As used herein, determining whether outputs and / or intermediate work products "match" is not limited to a 1:1 comparison. Such a 1:1 comparison may not be possible in all cases. Because student neural networks typically have a much smaller architecture than the teacher neural network, the outputs of the student neural network, particularly the intermediate work products, are of different dimensionality (i.e., vectors or tensors of different sizes) than the teacher neural network. For example, aggregate quantities and / or statistics can be calculated from the outputs and / or intermediate work products, and these aggregate quantities and / or statistics can be compared.

[0012] By using an augmented training dataset that includes training images x as well as augmented versions x' of these training images x, the tendency of the training of the student neural network to overfit to the training images x is reduced. Typically, the dataset of training images x is much smaller than the dataset of training images used to train the teacher neural network. For example, in an application involving the performance of a student neural network in the task of semantic segmentation of images of urban scenes, the primary training dataset for this application, Cityscape, contains only 2795 training images, all of which relate to sunny daytime conditions. Without data augmentation, it would be difficult, if not impossible, to generalize the training of the student model to conditions such as nighttime, rainy, or snowy conditions.

[0013] In a particularly advantageous embodiment, the step of generating an augmented version x′ of the training image x comprises: Extracting content c from training images x; obtaining a style s based at least in part on a style source image y; Processing the semantic content c and style s into the desired augmented version x' by a trained generative model G; Includes.

[0014] In this way, content c from training image x is mixed with style s from style source image y. This means that the new style s to be applied to training image x does not need to be explicitly formulated. Rather, the new style s to be applied to training image x can be implicitly supplied in the form of style source image y. This means that style source image y can contain multiple simultaneous style changes and does not need to be disentangled.

[0015] especially, Extracting semantic content c from training images x, and / or Obtaining style s based on style source image y may be implemented by feeding a training image x or a style source image y to a trained feature extractor network F configured to determine both the content c and the style s from the input image. This means that for each training image x and style source image y, a respective pair (c,s) of content c and style s is created. Only the content portion c from the pair (c,s) corresponding to training image x and the style portion s from the pair (c,s) corresponding to style source image y are used.

[0016] For example, first, the training images

number

number

[0017] In a further advantageous embodiment, the style s is selected to be an interpolation between a first style s' of the training image x and a second style s'' of the style source image y. In this way, multiple style-enhanced versions x' of the same training image x can be generated based on a single style source image y.

[0018] For example, a training image x may have been taken during the day, and thus its style s is that of a daytime image. A style source image y may have been taken at night, and thus its style is that of a nighttime image. Style interpolation can then be used to create new styles that transition between daytime and nighttime in varying degrees of twilight and apply them to the content c of training image x to create a new augmented version x'.

[0019] In particular, saying that the style of a source image y differs from the style of training images x can mean that the style source image y differs from one or more training images x in at least one aspect that distinguishes it from the content c. As mentioned above, this aspect need not be explicitly named. However, by selecting this aspect, it is possible to actively steer the direction in which the manifold of images underlying checking for a match between the student and teacher neural networks is expanded.

[0020] For example, the style source image y is different from all the training images x in the following manner: The time the image was taken, The season in which the image was acquired, the weather and / or lighting conditions under which the image was taken; defects and / or imperfections in the image; Camera settings used to capture the images may include one or more of:

[0021] These are typical variations that occur in real-world applications, where input images are acquired by at least one sensor, and where these input images must not cause the behavior of the student neural network used in the application to suddenly deviate from the behavior of the teacher neural network.

[0022] In a further particularly advantageous embodiment, at least one augmented version x' of a training image x and / or at least one style source image y are obtained from the trained generative diffusion model D using text prompts that indicate a desired style for the augmented version x' or the style source image y. In this way, if an explicit concept of the desired style change is available, this desired style change can be rendered by the trained generative diffusion model D. For example, when provided with a training image x, the generative diffusion model D can be instructed to create an augmented version x' that still has the concept of the original training image x but is modified according to the text prompt. Depending on how strongly conformance to the original training image x is desired, the appearance of the content of the original training image x, in addition to the style, can be altered by the generative diffusion model D, to the extent that this content still has a similar semantic meaning in the current application. For example, if the original training image x shows a castle and the generative diffusion model D is instructed to transform it into a winter image, in addition to this style change, the number and appearance of the castle's towers can be changed, although the castle will still remain a castle.

[0023] In a further particularly advantageous embodiment, the feature maps output by the intermediate layers of the respective neural networks are selected as intermediate work products. For example, these feature maps can be output by convolutional layers. In particular, feature maps of different sizes can be processed (e.g., by evaluating summaries of the features they contain) to check how well they match each other.

[0024] In a further particularly advantageous embodiment, the outputs of the neural network are selected to be logits and / or other unaggregated results output by the neural network. In particular, logits are outputs that have not yet been normalized by a normalization function such as a softmax function. Furthermore, the logits have not yet been condensed into a final decision, such as a one-hot vector of classification scores output by the final fully connected layer of the neural network used as a classifier. In this way, even minor differences between the behavior of the teacher neural network and the behavior of the student neural network can be captured.

[0025] In a further particularly advantageous embodiment, the loss function measures the distance between the outputs and / or intermediate work products generated by the teacher neural network, on the one hand, and the outputs and / or intermediate work products generated by the student neural network, on the other hand. This distance may be measured, for example, in a common space to which the outputs or intermediate work products of the teacher and student neural networks belong. If such a common space does not exist, for example, because the outputs or intermediate work products are of different sizes, another option is to transform both outputs or both work products into a common workspace and measure the distance in this workspace.

[0026] For example, if a common space exists, a norm such as the L2 norm can be used as the distance measure. Also, in particular, the architecture of the student neural network may be selected to include at least one layer or other building block with an output of the same dimensionality as the output of the corresponding layer or other building block in a given teacher neural network. In this way, the distance between the outputs of corresponding layers can be measured directly in a common space. Such a loss function L KD An example of this is:

number

[0027] In a further particularly advantageous embodiment, the Kullback-Leibler divergence is chosen as the distance measure. This distance measure was originally intended to measure the distance between distinct distributions, and therefore does not require that the outputs or work products of both the teacher and student neural networks be in the same space. Optionally, the outputs or work products can be divided by a scalar temperature constant τ to smooth the outputs or work products, respectively. Such a loss function L KD An example of this is: L KD =KL(p t / τ,p s / τ) where p t and p s are the outputs or intermediate work products ("predictions") of the teacher and student neural networks, respectively.

[0028] In a further particularly advantageous embodiment, the student neural network is configured to generate outputs for a given task, where the training images x are labeled with a ground truth for the given task, and the loss function further measures the difference and / or distance between the output generated from each training image x and the ground truth for each training image x. task If the task loss is, the total loss function L is, for example, L=L task +L KD It can be said that:

[0029] In this way, a student neural network can be trained for behavior that corresponds to the behavior of the teacher neural network, while also being trained to solve a given task. For example, a given teacher neural network may be a general-purpose trained foundation model trained to perform fairly abstract processing of input images based on unlabeled data, while a student neural network may be trained to solve a specific task for which ground truth is available. Thus, rather than training a neural network from scratch for a given task, which is impractical in many applications, the training of the student neural network builds on knowledge in the teacher neural network, allowing the student neural network to be refined to solve a specific task using only a small set of training images x.

[0030] In a further particularly advantageous embodiment, the loss function further measures the difference and / or distance between the output generated from each augmented image version x' and the ground truth for the training image x to which the augmented image version x' is associated. Since the augmented image versions x' have similar semantic content to the training images x to which they are associated, the ground truth annotated ("labeled") to the training images x is also valid for the augmented version x'. Thus, obtaining the augmented version x' according to the method proposed herein expands the pool of available training examples, even for supervised training, to solve a given task, without incurring additional overhead for labeling.

[0031] In further particularly advantageous embodiments, the student neural network is configured as an image classifier, a semantic segmentation model, and / or an object detector. These are typical tasks for which the response of the student neural network should not change abruptly with changes in image style. In this regard, it is advantageous that the approach presented herein for obtaining the augmented version x' is not tied to the classification task.

[0032] The ultimate goal of training a student neural network is to use it in real-world applications, especially on hardware platforms that do not allow the execution of a teacher neural network. Therefore, in a further particularly advantageous embodiment, input images acquired using at least one sensor are provided to the trained student neural network. An operating signal is then determined from the results of the trained student neural network. A vehicle, a driver assistance system, a robot, a monitoring system, a quality inspection system, and / or a medical imaging system are then operated by the operating signal. This increases the probability that the reaction of each technical system operated in response to the operating signal will be appropriate for the situation characterized by the acquired input images. In particular, sudden changes in image style, such as a change from daytime to nighttime or from summer conditions to winter conditions, will not induce sudden and unexpected changes in behavior. For example, if a car is crossing the Alps through a tunnel, the conditions at both ends of the tunnel may be completely different.

[0033] The method may be wholly or partly computer-implemented and may be implemented in software. Accordingly, the present invention also relates to a computer program comprising machine-readable instructions that, when executed on one or more computers and / or computing instances, cause the one or more computers and / or computing instances to implement the above method. In this specification, control units for vehicles or robots and for other embedded systems capable of executing machine-readable instructions should also be considered as computers. Computing instances include virtual machines, containers, or other execution environments that enable the execution of machine-readable instructions in the cloud.

[0034] The non-transitory machine-readable data carrier and / or download product may contain a computer program. A download product is an electronic product that may be sold online and transmitted over a network for immediate realization. One or more computers and / or computing instances may be equipped with said computer program and / or said non-transitory storage medium and / or download product.

[0035] The present invention will be described below with reference to the accompanying drawings, which are not intended to limit the scope of the present invention. [Brief explanation of the drawings]

[0036] [Figure 1] FIG. 1 illustrates an exemplary embodiment of a method 100 for training a student neural network 2. [Figure 2a] FIG. 1 illustrates an exemplary technique for comparing the behavior of a teacher neural network 1 and a student neural network 2 based on outputs 1b and 2b. [Figure 2b] FIG. 1 illustrates an exemplary technique for comparing the behavior of a teacher neural network 1 and a student neural network 2 based on intermediate work products 1a and 2a. [Figure 3] FIG. 1 illustrates exemplary data augmentation for training towards solving a given task using unlabeled style source images y. DETAILED DESCRIPTION OF THE INVENTION

[0037] 1 is a schematic flow chart of an example of a method 100 for training a student neural network 2 to adopt the behavior of a given teacher neural network 1. The teacher neural network 1 has been trained to perform a given operation on an input image.

[0038] In step 110, a set of training images x is provided.

[0039] Optionally, according to block 105, the student neural network 2 may be configured to generate an output 2b for a given task. Then, according to block 111, the training images x may be labeled with the ground truth 2b* for the given task.

[0040] In step 120, one or more style-enhanced versions x' are generated from at least one training image x, which have similar semantic content as the original training image x, but at the same time differ from the original training image x in their respective styles.

[0041] According to block 121, content c can be extracted from training image x. According to block 122, style s can be obtained based at least in part on style source image y. Then, according to block 123, the semantic content c and style s can be processed by a trained generative model G into the desired augmented version x'.

[0042] According to block 121a or 122a, extracting 121 or obtaining 122 may be performed by feeding training images x or style source images y to a trained feature extractor network F configured to determine both content c and style s from an input image.

[0043] According to block 124, style s may be selected to be an interpolation between a first style s' of training image x and a second style s'' of style source image y.

[0044] According to block 125, the style source image y may differ from one or more training images x in at least one aspect that distinguishes it from the content c.

[0045] According to block 126, at least one augmented version x′ of the training image x and / or at least one style source image y may be obtained from the trained generative diffusion model D using text prompts indicating the desired style of the augmented version x′ or style source image y.

[0046] In step 130, the training image x and the augmented version x′ are processed by a teacher neural network 1 which on the one hand generates intermediate work products 1a and outputs 1b, On the other hand it is processed by a student neural network 2 which generates intermediate work products 2a and outputs 2b.

[0047] In step 140, a predetermined loss function 3 is used to evaluate the extent to which the output 2b and / or intermediate work product 2a generated by the student neural network 2 from each image x, x' matches the output 1b and / or intermediate work product 1a generated by the teacher neural network 1 from the same image x, x'. The result of the loss function 3 is a value 3a.

[0048] According to block 141, the loss function 3 can measure the distance between, on the one hand, the outputs 1b and / or intermediate work products 1a generated by the teacher neural network 1 and, on the other hand, the outputs 2b and / or intermediate work products 2a generated by the student neural network 2. In particular, according to block 141a, the Kullback-Leibler divergence can be selected as the measure of distance.

[0049] According to block 142, when the student neural network 2 is configured to solve a particular task according to block 105, and when the training images x are labeled with corresponding ground truth 2b*, the loss function 3 can further measure the difference and / or distance between the output 2b generated from each training image x and the ground truth 2b* for each training image x.

[0050] In particular, according to block 142a, the loss function 3 may further measure the difference and / or distance between the output 2b generated from each augmented image version x′ and the ground truth 2b* for the training image x to which this augmented image version x′ is associated.

[0051] In step 150, parameters 2c characterizing the behavior of student neural network 2 are optimized with the goal of improving the value 3a of loss function 3. The final optimized state of parameters 2c is labeled with reference character 2c* and characterizes the fully trained state 2* of student neural network 2.

[0052] 1, in step 160, an input image 4 acquired using at least one sensor 5 is provided to a trained student neural network 2*. The trained student neural network 2* then generates an output 2b.

[0053] An operating signal 170a is determined from this output 2b in step 170. In step 180, the vehicle 50, the driver assistance system 51, the robot 60, the monitoring system 70, the quality inspection system 80 and / or the medical imaging system 90 are operated by this operating signal 170a.

[0054] FIG. 2 shows an exemplary technique for comparing the behavior of a teacher neural network 1 with that of a student neural network 2.

[0055] In the example shown in FIG. 2a, step 120 of method 100 creates an augmented version x' from training image x. Here, training image x, which shows vehicles on a road, was taken during the day, but the new style applied to training image x is that of a nighttime image. The resulting augmented version x' thus shows the same vehicles on the same road as in the original training image x, but at night. Both training image x and augmented version x' are provided to both teacher neural network 1 and student neural network 2. Teacher neural network 1 generates output 1b, while student neural network 2 generates output 2b. These outputs 1b and 2b are compared by loss function 3. The snowflake symbol on the box representing teacher neural network 1 indicates that the parameters characterizing the behavior of teacher neural network 1 remain frozen. The flame symbol on the box representing student neural network 2 indicates that parameter 2c characterizing the behavior of student neural network 2 is being optimized. That is, based on the value 3 a of the loss function 3 , an update Δ2 c for the parameters 2 c of the student neural network 2 is calculated and fed back to the student neural network 2 .

[0056] In the example shown in Figure 2b, an augmented version x' is created from training image x, just as in the example shown in Figure 2a. Also consistent with Figure 2a, both training image x and augmented version x' are provided to both teacher neural network 1 and student neural network 2. However, in contrast to Figure 2a, the intermediate work products 1a and 2a generated by teacher neural network 1 and student neural network 2 are taken into account by loss function 3. As in Figure 2a, an update Δ2c for parameters 2c of student neural network 2 is calculated and fed back to student neural network 2. Meanwhile, the parameters of teacher neural network 1 remain frozen.

[0057] Figure 3 shows how the above data augmentation can also be used for supervised training of a student neural network 2 towards solving a given task using unlabeled style source images y. In the example shown in Figure 3, the given task to be solved is semantic segmentation. The training images x and the augmented versions x' are the same as in Figure 2. Additionally, the source style images y from which the nightly styles were derived are shown.

[0058] Both the training image x and the source style image y are decomposed into their respective content c and style s by a trained feature extractor F. The content c derived from the training image x and the style derived from the source style image y are processed by a trained generative model G into an augmented version x'.

[0059] The training images x are labeled by the corresponding ground truth 2b*. The augmented version x' has similar semantic content, so it can be used to train a student neural network 2 for the task of semantic segmentation in exactly the same way. The achievement of this task is achieved by the loss function 3 presented above and its contribution L taskand is measured according to.

Claims

1. A method (100) for training a student neural network (2) to adopt the behavior of a given teacher neural network (1) that has been trained to perform a given operation on an input image, comprising: The method (100) comprises: Providing (110) a set of training images x; Generating (120) one or more style-enhanced versions x′ from at least one training image x, said style-enhanced versions x′ comprising: - has similar semantic content as the original training image x, but - differs in its style from the original training image x, Step (120); a step (130) of processing the training images x and the augmented versions x' by the teacher neural network (1) on the one hand and the student neural network (2) on the other hand; - using a predetermined loss function (3), assessing (140) to what extent the output (2b) and / or intermediate work product (2a) generated by the student neural network (2) from each image x, x' matches the output (1b) and / or intermediate work product (1a) generated by the teacher neural network (1) from the same image x, x'; a step (150) of optimizing parameters (2c) characterizing the behavior of said student neural network (2) with the goal of improving the value (3a) of said loss function (3); A method (100) comprising:

2. The step of generating 120 an augmented version x′ of a training image x comprises: Extracting (121) content c from said training images x; Obtaining a style s based at least in part on a style source image y (122); Processing 123 the semantic content c and the style s by a trained generative model G into the desired augmented version x′; The method (100) of claim 1, comprising:

3. Extracting 121 the semantic content c from the training images x, and / or Obtaining the style s based on the style source image y (122). is performed by feeding the training image x, each of the style source images y to a trained feature extractor network F configured to determine both the content c and style s from an input image (121a, 122a); The method (100) of claim 2.

4. The style s is selected (124) to be an interpolation between a first style s′ of the training image x and a second style s″ of the style source image y; The method (100) of claim 2 or 3.

5. The style source image y differs from one or more training images x in at least one aspect that distinguishes it from the content c (125); The method (100) of any one of claims 2 to 4.

6. The aspect in which the style source image y is different from all training images x specifically includes: the time at which the image was acquired; - the season in which the image was acquired; the weather and / or lighting conditions under which the image was acquired; - defects and / or imperfections in said image; The camera settings used to capture the image [0033] The method (100) of claim 4.

7. At least one augmented version x′ of the training image x and / or at least one style source image y are obtained from the trained generative diffusion model D using text prompts indicating the desired style of the augmented version x′, each of the style source images y (126); The method (100) of any one of claims 1 to 6.

8. The feature maps output by the intermediate layers of each of the neural networks (1, 2) are selected as intermediate work products (1a, 2a). The method (100) of any one of claims 1 to 7.

9. the outputs (1b, 2b) of the neural network (1, 2) are selected to be logits and / or other unaggregated results output by the neural network (1, 2); The method (100) of any one of claims 1 to 8.

10. The loss function (3) measures the distance (141) between the outputs (1b) and / or intermediate work products (1a) generated by the teacher neural network (1) on the one hand and the outputs (2b) and / or intermediate work products (2a) generated by the student neural network (2) on the other hand. The method (100) of any one of claims 1 to 9.

11. As the distance measure, the Kullback-Leibler divergence is selected (141a), The method (100) of claim 10.

12. The student neural network (2) is configured (105) to generate an output (2b) related to a given task; The training images x are labeled (111) by the ground truth (2b*) for the given task; The loss function (3) further measures (142) the difference and / or distance between the output (2b) generated from each training image x and the ground truth (2b*) for each training image x; The method (100) of any one of claims 1 to 11.

13. The loss function (3) further measures (142a) the difference and / or distance between the output (2b) generated from each augmented image version x′ and the ground truth (2b*) for the training image x to which the augmented version x′ is associated. The method (100) of claim 12.

14. The student neural network (2) is configured as an image classifier, a semantic segmentation model, and / or an object detector. The method (100) of any one of claims 1 to 13.

15. The method (100) comprises: providing (160) an input image (4) acquired using at least one sensor (5) to the trained student neural network (2*); determining (170) an operating signal (170a) from the results (2b) of the trained student neural network (2*); - operating (180) a vehicle (50), a driver assistance system (51), a robot (60), a monitoring system (70), a quality inspection system (80), and / or a medical imaging system (90) by said operating signal (170a); The method (100) of any one of claims 1 to 14, further comprising:

16. 16. A computer program comprising machine-readable instructions that, when executed on one or more computers and / or computing instances, cause said one or more computers and / or computing instances to perform the method (100) of any one of claims 1 to 15.

17. A non-transitory machine-readable data carrier and / or download product comprising a computer program according to claim 16.

18. One or more computing instances comprising a computer program according to claim 16 and / or a machine-readable data carrier and / or downloadable product according to claim 17.

Images