Methods and electronic devices for training neural networks for image processing

Abstract

A method and electronic device for training a neural network for image processing are disclosed. The method for training a neural network for image processing includes: obtaining a first output of each layer of at least two layers of a teacher network based on a first image; generating a pseudo-label corresponding to each first output based on the first output; obtaining a second output processed by at least one layer including an output layer in a student network based on the first image; generating a prediction result corresponding to each of the at least two layers based on the second output; and updating the student network based on the pseudo-label and the prediction result.

Description

Technical Field

[0001] This disclosure relates to the field of image processing. More specifically, it relates to a method and electronic device for training a neural network for image processing. Background Technology

[0002] Neural networks have been widely used in the field of image processing. However, existing neural networks for image processing have low accuracy and struggle to obtain accurate image processing results based on input images. Therefore, it is necessary to train neural networks to improve their image processing performance. Summary of the Invention

[0003] The present invention is provided in a brief form to introduce the choice of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0004] This application provides a method and electronic device for training a neural network for image processing, which can at least improve the performance of the neural network for image processing.

[0005] According to some example embodiments, a method for training a neural network for image processing includes: obtaining a first output of each layer of at least two layers of a teacher network based on a first image; generating a pseudo-label corresponding to each first output based on each first output; obtaining a second output processed by at least one layer including an output layer in a student network based on the first image; generating a prediction result corresponding to each of the at least two layers based on each second output; and updating the student network based on the pseudo-label and the prediction result.

[0006] The exemplary embodiments of this disclosure generate multiple pseudo-labels based on the outputs of different layers of the teacher network, which can produce pseudo-labels at different semantic levels, improve the quality of pseudo-label generation, and improve the accuracy of training the student network.

[0007] The method further includes: performing weak enhancement processing on the first image to generate at least one second image, and performing strong enhancement processing on the first image to generate at least one third image, wherein the number of the at least one second image is the same as the number of the at least one third image; wherein the step of obtaining the first output of each layer of at least two layers of the teacher network based on the first image includes: obtaining the first output of each layer of at least two layers of the teacher network corresponding to each second image based on each second image; wherein the step of obtaining the second output processed by at least one layer including an output layer in the student network based on the first image includes: obtaining the output of each layer of at least one layer including an output layer in the student network corresponding to each third image based on each third image, and merging the output of each layer corresponding to each third image into the second output corresponding to each third image.

[0008] The step of generating at least one second image includes: performing weak enhancement processing on a first image to generate multiple weak first images, performing weak enhancement processing on an additional image to generate multiple weak additional images, and performing one-to-one image fusion on the multiple weak first images and the multiple weak additional images to generate multiple second images; wherein, the step of generating at least one third image includes: performing strong enhancement processing on a first image to generate multiple strong first images of different sizes, performing strong enhancement processing on an additional image to generate multiple strong additional images of different sizes, and performing one-to-one image fusion on the multiple strong first images and the multiple strong additional images to generate multiple third images.

[0009] The exemplary embodiments of this disclosure, by performing image enhancement and image fusion on the enhanced images, can generate a sufficient number of training samples to increase the training difficulty, thereby improving the accuracy of training neural networks for image processing and addressing the dependence on a large number of samples and the requirements for large hardware resources. Furthermore, the exemplary embodiments of this disclosure reduce noise in pseudo-labels by inputting samples with varying degrees of enhancement into the teacher and student networks.

[0010] The step of generating the plurality of second images by performing one-to-one image fusion on the plurality of weak first images and the plurality of weak additional images includes: adjusting the size of each weak additional image to the product of the size of the corresponding weak first image to be fused and a predetermined ratio, and overlaying each size-adjusted weak additional image on a predetermined area of ​​the corresponding weak first image to be fused to generate the plurality of second images; the step of generating the plurality of third images by performing one-to-one image fusion on the plurality of strong first images and the plurality of strong additional images includes: adjusting the size of each strong additional image to the product of the size of the corresponding strong first image to be fused and a predetermined ratio, and overlaying each size-adjusted strong additional image on a predetermined area of ​​the corresponding strong first image to be fused to generate the plurality of third images.

[0011] The exemplary embodiments of this disclosure perform image fusion by controlling the size ratio of the two images in the fused image and controlling the position of the two images in the fused image, which can improve the accuracy of training student networks.

[0012] The method further includes: adjusting the size of each weak first image to the product of the size of the corresponding weak additional image to be fused and a predetermined ratio, and overlaying each size-adjusted weak first image on a predetermined region of the corresponding weak additional image to be fused to generate an intermediate image corresponding to each second image, wherein each second image and the intermediate image corresponding to each second image are generated based on the same weak first image and weak additional image; obtaining a third output for each of the at least two layers of the teacher network corresponding to each intermediate image; generating a pseudo-label corresponding to each third output based on each third output; fusing the pseudo-labels corresponding to each first output and the pseudo-labels corresponding to each third output to generate a final pseudo-label corresponding to each first output, wherein the step of updating the student network based on the pseudo-labels and prediction results includes: updating the student network according to the final pseudo-labels and prediction results.

[0013] The step of generating the final pseudo-label corresponding to each of the first outputs includes: using the following equation to generate the final pseudo-label corresponding to each of the first outputs:

[0014] Yij=(1-r)×Yaij+r×Ybij

[0015] Yij represents the final pseudo-label corresponding to the i-th second image in the j-th layer of the at least two layers, r represents a predetermined ratio, Yaij represents the pseudo-label corresponding to the first output of the i-th second image in the j-th layer of the at least two layers, and Ybij represents the pseudo-label corresponding to the third output of the intermediate image corresponding to the i-th second image in the j-th layer of the at least two layers, wherein the predetermined ratio is less than 0.5.

[0016] The exemplary embodiments of this disclosure determine the final pseudo-label based on the pseudo-label corresponding to the second image and the pseudo-label corresponding to the intermediate image corresponding to the second image, and update the student network based on the final pseudo-label. This can fully take into account the information of the two images in the second image generated by image fusion, thereby improving the accuracy of training the student network.

[0017] The at least one layer in the student network, including the output layer, is adjacent.

[0018] The step of updating the student network based on the pseudo-labels and prediction results includes: determining a loss based on the pseudo-labels and prediction results, and using the determined loss to update the student network; the loss includes the difference between each prediction result of each third image and the pseudo-label corresponding to each prediction result of any second image, wherein each prediction result and the pseudo-label corresponding to each prediction result correspond to the same layer in the at least two layers.

[0019] The step of generating pseudo-labels corresponding to each first output includes: calculating the similarity between each first output and the historical outputs of the layer corresponding to each first output, performing momentum normalization on the similarity, and generating a probability value as the pseudo-label corresponding to each first output, wherein the historical outputs of the layer corresponding to each first output include the outputs generated from the layer corresponding to each first output of the teacher network in the past when the teacher network was executed.

[0020] The step of calculating the similarity between each first output and the historical output of the layer corresponding to each first output includes: calculating the Euclidean distance between each first output and the historical output of the layer corresponding to each first output as the similarity; and the step of performing momentum normalization on the similarity includes: normalizing the Euclidean distance using at least one of a first parameter determined based on a statistical value of the Euclidean distance and a second parameter determined based on an extreme value of the Euclidean distance.

[0021] The exemplary embodiments of this disclosure improve the stability of training student networks by momentum normalizing the similarity between the output of the teacher network and historical outputs, thereby reducing the accuracy loss during training.

[0022] The step of generating a prediction result corresponding to each of the at least two layers based on each second output includes: calculating the similarity between the historical output of each of the at least two layers in the teacher network and each second output, as the prediction result corresponding to each of the at least two layers.

[0023] The step of calculating the similarity between the historical output of each layer of the at least two layers in the teacher network and each second output includes: when the dimension of each second output is different from the dimension of the historical output of each layer of the at least two layers, converting the dimension of the historical output of each layer of the at least two layers into the dimension of each second output; calculating the similarity between the converted historical output of each layer of the at least two layers and each second output as the prediction result corresponding to each layer of the at least two layers.

[0024] The steps for updating the student network based on the loss include: averaging the loss corresponding to the third image of the first size as the loss corresponding to the third image of the first size; averaging the loss corresponding to the third image of the second size as the loss corresponding to the third image of the second size; and updating the student network based on the sum of the loss corresponding to the third image of the first size and the loss corresponding to the third image of the second size as the total loss.

[0025] The exemplary embodiments of this disclosure determine the total loss for updating the student network by using the average of the losses corresponding to third images of the same size as the loss corresponding to the third image of that size, and by summing the losses corresponding to third images of different sizes, thereby improving the accuracy of training the student network.

[0026] According to some example embodiments, an electronic device includes: a backbone network module, the backbone network module including a teacher network and a student network, configured to: obtain a first output of each layer of at least two layers of the teacher network based on a first image; generate a pseudo-label corresponding to each first output based on each first output; obtain a second output processed by at least one layer of the student network including an output layer based on the first image; and generate a prediction result corresponding to each of the at least two layers based on each second output; and a knowledge distillation module configured to: update the student network based on the pseudo-labels and the prediction results.

[0027] The exemplary embodiments of this disclosure generate pseudo-labels based on the outputs of different layers of the teacher network, which can produce pseudo-labels at different semantic levels, thereby improving the quality of pseudo-label generation and increasing the accuracy of training the student network.

[0028] The electronic device further includes an image processing module, wherein the image processing module is configured to: perform weak enhancement processing on a first image to generate at least one second image, and perform strong enhancement processing on the first image to generate at least one third image, wherein the number of the at least one second image is the same as the number of the at least one third image. The backbone network module is further configured to: based on each second image, obtain a first output of each layer in at least two layers of the teacher network corresponding to each second image; based on each third image, obtain the output of each layer in at least one layer including an output layer in the student network corresponding to each third image, and merge the output of each layer corresponding to each third image into a second output corresponding to each third image.

[0029] The image processing module is also configured to: perform weak enhancement processing on the first image to generate multiple weak first images; perform weak enhancement processing on the additional image to generate multiple weak additional images; perform one-to-one image fusion on the multiple weak first images and the multiple weak additional images to generate multiple second images; perform strong enhancement processing on the first image to generate multiple strong first images of different sizes; perform strong enhancement processing on the additional image to generate multiple strong additional images of different sizes; and generate multiple third images by performing one-to-one image fusion on the multiple strong first images and the multiple strong additional images.

[0030] The exemplary embodiments of this disclosure, by performing image enhancement and image fusion on the enhanced images, can generate a sufficient number of training samples to increase the training difficulty, thereby improving the accuracy of training neural networks for image processing and addressing the dependence on a large number of samples and the requirements for large hardware resources. Furthermore, the exemplary embodiments of this disclosure reduce noise in pseudo-labels by inputting samples with varying degrees of enhancement into the teacher and student networks.

[0031] The image processing module is further configured to: adjust the size of each weak additional image to the product of the size of the corresponding weak first image to be fused and a predetermined ratio, and overlay each size-adjusted weak additional image on a predetermined area of ​​the corresponding weak first image to be fused to generate the plurality of second images; adjust the size of each strong additional image to the product of the size of the corresponding strong first image to be fused and a predetermined ratio, and overlay each size-adjusted strong additional image on a predetermined area of ​​the corresponding strong first image to be fused to generate the plurality of third images.

[0032] The exemplary embodiments of this disclosure perform image fusion by controlling the size ratio of the two images in the fused image and controlling the position of the two images in the fused image, which can improve the accuracy of training student networks.

[0033] The image processing module is further configured to: adjust the size of each weak first image to the product of the size of the corresponding weak additional image to be fused and a predetermined ratio, and overlay each size-adjusted weak first image on a predetermined region of the corresponding weak additional image to be fused to generate an intermediate image corresponding to each second image, wherein each second image and the intermediate image corresponding to each second image are generated based on the same weak first image and weak additional image; wherein the backbone network module is further configured to: obtain a third output of each of the at least two layers of the teacher network corresponding to each intermediate image based on each intermediate image; generate a pseudo label corresponding to each third output based on each third output; fuse the pseudo labels corresponding to each first output and the pseudo labels corresponding to each third output to generate a final pseudo label corresponding to each first output, wherein the knowledge distillation module is further configured to: update the student network based on the final pseudo label and the prediction result.

[0034] The backbone network module is also configured to generate the final pseudo-label corresponding to each of the first outputs using the following equation:

[0035] Yij=(1-r)×Yaij+r×Ybij

[0036] Yij represents the final pseudo-label corresponding to the i-th second image in the j-th layer of the at least two layers, r represents a predetermined ratio, Yaij represents the pseudo-label corresponding to the first output of the i-th second image in the j-th layer of the at least two layers, and Ybij represents the pseudo-label corresponding to the third output of the intermediate image corresponding to the i-th second image in the j-th layer of the at least two layers, wherein the predetermined ratio is less than 0.5.

[0037] The exemplary embodiments of this disclosure determine the final pseudo-label based on the pseudo-label corresponding to the second image and the pseudo-label corresponding to the intermediate image corresponding to the second image, and update the student network based on the final pseudo-label. This can fully take into account the information of the two images in the second image generated by image fusion, thereby improving the accuracy of training the student network.

[0038] The at least one layer in the student network, including the output layer, is adjacent.

[0039] The knowledge distillation module is configured to: determine a loss based on the pseudo-labels and prediction results, and update the student network based on the loss, wherein the loss includes the difference between each prediction result for each third image and the pseudo-label corresponding to each prediction result for any second image, wherein each prediction result and the pseudo-label corresponding to each prediction result correspond to the same layer in the at least two layers.

[0040] The backbone network module is configured to: calculate the similarity between each first output and the historical output of the layer corresponding to each first output, perform momentum normalization on the similarity, and generate a probability value as a pseudo-label corresponding to each first output, wherein the historical output of the layer corresponding to each first output includes the output generated from the layer corresponding to each first output of the teacher network in the past when the teacher network was executed.

[0041] The backbone network module is configured to: calculate the Euclidean distance between each first output and the historical output of the layer corresponding to each first output as the similarity, and normalize the Euclidean distance using at least one of a first parameter determined based on the statistical value of the Euclidean distance and a second parameter determined based on the extreme value of the Euclidean distance.

[0042] The backbone network module is configured to: calculate the similarity between the historical output of each of the at least two layers in the teacher network and each of the second outputs, as a prediction result corresponding to each of the at least two layers.

[0043] The backbone network module is configured to: when the dimension of each second output is different from the dimension of the historical output of each layer of at least two layers, convert the dimension of the historical output of each layer of at least two layers into the dimension of each second output; calculate the similarity between the historical output of each layer of at least two layers after conversion and each second output, as the prediction result corresponding to each layer of at least two layers.

[0044] The knowledge distillation module is configured to: average the loss corresponding to the third image of the first size among the losses, and use it as the loss corresponding to the third image of the first size; average the loss corresponding to the third image of the second size among the losses, and use it as the loss corresponding to the third image of the second size; and update the student network based on the sum of the losses corresponding to the third image of the first size and the losses corresponding to the third image of the second size as the total loss.

[0045] In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform the methods described above. Attached Figure Description

[0046] The above and other objects, features and advantages of this disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0047] Figure 1 This is a block diagram illustrating an electronic device according to some example embodiments.

[0048] Figure 2 This is a flowchart illustrating a method for training a neural network for image processing according to some example embodiments.

[0049] Figure 3A An example of generating a second image according to some example embodiments is shown. Figure 3B An example of generating a third image according to some example embodiments is shown.

[0050] Figure 4 A flowchart illustrating a method for generating pseudo-labels according to some example embodiments is shown.

[0051] Figure 5 A flowchart illustrating a method for generating prediction results according to some example embodiments is shown.

[0052] Figure 6 A flowchart illustrating a method for determining loss according to some example embodiments is shown.

[0053] Figure 7 A flowchart illustrating a method for updating a student network based on loss according to some example embodiments is shown.

[0054] Figure 8 A block diagram of an electronic device according to another example embodiment is shown. Detailed Implementation

[0055] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, except for those that must occur in a specific order, but may be changed as will become clear upon understanding the disclosure of this application. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0056] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0057] The structural or functional descriptions of the examples disclosed herein are intended for illustrative purposes only, and the examples may be implemented in various forms. The examples are not intended to be limiting, but rather to include various modifications, equivalents, and substitutions within the scope of the claims.

[0058] Although the terms “first” or “second” are used to describe various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the claims based on the concept of this disclosure, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0059] It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected to or combined with the other component, or there may be an intermediate component.

[0060] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0061] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized manner.

[0062] The examples will be described in detail below with reference to the accompanying drawings. Regarding the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be represented by the same reference numerals, and redundant descriptions will be omitted.

[0063] Figure 1 This is a block diagram illustrating an electronic device according to some example embodiments.

[0064] like Figure 1 As shown, electronic device 100 according to some example embodiments may include processor 110 and storage device 120.

[0065] Electronic device 100 may correspond to a computing device. For example, as a non-limiting example, electronic device 100 may be a personal computer (PC), a server, a mobile device, etc., and may also correspond to an autonomous vehicle, robot, smartphone, tablet device, augmented reality (AR) device, Internet of Things (IoT) device, and similar device that performs image recognition by executing a neural network, or a device installed in an autonomous vehicle, robot, smartphone, tablet device, augmented reality (AR) device, Internet of Things (IoT) device, and similar device that performs image recognition by executing a neural network. However, this disclosure is not limited thereto, and electronic device 100 may correspond to various other types of devices.

[0066] Storage device 120 stores data or information for processing operations of electronic device 100. For example, storage device 120 may store training data for training a neural network. Furthermore, storage device 120 may store instructions to be executed by processor 110. Storage device 120 includes non-transitory computer-readable storage media (e.g., random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), magnetic hard disk, optical disk, flash memory, electrically programmable read-only memory (EPROM), or other types of non-transitory computer-readable storage media known in the art).

[0067] Processor 110 controls the overall operation of electronic device 100 and performs functions and / or executes instructions within electronic device 100. Processor 110 can execute methods for training neural networks for image processing. For example, processor 110 can typically control electronic device 100 by executing instructions stored in storage device 120. Processor 110 can be implemented by a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), neural processing unit (NPU), etc., as non-limiting examples disposed in electronic device 100.

[0068] The following is combined Figure 2 This describes a method for training a neural network for image processing according to some example embodiments.

[0069] Figure 2 This is a flowchart illustrating a method for training a neural network for image processing according to some example embodiments.

[0070] like Figure 2 As shown, in step S210, the processor 110 can obtain the first output of each of at least two layers of the teacher network based on the first image.

[0071] For example, the first image could be an image obtained from the ImageNet database. However, the example is not limited to this; the first image could also be an image obtained from any other database or using various other methods.

[0072] The processor 110 can input each second image obtained based on the first image into the teacher network to obtain a first output for each of at least two layers in the teacher network corresponding to each second image.

[0073] According to some example embodiments, the second image may be the first image. In this case, the processor 110 may input the first image into the teacher network to obtain a first output at each of at least two layers of the teacher network.

[0074] According to other example embodiments, processor 110 can generate at least one second image by performing only weak enhancement processing on the first image.

[0075] According to some other example embodiments, processor 110 can generate multiple second images by performing additional processing on a first image after weak enhancement processing. References will follow. Figure 3A This describes an example of generating a second image by performing additional processing on a first image that has undergone weak enhancement.

[0076] The enhancement processing performed on the image may include at least one of the following, but is not limited to: random image cropping, image scaling, image color change, random image grayscale change, image Gaussian blur, random image flipping, etc.

[0077] In step S220, the processor 110 may generate a pseudo-label corresponding to each first output based on each first output.

[0078] The processor 110 can generate a pseudo-label corresponding to each first output based on the similarity between each first output and the historical outputs of the teacher network layers corresponding to each first output. The historical outputs of each layer of the teacher network may include outputs generated from each layer of the teacher network during historical (or previous) execution of the teacher network.

[0079] However, this is not the only example; processor 110 may use other methods to generate pseudo-labels for each first output.

[0080] According to some example embodiments, the processor 110 can generate pseudo-labels for each second image corresponding to each first output based on the similarity between each first output corresponding to each second image and the historical outputs of corresponding layers of at least two layers.

[0081] In step S230, the processor 110 may obtain a second output based on the first image, which has been processed by at least one layer in the student network, including the output layer.

[0082] The processor 110 can input each third image obtained based on the first image into the student network to obtain the output of each layer of the student network corresponding to each third image, including at least one layer of the output layer, and merge the outputs of each third image corresponding to each layer into (e.g., concatenate the outputs of each third image corresponding to each layer as) a second output corresponding to each third image.

[0083] According to some example embodiments, the third image may be the first image. In this case, the processor 110 may input the first image into the student network to obtain the output of each layer in the student network, including at least one layer with an output layer.

[0084] According to other example embodiments, processor 110 can generate at least one third image by performing strong enhancement processing only on the first image.

[0085] According to some other example embodiments, processor 110 can generate multiple third images by performing additional processing on a first image after strong enhancement processing. References will follow. Figure 3B This describes an example of generating multiple third images by performing additional processing on a first image that has undergone strong enhancement.

[0086] In some example embodiments, the number of second images is the same as the number of third images.

[0087] In step S240, the processor 110 may generate prediction results corresponding to each of the at least two layers based on each second output.

[0088] The processor 110 can generate prediction results corresponding to each of the at least two layers based on the similarity between the historical outputs corresponding to each of the at least two layers of the teacher network and each second output.

[0089] However, the examples are not limited to this. The processor 110 may use other methods in the prior art to generate pseudo-labels and prediction results, and this application does not impose any specific restrictions on this.

[0090] According to some example embodiments, the processor 110 can generate a prediction result for each third image corresponding to each of the at least two layers based on the similarity between the historical output corresponding to each of the at least two layers of the teacher network and the second output corresponding to each third image.

[0091] In step S250, the processor 110 may update the student network based on the pseudo-labels and prediction results.

[0092] The processor 110 can determine the loss based on the pseudo-label and the prediction result, and use the determined loss to update the parameters of the student network so that the loss determined based on the updated student network parameters (e.g., re-execute steps S230 to S240 based on the updated student network parameters to obtain the prediction result again, and determine the loss based on the pseudo-label and the re-obtained prediction result) is minimized or less than a predetermined threshold.

[0093] For example, the loss includes the difference between each pseudo-label and the prediction corresponding to each pseudo-label. Each pseudo-label and the prediction corresponding to each pseudo-label correspond to the same layer in at least two layers of the teacher network.

[0094] In some embodiments, where the second and third images are the first images, or where the processor 110 performs weak enhancement processing only on the first image to generate the second image and strong enhancement processing only on the first image to generate the third image, the processor 110 may determine the loss based on the difference between each prediction result of each third image and the pseudo-label of a second image corresponding to each prediction result.

[0095] It's important to understand that, to improve training accuracy, this application no longer uses a single label and prediction result to calculate the loss, but instead introduces a method that calculates the loss using multiple labels and corresponding prediction results. For example, based on an image 1 input to the teacher network, assuming three layers T1, T2, and T3 are selected, the first outputs of layers T1, T2, and T3 will be obtained respectively. Based on these first outputs and the historical outputs of these three layers, pseudo-labels for image 1 corresponding to these three layers will be calculated. Similarly, based on an image 1 input to the student network, assuming layers S1 and the output layer are selected, after obtaining the outputs of layers S1 and the output layer respectively, the outputs of layers S1 and the output layer are merged to obtain the second output of image 1. Then, using the second output of image 1 and the historical outputs of layers T1, T2, and T3 of the teacher network, prediction results 1, 2, and 3 corresponding to layers T1, T2, and T3 are calculated for image 1. Of course, the same processing method applies to multiple images.

[0096] In one exemplary embodiment, Kullback-Leibler divergence can be used to calculate the difference between the prediction and the pseudo-label. However, this example is not limited to this, and other methods can be used to calculate the difference between the prediction and the pseudo-label.

[0097] In some embodiments, when processor 110 generates a second image by performing additional processing on a first image after weak enhancement processing and generates a third image by performing additional processing on a first image after strong enhancement processing, processor 110 may obtain a final pseudo-label based on the pseudo-label obtained in step S220, and determine the loss based on the difference between the prediction result and the final pseudo-label. (See below for further details.) Figure 6 Detailed description.

[0098] Figure 3A An example of generating a second image according to some example embodiments is shown.

[0099] According to some example embodiments, the processor 110 may also generate a second image by fusing the first image that has undergone enhancement processing with an additional image that has undergone enhancement processing.

[0100] For example, such as Figure 3A As shown, in step S310, the processor 110 can perform weak enhancement processing on the first image to generate multiple weak first images.

[0101] In step S320, the processor 110 may acquire additional images.

[0102] For example, an additional image is an image that is different from the first image.

[0103] In step S330, the processor 110 may perform weak enhancement processing on the additional images to generate multiple weak additional images.

[0104] In step S340, the processor 110 can perform one-to-one image fusion on multiple weak first images and multiple weak additional images to generate multiple second images.

[0105] For example, the processor 110 may adjust the size of each weak additional image to be multiplied by the size of the corresponding weak first image to be fused and a predetermined ratio, and overlay each size-adjusted weak additional image on a predetermined region of the corresponding weak first image to be fused (e.g., replace the image of the predetermined region of the weak first image with the processed weak additional image) to generate multiple second images. For example, the predetermined ratio is less than 0.5.

[0106] Figure 3B An example of generating a third image according to some example embodiments is shown.

[0107] According to some example embodiments, the processor 110 can also generate a plurality of third images by fusing a first image that has undergone enhancement processing with an additional image that has undergone enhancement processing.

[0108] like Figure 3BAs shown, in step S311, the processor 110 can perform strong enhancement processing on the first image to generate multiple strong first images of different sizes.

[0109] In step S321, the processor 110 may perform strong enhancement processing on the additional images to generate multiple strong additional images of different sizes.

[0110] In step S331, the processor 110 can generate multiple third images by performing one-to-one image fusion on multiple strong first images and multiple strong additional images.

[0111] For example, the processor 110 may adjust the size of each strong additional image to the product of the size of the corresponding strong first image to be merged and a predetermined ratio, and overlay each size-adjusted strong additional image on a predetermined region of the corresponding strong first image to be merged (e.g., replace the image of the predetermined region of the corresponding strong first image to be merged with each size-adjusted strong additional image) to generate a plurality of third images.

[0112] Here, the number of each of the strong first image, strong additional image, and third image can be equal to or greater than 2.

[0113] In this disclosure, by performing image augmentation on the original samples and image fusion on the augmented images, a sufficient number of training samples can be generated to increase the training difficulty, thereby improving the accuracy of training neural networks for image processing and solving the problems of dependence on a large number of samples and large hardware resource requirements.

[0114] In this disclosure, noise in pseudo-labels is reduced by inputting samples with varying degrees of enhancement into the teacher network and the student network.

[0115] Although the method for performing image fusion according to an example embodiment of the present disclosure has been shown above, the present disclosure is not limited thereto, and other existing methods for performing image fusion may also be used.

[0116] Figure 4 A flowchart illustrating a method for generating pseudo-labels according to some example embodiments is shown.

[0117] like Figure 4 As shown, in step S410, the processor 110 can calculate the similarity between the first output of each of at least two layers of the teacher network and the historical output of the corresponding layer of the teacher network.

[0118] For example, processor 110 may calculate a first similarity between a first output of a first predetermined layer of at least two layers of a teacher network and a historical output of the first predetermined layer of the teacher network, and similarly, processor 110 may calculate a first similarity between a first output of each of the remaining layers of at least two layers of the teacher network and a historical output of the corresponding layer of the remaining layers of the teacher network.

[0119] For example, the historical output of the first predetermined layer of at least two layers of the teacher network may include the output generated from the first predetermined layer of the teacher network when the teacher network was historically executed.

[0120] According to some example embodiments, the processor 110 can calculate the similarity between the first output of each of the at least two layers of the teacher network and the historical output of the corresponding layer of the at least two layers in the teacher network by calculating the Euclidean distance between the first output of each of the at least two layers of the teacher network and the historical output of the corresponding layer of the at least two layers in the teacher network.

[0121] For example, processor 110 can calculate a first Euclidean distance between the first output of a first predetermined layer of at least two layers of the teacher network and each historical output of the first predetermined layer of the teacher network, as a first similarity.

[0122] In step S420, the processor 110 may perform momentum normalization on the similarity between the first output of each of the at least two layers of the teacher network and the historical output of the corresponding layer of the at least two layers of the teacher network, so as to generate pseudo-labels corresponding to each first output.

[0123] For example, processor 110 can generate a first pseudo-label corresponding to the first output of a first predetermined layer of at least two layers of the teacher network by performing momentum normalization on the first similarity.

[0124] According to some example embodiments, the processor 110 may normalize the first Euclidean distance using at least one of a first parameter determined based on a statistical value of the first Euclidean distance and a second parameter determined based on an extreme value of the first Euclidean distance.

[0125] The first parameter is calculated using the following equation (1):

[0126] M1=(1-M)×M2+M×M3 (1)

[0127] In equation (1), M1 represents the first parameter, M2 represents the statistical value of the first Euclidean distance, M3 represents the statistical value of the first Euclidean distance calculated from the previous output generated from the first predetermined layer of the teacher network during the last execution of the teacher network, and M is the predetermined momentum value.

[0128] The second parameter is calculated using the following equation:

[0129] L1=(1-M)×L2+M×L3 (2)

[0130] In equation (2), L1 represents the second parameter, L2 represents the extreme value of the first Euclidean distance, and M3 represents the extreme value of the first Euclidean distance calculated from the previous output generated from the first predetermined layer of the teacher network during the last execution of the teacher network.

[0131] In one example, the statistical value of the first Euclidean distance can be the mean or variance of the first Euclidean distance, and the extreme value of the first Euclidean distance can be the maximum or minimum value of the first Euclidean distance.

[0132] For example, processor 110 may perform max-min normalization, mean-variance normalization, or minimum normalization on the first Euclidean distance, and use the above methods to determine the parameters for performing max-min normalization, mean-variance normalization, or minimum normalization.

[0133] For example, when processor 110 performs minimum normalization on the above similarity or Euclidean distance, processor 110 calculates pseudo-labels using the following equation (3).

[0134]

[0135] In equation (3), Aij represents the ij-th Euclidean distance between the first output of the i-th predetermined layer in the teacher network and the j-th output generated from the i-th predetermined layer during the j-th execution of the teacher network in history; Pij represents the ij-th probability value of the pseudo-label corresponding to the ij-th Euclidean distance in the pseudo-label corresponding to the first output of the i-th predetermined layer; MeanAi represents the first parameter determined based on the average of the Euclidean distances determined between the first output of the i-th predetermined layer and each historical output of the i-th predetermined layer; and MinAi represents the second parameter determined based on the minimum of the Euclidean distances determined between the first output of the i-th predetermined layer and each historical output of the i-th predetermined layer.

[0136] In this disclosure, pseudo-labels for images are generated based on the outputs of different layers of the teacher network, which can produce pseudo-labels for images at different semantic levels, thereby improving the quality of pseudo-label generation and increasing the accuracy of training the student network.

[0137] In this disclosure, the stability of the training student network is improved by using the aforementioned momentum normalization method, thereby reducing the accuracy loss during the training process.

[0138] Although the method for generating pseudo-tags according to an example embodiment of the present disclosure has been shown above, the present disclosure is not limited thereto, and other existing methods for generating pseudo-tags may also be used.

[0139] Figure 5A flowchart illustrating a method for generating prediction results according to some example embodiments is shown.

[0140] like Figure 5 As shown, in S510, the processor 110 can input each third image into the student network, obtain the output of at least one layer in the student network including the output layer, and merge the output of at least one layer into a second output corresponding to each third image.

[0141] According to some example embodiments, processor 110 may input a third image into a student network and obtain the output of at least one layer in the student network, including an output layer. Here, the at least one layer in the student network including the output layer is adjacent. That is, the at least one layer is a consecutive layer. For example, processor 110 may obtain the output of an output layer in the student network, or obtain the output of an output layer in the student network and the output of at least one layer adjacent to or connected to the output layer.

[0142] According to some example embodiments, when the processor 110 only obtains the output of the output layer in the student network, the processor 110 can use the output of the output layer as the second output corresponding to the aforementioned third image. When the processor 110 obtains the output of the output layer and the output of at least one layer adjacent to or connected to the output layer, the processor 110 can combine the output of the output layer with the output of at least one layer adjacent to or connected to the same output layer to form the second output corresponding to the aforementioned third image.

[0143] For example, processor 110 can combine the outputs of at least one layer adjacent to or connected to the output layer and the output of the output layer according to the order of the output layers in the student network. For example, when the output layer is the nth layer of the student network, and at least one layer adjacent to or connected to the output layer is the (n-1)th layer of the student network, the output of the nth layer is (m1, ..., m...). x And the output of the (n-1)th layer is (x1, ..., x). s When the output of the output layer is combined with the output of at least one adjacent layer, the result is (x1, ..., x). s m1, ..., m x (Each of s and x can be a positive integer greater than 1).

[0144] In S520, the processor 110 can generate prediction results corresponding to at least two layers based on the second output corresponding to each third image.

[0145] According to some example embodiments, processor 110 can calculate the similarity between the historical output of each of at least two layers in the teacher network and the second output corresponding to each third image, as a prediction result for each third image corresponding to at least two layers.

[0146] For example, processor 110 can calculate the similarity between the historical output of each of at least two layers of the teacher network and the second output corresponding to a third image, as a prediction result of the third image corresponding to at least two layers.

[0147] Since the dimension of the historical output of each layer of at least two layers may be different from the dimension of the second output corresponding to each third image, this may lead to inaccurate similarity between the calculated historical output of each layer of at least two layers and the second output corresponding to each third image. Therefore, in order to solve the above problem, the dimension of the historical output of each layer of at least two layers can be transformed.

[0148] For example, when the dimension of the second output corresponding to each third image is different from the dimension of the historical output of each layer of at least two layers in the teacher network, the processor 110 can convert the dimension of the historical output of each layer of at least two layers in the teacher network to the same dimension as the dimension of the second output corresponding to each third image. Furthermore, the processor 110 can calculate the similarity between the historical output of each layer of at least two layers in the converted teacher network and the second output corresponding to each third image, as a prediction result for each third image corresponding to at least two layers in the teacher network.

[0149] For example, when the dimension of the second output corresponding to each third image is different from the dimension of the historical output of the first predetermined layer of at least two layers of the teacher network, the processor 110 can convert the dimension of the historical output of the first predetermined layer to the same dimension as the dimension of the second output corresponding to each third image, and calculate the similarity between the converted historical output of the first predetermined layer and the second output corresponding to each third image as the prediction result of each third image corresponding to the first predetermined layer.

[0150] According to some example embodiments, the processor 110 can calculate the similarity between the historical outputs of each of the at least two layers of the teacher network and the second output corresponding to each third image by calculating the inner product of each historical output of each of the at least two layers of the teacher network and the second output corresponding to each third image. However, the examples are not limited to this; the processor 110 can also calculate the similarity between the historical outputs of each of the at least two layers of the teacher network and the second output corresponding to each third image by calculating the Euclidean distance between each historical output of each of the at least two layers of the teacher network and the second output corresponding to each third image, or by using other methods.

[0151] Although the method for generating prediction results according to an example embodiment of the present disclosure has been shown above, the present disclosure is not limited thereto, and other existing methods for generating prediction results may also be used.

[0152] Figure 6 A flowchart illustrating a method for determining loss according to some example embodiments is shown.

[0153] like Figure 6 As shown, in step S610, the processor 110 can adjust the size of each weak first image to the product of the size of the corresponding weak additional image to be fused and a predetermined ratio, and overlay each size-adjusted weak first image on a predetermined area of ​​the corresponding weak additional image to be fused to generate an intermediate image corresponding to each second image.

[0154] Each second image and its corresponding intermediate image are generated based on the same weak first image and weak supplementary image.

[0155] In step S620, the processor 110 can input each intermediate image into the teacher network and obtain the third output of each of at least two layers in the teacher network corresponding to each intermediate image.

[0156] In step S630, the processor 110 can generate a pseudo-label corresponding to each third output based on each third output.

[0157] According to some example embodiments, processor 110 may generate pseudo-labels for each third output in a manner similar to that used to generate pseudo-labels for each first output.

[0158] In step S640, the processor 110 can fuse the pseudo-label corresponding to each first output with the pseudo-label corresponding to each third output to generate the final pseudo-label corresponding to each first output.

[0159] According to some example embodiments, processor 110 can use the following equation (4) to generate the final pseudo-tag:

[0160] Yij=(1-r)×Yaij+r×Ybij (4)

[0161] In equation (4), Yij represents the final pseudo-label corresponding to the i-th second image in the j-th layer of at least two layers, r represents a predetermined ratio, Yaij represents the pseudo-label corresponding to the first output of the i-th second image in the j-th layer of at least two layers, and Ybij represents the pseudo-label corresponding to the third output of the intermediate image corresponding to the i-th second image in the j-th layer of at least two layers.

[0162] In step S650, the processor 110 may determine the loss based on the difference between the prediction result and the final pseudo-label.

[0163] For example, the loss includes the difference between each final pseudo-label and each prediction corresponding to that final pseudo-label. Each final pseudo-label and each prediction corresponding to that final pseudo-label corresponds to the same layer in at least two layers. For example, KL divergence can be used to calculate the difference between the prediction and the final pseudo-label. However, the example is not limited to this, and other methods can be used to calculate the difference between the prediction and the final pseudo-label.

[0164] Although the method for determining loss according to an example embodiment of the present disclosure has been shown above, the present disclosure is not limited thereto, and other existing methods for determining loss may also be used.

[0165] Figure 7 A flowchart illustrating a method for updating a student network based on loss according to some example embodiments is shown.

[0166] In at least one third image generated by performing strong enhancement processing on the first image, there may be at least two third images of the same size but different sizes from the other third images. If the total loss used to update the student network is determined by directly summing the losses corresponding to each third image, this may lead to a decrease in the accuracy of training the student network. Therefore, to solve the above problem, the total loss used to update the student network can be determined in the following way.

[0167] In step S710, the processor 110 may average the losses corresponding to the third image of size i, and use this average as the i-th loss corresponding to the third image of size i. For example, i can be an integer from 1 to n, and n represents the number of different sizes of the third images and can be an integer greater than 1.

[0168] In step S720, the processor 110 can update the student network by summing the first loss to the nth loss as the total loss.

[0169] For example, when multiple third images consist of third image a, third image b, and third image c, and third image a and third image b have a first size, and third image c has a second size, processor 110 can average the losses of third image a and third image b as the loss corresponding to the third image of the first size, determine the loss corresponding to third image c as the loss corresponding to the third image of the second size, and sum the average of the losses of third image a and third image b with the loss corresponding to third image c as the total loss. Although multiple third images with two sizes are shown here, the example is not limited to this; multiple third images may have one size or at least three sizes.

[0170] Those skilled in the art will understand that various known methods can be used to update the student network based on the total loss, and this disclosure makes no limitation.

[0171] Figure 8 A block diagram of an electronic device according to another example embodiment is shown.

[0172] like Figure 8 As shown, in some example embodiments, electronic device 200 includes controller 210. Those skilled in the art will understand that electronic device 200 may additionally include other components, and the components included in electronic device 200 may be separated or combined.

[0173] According to some example embodiments, controller 210 can process neural networks used for image processing (e.g., using...). Figure 2 The method shown is for training a neural network for image processing.

[0174] In one example, controller 210 may include backbone network module 211 and knowledge distillation module 212.

[0175] The backbone network module 211 includes a teacher network and a student network, wherein the backbone network module 211 is configured to obtain the first output of each of at least two layers of the teacher network based on the first image.

[0176] For example, the first image could be an image obtained from the ImageNet database. However, the example is not limited to this; the first image could also be an image obtained from any other database or using various other methods.

[0177] The backbone network module 211 can input each second image obtained based on the first image into the teacher network to obtain the first output of each of the at least two layers of the teacher network corresponding to each second image.

[0178] According to some example embodiments, the second image may be the first image. In this case, the backbone network module 211 may input the first image into the teacher network to obtain a first output at each of at least two layers in the teacher network.

[0179] According to other example embodiments, the electronic device 200 (e.g., the controller 210 of the electronic device 200) also includes an image processing module (not shown), which can generate at least one second image by performing weak enhancement processing on only the first image. In one example, the image processing module may be included in or separate from the backbone network module 211.

[0180] According to some other example embodiments, the image processing module can generate multiple second images by performing additional processing on a first image after weak enhancement processing. This has been referred to above. Figure 3A The description provides an example of generating a second image by performing additional processing on a first image that has undergone weak enhancement; repeated descriptions will be omitted here to avoid redundancy.

[0181] The enhancement processing performed on the image may include at least one of the following, but is not limited to: random image cropping, image scaling, image color change, random image grayscale change, image Gaussian blur, random image flipping, etc.

[0182] The backbone network module 211 can generate pseudo-labels corresponding to each first output based on each first output.

[0183] According to some example embodiments, the backbone network module 211 generates pseudo-labels for each second image corresponding to each first output based on the similarity between each first output corresponding to each second image and the historical outputs of corresponding layers of at least two layers.

[0184] The backbone network module 211 can obtain a second output based on the first image, which has been processed by at least one layer in the student network, including the output layer.

[0185] The backbone network module 211 can input each third image obtained based on the first image into the student network to obtain the output of each layer of at least one layer in the student network, including the output layer, corresponding to each third image, and merge the outputs of each third image corresponding to each layer into (e.g., concatenate the outputs of each third image corresponding to each layer as) a second output corresponding to each third image.

[0186] According to some example embodiments, the third image may be the first image. In this case, the backbone network module 211 may input the first image into the student network to obtain the output of each layer of the student network, including at least one layer of the output layer.

[0187] According to other example embodiments, the image processing module can generate at least one third image by performing strong enhancement processing only on the first image.

[0188] According to some other example embodiments, the image processing module can generate multiple third images by performing additional processing on a first image that has undergone strong enhancement processing. This has been referred to above. Figure 3B The description provides an example of generating multiple third images by performing additional processing on a first image that has undergone strong enhancement. Repeated descriptions will be omitted here to avoid redundancy.

[0189] In some example embodiments, the number of second images is the same as the number of third images.

[0190] The backbone network module 211 can generate prediction results corresponding to each of the at least two layers based on each second output.

[0191] The backbone network module 211 can generate prediction results corresponding to each of the at least two layers based on the similarity between the historical outputs corresponding to each layer of the teacher network and each second output.

[0192] However, the examples are not limited to this. Backbone network module 211 may use other methods in the prior art to generate pseudo-labels and prediction results, and this application does not impose any specific restrictions on this.

[0193] According to some example embodiments, the backbone network module 211 can generate a prediction result for each third image corresponding to each of the at least two layers based on the similarity between the historical output corresponding to each of the at least two layers of the teacher network and the second output corresponding to each third image.

[0194] The knowledge distillation module 212 can update the student network based on pseudo-labels and prediction results.

[0195] The knowledge distillation module 212 can determine the loss based on the pseudo-labels and the prediction results, and use the determined loss to update the parameters of the student network so that the loss determined based on the updated student network parameters (e.g., obtaining the prediction results again based on the updated student network parameters, and determining the loss based on the pseudo-labels and the obtained prediction results again) is minimized or less than a predetermined threshold.

[0196] For example, the loss includes the difference between each pseudo-label and the prediction corresponding to each pseudo-label. Each pseudo-label and the prediction corresponding to each pseudo-label correspond to the same layer in at least two layers of the teacher network.

[0197] In some embodiments, where the second and third images are the first images, or where the image processing module performs weak enhancement processing only on the first image to generate the second image and strong enhancement processing only on the first image to generate the third image, the knowledge distillation module 212 may determine the loss based on the difference between each prediction result of each third image and the pseudo-label of a second image corresponding to each prediction result.

[0198] In one exemplary embodiment, KL divergence can be used to calculate the difference between the prediction and the pseudo-label. However, the example is not limited to this, and other methods can be used to calculate the difference between the prediction and the pseudo-label.

[0199] In some embodiments, when the image processing module generates a second image by performing additional processing on a first image after weak enhancement processing and generates a third image by performing additional processing on a first image after strong enhancement processing, the knowledge distillation module 212 can obtain a final pseudo-label based on the pseudo-label corresponding to each of the first outputs, and determine the loss based on the difference between the prediction result and the final pseudo-label. (The above has already been referred to...) Figure 6 A detailed description of this will be omitted here to avoid redundancy.

[0200] The devices, units, modules, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more of the hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, single-instruction single-data (SISD) multiple processing, single-instruction multiple-data (SIMD) multiple processing, multiple-instruction single-data (MISD) multiple processing, and multiple-instruction multiple-data (MIMD) multiple processing.

[0201] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

[0202] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art can readily write instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0203] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drive (HDD), solid-state drive (SSD), flash memory, card memory (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disk, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

[0204] Although this disclosure has been specifically shown and described with reference to exemplary embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. A method for training a neural network for image processing, comprising: Based on the first image, obtain the first output of each of at least two layers of the teacher network; Based on each first output, generate a pseudo-label corresponding to each first output; Based on the first image, a second output is obtained after being processed by at least one layer in the student network, including the output layer. Based on each of the second outputs, a prediction result corresponding to each of the at least two layers is generated; The student network is updated based on the pseudo-labels and prediction results. The step of generating pseudo-labels corresponding to each first output includes: generating pseudo-labels corresponding to each first output based on the similarity between each first output and the historical outputs of the layers of the teacher network corresponding to each first output. The step of generating a prediction result corresponding to each of the at least two layers based on each second output includes: calculating the similarity between the historical output of each of the at least two layers in the teacher network and each second output, as the prediction result corresponding to each of the at least two layers.

2. The method according to claim 1, wherein, The method further includes: Perform weak enhancement processing on the first image to generate at least one second image. A strong enhancement process is performed on the first image to generate at least one third image, wherein the number of the at least one second image is the same as the number of the at least one third image; The step of obtaining the first output of each of at least two layers of the teacher network based on the first image includes: Based on each second image, obtain the first output of each of the at least two layers in the teacher network corresponding to each second image; The step of obtaining a second output based on the first image, after processing by at least one layer in the student network including an output layer, includes: Based on each third image, obtain the output of each layer in the student network corresponding to each third image, including at least one layer with an output layer, and merge the output of each third image corresponding to each layer into a second output corresponding to each third image.

3. The method according to claim 2, wherein, The steps for generating at least one second image include: Perform weak enhancement processing on the first image to generate multiple weak first images. Weak enhancement processing is performed on the additional images to generate multiple weak additional images. Perform one-to-one image fusion on the plurality of weak first images and the plurality of weak additional images to generate a plurality of second images; The step of generating at least one third image includes: Perform strong enhancement processing on the first image to generate multiple strong first images of different sizes. Strong enhancement processing is performed on the attached images to generate multiple strongly enhanced attached images of different sizes. Multiple third images are generated by performing one-to-one image fusion on the multiple strong first images and the multiple strong additional images.

4. The method according to claim 3, wherein, The step of generating the plurality of second images by performing one-to-one image fusion on the plurality of weak first images and the plurality of weak additional images includes: The size of each weak additional image is adjusted to be the product of the size of the corresponding weak first image to be fused and a predetermined ratio, and each size-adjusted weak additional image is superimposed on a predetermined area of ​​the corresponding weak first image to be fused to generate the plurality of second images. The step of generating the plurality of third images by performing one-to-one image fusion on the plurality of strong first images and the plurality of strong additional images includes: The size of each strong additional image is adjusted to the product of the size of the corresponding strong first image to be merged and a predetermined ratio, and each size-adjusted strong additional image is superimposed on a predetermined area of ​​the corresponding strong first image to be merged to generate the plurality of third images.

5. The method according to claim 4, wherein, The method further includes: The size of each weak first image is adjusted to the product of the size of the corresponding weak additional image to be fused and a predetermined ratio, and each size-adjusted weak first image is superimposed on a predetermined area of ​​the corresponding weak additional image to be fused to generate an intermediate image corresponding to each second image, wherein each second image and the intermediate image corresponding to each second image are generated based on the same weak first image and weak additional image. Based on each intermediate image, obtain the third output of each of the at least two layers in the teacher network corresponding to each intermediate image; Based on each third output, generate a pseudo-label corresponding to each third output; The pseudo-tags corresponding to each first output and each third output are fused to generate the final pseudo-tag corresponding to each first output. The steps for updating the student network based on the pseudo-labels and prediction results include: The student network is updated based on the final pseudo-labels and prediction results.

6. The method according to claim 5, wherein, The step of generating the final pseudo-label corresponding to each of the first outputs includes: using the following equation to generate the final pseudo-label corresponding to each of the first outputs: Yij represents the final pseudo-label corresponding to the i-th second image in the j-th layer of the at least two layers, r represents a predetermined ratio, Yaij represents the pseudo-label corresponding to the first output of the i-th second image in the j-th layer of the at least two layers, and Ybij represents the pseudo-label corresponding to the third output of the intermediate image corresponding to the i-th second image in the j-th layer of the at least two layers, wherein the predetermined ratio is less than 0.

5.

7. The method according to claim 1, wherein, The at least one layer in the student network, including the output layer, is adjacent.

8. The method according to claim 2, wherein, The step of updating the student network based on the pseudo-labels and prediction results includes: determining the loss based on the pseudo-labels and prediction results, and updating the student network based on the loss. The loss includes the difference between each prediction result of each third image and the pseudo label corresponding to each prediction result of any second image, wherein each prediction result and the pseudo label corresponding to each prediction result correspond to the same layer in the at least two layers.

9. The method according to claim 1, wherein, The steps for generating the pseudo-label corresponding to each first output include: Calculate the similarity between each first output and the historical outputs of the layer corresponding to each first output. Momentum normalization is performed on the similarity to generate probability values ​​as pseudo-labels for each first output. The historical output of the layer corresponding to each first output includes the output generated from the layer corresponding to each first output of the teacher network when the teacher network was executed in the past.

10. The method according to claim 9, in, The step of calculating the similarity between each first output and the historical outputs of the layer corresponding to each first output includes: calculating the Euclidean distance between each first output and the historical outputs of the layer corresponding to each first output as the similarity. The step of performing momentum normalization on the similarity includes: The Euclidean distance is normalized using at least one of a first parameter determined by statistics based on the Euclidean distance and a second parameter determined by extreme values ​​of the Euclidean distance.

11. The method according to claim 1, wherein, The step of calculating the similarity between the historical output of each of the at least two layers in the teacher network and each of the second outputs includes: When the dimension of each second output is different from the dimension of the historical output of each layer of at least two layers, the dimension of the historical output of each layer of at least two layers is converted into the dimension of each second output; The similarity between the historical output of each layer of the at least two layers after transformation and each second output is calculated as the prediction result corresponding to each layer of the at least two layers.

12. The method according to claim 8, wherein, The steps for updating the student network based on the loss include: The average of the losses corresponding to the third image of the first size is calculated and used as the loss corresponding to the third image of the first size. The average of the losses corresponding to the third image of the second size is calculated and used as the loss corresponding to the third image of the second size. The student network is updated using the sum of the losses corresponding to the third image of the first size and the third image of the second size as the total loss.

13. An electronic device for training a neural network for image processing, comprising: The backbone network module, which includes the teacher network and the student network, is configured as follows: Based on the first image, obtain the first output of each of at least two layers of the teacher network. Based on each first output, generate a pseudo-label corresponding to each first output. Based on the first image, a second output is obtained after processing by at least one layer in the student network, including the output layer. Based on each of the second outputs, a prediction result corresponding to each of the at least two layers is generated; The knowledge distillation module is configured to update the student network based on the pseudo-labels and prediction results. The backbone network module is configured to generate pseudo-labels corresponding to each first output based on the similarity between each first output and the historical outputs of the layers of the teacher network corresponding to each first output. The backbone network module is configured to: calculate the similarity between the historical output of each of the at least two layers in the teacher network and each of the second outputs, as a prediction result corresponding to each of the at least two layers.

14. The electronic device of claim 13, further comprising an image processing module. in, The image processing module is configured to: perform weak enhancement processing on a first image to generate at least one second image, and perform strong enhancement processing on the first image to generate at least one third image, wherein the number of the at least one second image is the same as the number of the at least one third image. The backbone network module is further configured to: based on each second image, obtain the first output of each layer in at least two layers of the teacher network corresponding to each second image; based on each third image, obtain the output of each layer in at least one layer including the output layer of the student network corresponding to each third image, and merge the output of each layer corresponding to each third image into the second output corresponding to each third image.

15. The electronic device according to claim 14, wherein, The image processing module is also configured as follows: Perform weak enhancement processing on the first image to generate multiple weak first images. Weak enhancement processing is performed on the additional images to generate multiple weak additional images. Perform one-to-one image fusion on the plurality of weak first images and the plurality of weak additional images to generate a plurality of second images. Perform strong enhancement processing on the first image to generate multiple strong first images of different sizes. Strong enhancement processing is performed on the attached images to generate multiple strongly enhanced attached images of different sizes. Multiple third images are generated by performing one-to-one image fusion on the multiple strong first images and the multiple strong additional images.

16. The electronic device according to claim 15, wherein, The image processing module is also configured as follows: The size of each weak additional image is adjusted to be the product of the size of the corresponding weak first image to be merged and a predetermined ratio. Each resized weak additional image is then overlaid on a predetermined area of ​​the corresponding weak first image to be merged, generating the plurality of second images. The size of each strong additional image is adjusted to the product of the size of the corresponding strong first image to be merged and a predetermined ratio, and each size-adjusted strong additional image is superimposed on a predetermined area of ​​the corresponding strong first image to be merged to generate the plurality of third images.

17. The electronic device according to claim 16, wherein, The image processing module is also configured as follows: The size of each weak first image is adjusted to be the product of the size of the corresponding weak additional image to be fused and a predetermined ratio. Each resized weak first image is then overlaid on a predetermined region of the corresponding weak additional image to be fused, generating an intermediate image corresponding to each second image. Each second image and its corresponding intermediate image are generated based on the same weak first image and weak additional image. The backbone network module is further configured to: obtain the third output of each of the at least two layers of the teacher network corresponding to each intermediate image; generate a pseudo-label corresponding to each third output based on each third output; and fuse the pseudo-labels corresponding to each first output and each third output to generate the final pseudo-label corresponding to each first output. The knowledge distillation module is also configured to update the student network based on the final pseudo-labels and prediction results.

18. The electronic device according to claim 17, wherein, The backbone network module is also configured to generate the final pseudo-label corresponding to each of the first outputs using the following equation: Yij represents the final pseudo-label corresponding to the i-th second image in the j-th layer of the at least two layers, r represents a predetermined ratio, Yaij represents the pseudo-label corresponding to the first output of the i-th second image in the j-th layer of the at least two layers, and Ybij represents the pseudo-label corresponding to the third output of the intermediate image corresponding to the i-th second image in the j-th layer of the at least two layers, wherein the predetermined ratio is less than 0.

5.

19. The electronic device according to claim 13, wherein, The at least one layer in the student network, including the output layer, is adjacent.

20. The electronic device according to claim 14, wherein, The knowledge distillation module is configured to: determine the loss based on the pseudo-labels and prediction results, and update the student network based on the loss. The loss includes the difference between each prediction result of each third image and the pseudo label corresponding to each prediction result of any second image, wherein each prediction result and the pseudo label corresponding to each prediction result correspond to the same layer in the at least two layers.

21. The electronic device according to claim 13, wherein, The backbone network module is configured as follows: Calculate the similarity between each first output and the historical outputs of the layer corresponding to each first output. Momentum normalization is performed on the similarity to generate probability values ​​as pseudo-labels for each first output. The historical output of the layer corresponding to each first output includes the output generated from the layer corresponding to each first output of the teacher network when the teacher network was executed in the past.

22. The electronic device according to claim 21, wherein, The backbone network module is configured as follows: The similarity is calculated as the Euclidean distance between each first output and the historical outputs of the layer corresponding to each first output. The Euclidean distance is normalized using at least one of a first parameter determined by statistics based on the Euclidean distance and a second parameter determined by extreme values ​​of the Euclidean distance.

23. The electronic device according to claim 13, wherein, The backbone network module is configured as follows: When the dimension of each second output is different from the dimension of the historical output of each layer of at least two layers, the dimension of the historical output of each layer of at least two layers is converted into the dimension of each second output; The similarity between the historical output of each layer of the at least two layers after transformation and each second output is calculated as the prediction result corresponding to each layer of the at least two layers.

24. The electronic device according to claim 20, wherein, The knowledge distillation module is configured as follows: The average of the losses corresponding to the third image of the first size is calculated and used as the loss corresponding to the third image of the first size. The average of the losses corresponding to the third image of the second size is calculated and used as the loss corresponding to the third image of the second size. The student network is updated using the sum of the losses corresponding to the third image of the first size and the third image of the second size as the total loss.

25. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 12.

Images