Learning device and testing device for training student neural network

Abstract

A machine learning and testing apparatus is provided. The apparatus applies a neural network operation of the first student neural network to the training image to generate first prediction information, applies a neural network operation of the second student neural network to the second prediction information, applies an error identification operation to the first integrated image to generate first error identification prediction information, applying an error identification operation to the second integrated image to generate second error identification prediction information, applying a network operation of the first teacher neural network to the training image to generate first pseudo-tag information, and applying a neural network operation of the second teacher neural network to the training image to generate second pseudo-tag information, the loss is back-propagated and parameters of the first and second student neural networks are updated.

Description

Technical Field

[0001] The present disclosure relates to a learning device for training a student neural network based on knowledge extraction using a plurality of teacher neural networks, a learning method thereof, a testing device using the learning device, and a testing method using the testing device. Background Art

[0002] Recently, with the development of computer vision technology based on deep neural networks in the field of autonomous driving, various artificial intelligence models have been studied, such as object detection, semantic segmentation, depth estimation, and lane detection.

[0003] For example, using the cameras of autonomous vehicles, semantic segmentation is variously used to recognize the vehicle’s surroundings and spaces, lines, free spaces, etc.

[0004] In addition, recently, computer vision techniques based on deep neural networks have constructed large datasets including high-quality labels, and have achieved significant performance improvements with the help of supervised learning using large datasets.

[0005] However, due to the significant cost, time, and labor required to build large training datasets with high-quality labels, the training dataset construction stage can create a bottleneck in the workflow even before AI models can be seriously trained.

[0006] Semi-supervised learning is proposed as a solution to the above problems. Semi-supervised learning refers to a learning scheme that uses both labeled and unlabeled data.

[0007] In this type of semi-supervised learning, pseudo-labeling schemes can be used with unlabeled data. Pseudo-labels can be generated by a neural network that is pre-trained using a large amount of training data.

[0008] However, because it is difficult to achieve perfect performance in all aspects (e.g., perfect accuracy, etc.) due to the nature of neural networks, and neural networks are susceptible to specific characteristics (e.g., because neural networks are susceptible to local regions and wide receptive fields), when pseudo labels generated by a neural network (e.g., a teacher neural network) are used to train a new neural network (e.g., a student neural network), the student neural network may have the same or similar vulnerabilities as the teacher neural network. Summary of the invention

[0009] The present disclosure can address the above-mentioned issues while maintaining any advantages achieved.

[0010] One aspect of the present disclosure provides a learning device for training a student neural network based on knowledge extraction using a plurality of teacher neural networks, a learning method thereof, a testing device using the learning device, and a testing method using the learning device.

[0011] Another aspect of the present disclosure provides a learning device for transferring the advantages of each of a plurality of teacher neural networks to a student neural network, a learning method thereof, a testing device using the learning device, and a testing method using the learning device.

[0012] Another aspect of the present disclosure provides a learning device for reducing the cost of constructing a training data set, a learning method thereof, a testing device using the learning device, and a testing method using the learning device.

[0013] Another aspect of the present disclosure provides a learning device for improving the performance of a student neural network based on knowledge extraction between multiple student neural networks, a learning method thereof, a testing device using the learning device, and a testing method using the learning device.

[0014] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and those skilled in the art to which the present disclosure belongs will clearly understand any other technical problems not mentioned herein from the following illustrations.

[0015] According to one or more example embodiments of the present disclosure, a learning device may include: one or more processors; and a memory. The memory may store instructions that, when executed by one or more processors, cause the learning device to: apply a neural network operation of a first student neural network to a training image to generate first prediction information corresponding to a specific task; apply a neural network operation of a second student neural network to a training image to generate second prediction information corresponding to the specific task; apply an error identification operation of an error identification network to a first integrated image based on the first prediction information and the training image to generate first error identification prediction information for predicting the accuracy of the first prediction information; apply an error identification operation of an error identification network to a second integrated image based on the second prediction information and the training image to generate second error identification prediction information for predicting the accuracy of the second prediction information; apply a neural network operation of a first teacher neural network associated with the first student neural network to the training image to generate first pseudo-label information corresponding to the specific task; apply a neural network operation of a second teacher neural network associated with the second student neural network to the training image to generate second pseudo-label information corresponding to the specific task; back-propagate a loss using a parameter updating device based on at least one of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo-label information, or the second pseudo-label information; and update parameters of the first student neural network and the second student neural network based on the loss using the parameter updating device.

[0016] The loss may include a first loss, a second loss, and a third loss. When executed by one or more processors, the instructions may further cause the learning device to: generate a first loss based on the first prediction information, the first error identification prediction information, and the first pseudo label information; generate a second loss based on the second prediction information, the second error identification prediction information, and the second pseudo label information; and generate a third loss based on the first prediction information and the second prediction information.

[0017] When executed by one or more processors, the instructions further cause the learning device to perform at least one of the following operations: using the error identification network and based on the determination that the first prediction information is accurate, generate first error identification information by determining a first specific value within a first specific range corresponding to the first identification value, or using the error identification network and based on the determination that the first prediction information is inaccurate, generate first error identification prediction information by determining a second specific value within a second specific range corresponding to a second identification value less than the first identification value. When executed by one or more processors, the instructions may additionally cause the learning device to perform at least one of the following items: using the error identification network and based on the determination that the second prediction information is accurate, generate second error identification prediction information by determining a third specific value within the first specific range, or using the error identification network and based on the determination that the second prediction information is inaccurate, generate second error identification prediction information by determining a fourth specific value within the second specific range.

[0018] When executed by one or more processors, the instructions further cause the learning device to: generate a first integrated image using a concatenated operation on the first prediction information and the training image; and generate a second integrated image using a concatenated operation on the second prediction information and the training image.

[0019] The specific task may include one of the following tasks: a segmentation task, a depth estimation task, or an image restoration task.

[0020] According to one or more example embodiments of the present disclosure, a testing device may include: one or more processors; and a memory. The memory stores instructions that, when executed by the one or more processors, may cause the learning device to: apply a first neural network operation of a first student neural network to a training image to generate first prediction information corresponding to a specific task, wherein the first student neural network includes a task-specific neural network; apply a neural network operation of a second student neural network to the training image to generate second prediction information corresponding to the specific task; based on the first prediction information and the training image, apply an error identification operation of an error identification network to a first integrated image to generate first error identification prediction information for predicting the accuracy of the first prediction information; based on the second prediction information and the training image, apply an error identification operation of an error identification network to a second integrated image to generate second error identification prediction information for predicting the accuracy of the second prediction information; and apply an error identification operation of an error identification network to a second integrated image to generate second error identification prediction information for predicting the accuracy of the second prediction information. The invention relates to a method for generating a task-specific neural network comprising: applying a neural network operation of a first teacher neural network associated with a second student neural network to a training image to generate first pseudo-label information corresponding to a specific task; applying a neural network operation of a second teacher neural network associated with a second student neural network to the training image to generate second pseudo-label information corresponding to the specific task; back-propagating a first loss based on at least one of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo-label information, or the second pseudo-label information using a parameter updating device; updating parameters of the first student neural network and the second student neural network based on the first loss using a parameter updating device; receiving a test image using a test image input device; and applying a specific neural network operation of a specific task neural network to the test image to generate prediction information for testing, which corresponds to a specific task.

[0021] The loss may include a first loss, a second loss, and a third loss. When the instruction is executed by one or more processors, it may cause the learning device to: generate a first loss based on the first prediction information, the first error identification prediction information, and the first pseudo label information; generate a second loss based on the second prediction information, the second error identification prediction information, and the second pseudo label information; and generate a third loss based on the first prediction information and the second prediction information.

[0022] When the instructions are executed by one or more processors, they may also cause the learning device to perform at least one of the following operations: using the error identification network and based on the accurate determination of the first prediction information, by determining a first specific value within a first specific range corresponding to the first identification value, to generate first error identification prediction information, or using the error identification network and based on the inaccurate determination of the first prediction information, by determining a second specific value within a second specific range corresponding to a second identification value less than the first identification value to generate first error identification prediction information. When the instructions are executed by one or more processors, they may additionally cause the learning device to perform at least one of the following: using the error identification network and based on the accurate determination of the second prediction information, by determining a third specific value within the first specific range to generate second error identification prediction information, or using the error identification network and based on the inaccurate determination of the second prediction information, by determining a fourth specific value within the second specific range to generate second error identification prediction information.

[0023] When executed by one or more processors, the instructions may also cause the learning device to: generate a first integrated image using a cascade operation on the first prediction information and the training image; and generate a second integrated image using a cascade operation on the second prediction information and the training image.

[0024] The specific task may include one of the following tasks: a segmentation task, a depth estimation task, or an image restoration task.

[0025] According to one or more example embodiments of the present disclosure, a method may include: applying a neural network operation of a first student neural network to a first training image to generate first prediction information corresponding to a specific task; applying a neural network operation of a second student neural network to the first training image to generate second prediction information corresponding to the specific task; applying an error identification operation of an error identification network to a first integrated image based on the first prediction information and the training image to generate first error identification prediction information for predicting the accuracy of the first prediction information; applying the error identification operation of the error identification network to a second integrated image based on the second prediction information and the training image to generate second error identification prediction information for predicting the accuracy of the second prediction information; applying a neural network operation of a first teacher neural network associated with the first student neural network to the first training image to generate first pseudo label information corresponding to the specific task; applying a neural network operation of a second teacher neural network associated with the second student neural network to the first training image to generate second pseudo label information corresponding to the specific task; back-propagating a loss based on at least one of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo label information, or the second pseudo label information using a parameter updating device; and updating parameters of the first student neural network and the second student neural network based on the loss using the parameter updating device.

[0026] Updating the parameters may include: updating the device using the parameters, back-propagating the loss by:

[0027] generating a first loss based on the first prediction information, the first error identification prediction information, and the first pseudo label information;

[0028] generating a second loss based on the second prediction information, the second false identification prediction information, and the second pseudo label information; and

[0029] A third loss is generated based on the first prediction information and the second prediction information.

[0030] The method may also include one of the following: using the error identification network and based on the accurate determination of the first prediction information, generating the first error identification information by determining a first specific value within a first specific range corresponding to the first identification value, or using the error identification network and based on the inaccurate determination of the first prediction information, generating the first error identification prediction information by determining a second specific value within a second specific range corresponding to a second identification value less than the first identification value. The method may further include one of the following: using the error identification network and based on the accurate determination of the second prediction information, generating the second error identification prediction information by determining a third specific value within the first specific range, or using the error identification network and based on the inaccurate determination of the second prediction information, generating the second error identification prediction information by determining a fourth specific value within the second specific range.

[0031] One of the first student neural network or the second student neural network is a specific student neural network. Before generating the first prediction information and the second prediction information, the method may further include: using the specific student neural network, applying the specific neural network operation to the second training image including the specific label information corresponding to the specific task to generate the specific prediction information corresponding to the specific task; using the parameter updating device, back-propagating the second loss based on the specific prediction information and the specific label information; and using the parameter updating device and updating the parameters of the specific student neural network based on the second loss.

[0032] After updating the parameters of the specific student neural network, the method may also include: using the error identification network and applying the error identification operation to the specific integrated image based on the specific prediction information and the second training image to generate specific error identification prediction information for predicting the accuracy of the specific prediction information; and using a parameter updating device to back-propagate a third loss based on the specific error identification prediction information and the specific error identification label information corresponding to the specific error identification prediction information; and using the parameter updating device to update the parameters of the error identification network.

[0033] According to one aspect of the present disclosure, a learning device may include: a first student neural network that applies a 1_1st neural network operation to a training image to generate first prediction information corresponding to a specific task; a second student neural network that applies a 2_1st neural network operation to the training image to generate second prediction information corresponding to the specific task; an error identification network that applies an error identification operation to a first integrated image based on the first prediction information and the training image to generate first error identification prediction information for predicting the accuracy of the first prediction information, and applies the error identification operation to the second integrated image based on the second prediction information and the training image to generate second error identification prediction information for predicting the accuracy of the second prediction information, a first teacher neural network that applies a 1_2nd neural network operation to the training image to generate first pseudo-label information corresponding to the specific task; a second teacher neural network that applies a 2_2nd neural network operation to the training image to generate second pseudo-label information corresponding to the specific task; and a parameter updating device that back-propagates a first loss based on at least some of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo-label information, or the second pseudo-label information, and updates parameters of the first student neural network and the second student neural network.

[0034] In one embodiment, the first loss may include a 1_1st loss, a 1_2nd loss, and a 1_3rd loss. The 1_1st loss may be generated based on the first prediction information, the first error identification prediction information, and the first pseudo label information, the 1_2nd loss may be generated based on the second prediction information, the second error identification prediction information, and the second pseudo label information, and the 1_3rd loss may be generated based on the first prediction information and the second prediction information.

[0035] In one embodiment, when the first prediction information is determined to be accurate, the error identification network may generate a first specific value within a first specific range corresponding to the first identification value as the first error identification prediction information; when the first prediction information is determined to be inaccurate, a second specific value within a second specific range corresponding to a second identification value less than the first identification value may be generated as the first error identification prediction information; when the second prediction information is determined to be accurate, a third specific value within the first specific range may be generated as the second error identification prediction information; and when the second prediction information is determined to be inaccurate, a fourth specific value within the second specific range may be generated as the second error identification prediction information.

[0036] In one embodiment, when the concatenation operation is applied to the first prediction information and the training image, a first integrated image may be generated, and when the concatenation operation is applied to the second prediction information and the training image, a second integrated image may be generated.

[0037] In one embodiment, the specific task may be any one of a segmentation task, a depth estimation task, or an image restoration task.

[0038] According to another aspect of the present disclosure, a test device may include a test image input device that receives a test image and a task-specific neural network that applies a specific neural network operation to the test image to generate prediction information for testing, the prediction information corresponding to a specific task.

[0039] According to another aspect of the present disclosure, a learning method may include: applying a 1_1st neural network operation to a first training image by a first student neural network to generate first prediction information corresponding to a specific task, applying a 2_1st neural network operation to the first training image by a second student neural network to generate second prediction information corresponding to the specific task, applying an error identification operation to a first integrated image based on the first prediction information and the first training image by an error identification network to generate first error identification prediction information for predicting the accuracy of the first prediction information, and applying an error identification operation to a second integrated image based on the second prediction information and the first training image by the error identification network to generate a prediction information for predicting the accuracy of the first prediction information. The method comprises the steps of: generating a first student neural network and a second student neural network, applying a 1_2nd neural network operation to the first training image to generate first pseudo-label information corresponding to the specific task, applying a 2_2nd neural network operation to the first training image to generate second pseudo-label information corresponding to the specific task, and back-propagating a first loss based on at least some of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo-label information or the second pseudo-label information by a parameter updating device, and updating parameters of the first student neural network and the second student neural network by the parameter updating device.

[0040] In one embodiment, the updating of parameters may include back-propagating, by a parameter updating device, a 1_1st loss generated based on the first prediction information, the first error identification prediction information, and the first pseudo label information, and a 1_2nd loss generated based on the second prediction information, the second error identification prediction information, and the second pseudo label information, and a 1_3rd loss generated based on the first prediction information and the second prediction information as the first loss, and updating, by the parameter updating device, the parameters of the first student neural network and the second student neural network.

[0041] In one embodiment, the generation of first error identification prediction information and second error identification prediction information may include, when the first prediction information is determined to be accurate, the error identification network generates a first specific value within a first specific range corresponding to the first identification value as the first error identification prediction information; when the first prediction information is determined to be inaccurate, the error identification network generates a second specific value within a second specific range corresponding to a second identification value less than the first identification value as the first error identification prediction information; when the second prediction information is determined to be accurate, the error identification network generates a third specific value within the first specific range as the second error identification prediction information; and when the second prediction information is determined to be inaccurate, the error identification network generates a fourth specific value within the second specific range as the second error identification prediction information.

[0042] In one embodiment, before generating the first prediction information and the second prediction information, the learning method may also include, by a specific student neural network between the first student neural network and the second student neural network, applying a specific neural network operation to a second training image including specific label information corresponding to a specific task to generate specific prediction information corresponding to the specific task, and by a parameter updating device back-propagating a second loss based on the specific prediction information and the specific label information, and by the parameter updating device updating parameters of the specific student neural network.

[0043] In one embodiment, after updating the parameters of the specific student neural network, the learning method may also include applying an error identification operation to a specific integrated image by the error identification network based on the specific prediction information and the second training image to generate specific error identification prediction information for predicting the accuracy of the specific prediction information, and back-propagating a third loss by a parameter updating device based on the specific error identification prediction information and specific error identification label information corresponding to the specific error identification prediction information, and updating the parameters of the error identification network by the parameter updating device. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other objects, features and advantages of the present disclosure will become more apparent through the following detailed description in conjunction with the accompanying drawings:

[0045] Figure 1 is a block diagram showing the configuration of a learning device;

[0046] Figure 2 , Figure 3 , Figure 4A , Figure 4B , Figure 5 and Figure 6 is a diagram for illustrating the operation of a learning device;

[0047] Figure 7 A flowchart showing a learning method;

[0048] Figure 8 A block diagram showing the configuration of a test device;

[0049] Fig. 9 is a diagram for illustrating the operation of the test device; and

[0050] Fig. 10A and 10B A graph comparing the output of the test device to the output of the device.

[0051] Regarding the description of the drawings, the same or similar reference numerals may be used for the same or similar components. DETAILED DESCRIPTION

[0052] Hereinafter, one or more example embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. When adding reference numerals to the components of each figure, it should be noted that the same components are represented by the same reference numerals even if they are shown on other figures. In addition, when describing the example embodiment (one or more) of the present disclosure, detailed descriptions of well-known features or functions will be excluded so as not to unnecessarily obscure the main points of the present disclosure.

[0053] When describing components according to exemplary embodiment (one or more) of the present disclosure, the terms of first, second, "A", "B", (a), (a) and the like may be used. These terms are intended only to distinguish one component from another, and these terms do not limit the properties, order or sequence of the corresponding components. In addition, unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art to which the present disclosure belongs. These terms defined in commonly used dictionaries will be interpreted as having the same meanings as the contextual meanings in the relevant technical field, and shall not be interpreted as having ideal or overly formal meanings unless expressly defined as having such meanings in this application.

[0054] In the following, reference will be made to Figures 1 to 10B One or more example embodiments of the present disclosure are shown in detail.

[0055] Figure 1 A block diagram illustrating the configuration of a learning (eg, machine learning) device 1000 .

[0056] refer to Figure 1 , the learning device 1000 may include a first student neural network 1100, a second student neural network 1200, an error identification network 1300, a first teacher neural network 1400, a second teacher neural network 1500 and a parameter updating device 1600. This will refer to Figures 2 to 6For reference, in the description of each of the second student neural network 1200 and the second teacher neural network 1500, repeated description of the same / similar parts as the description of each of the first student neural network 1100 and the first teacher neural network 1400 will be omitted. Figures 3 to 5 Only the first student neural network 1100 and / or the first teacher neural network 1400 are shown.

[0057] Figure 1 One or more of the example components shown may be implemented by software, hardware, or a combination of both. Figure 1 One or more of the example components shown may be implemented using a computing system.

[0058] The computing system may include at least one processor, memory, a user interface input device, a user interface output device, storage, and / or a network interface connected via a bus.

[0059] The processor may be a central processing unit (CPU) or a semiconductor device that processes commands stored in a memory and / or storage element. The memory and storage element may include various types of volatile or non-volatile storage media. For example, the memory may include a read-only memory (ROM) and a random access memory (RAM).

[0060] Thus, the operations of the methods or algorithms described herein may be embodied directly in hardware or in a software module executed by a processor, or in a combination thereof. The software module may reside on a storage medium (i.e., memory and / or storage) such as RAM, flash memory, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, removable disk and / or compact disk ROM (CD-ROM).

[0061] An example storage medium may be coupled to a processor, and the processor may read information from the storage medium and write information to the storage medium. In another approach, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may be located in a user terminal. In another approach, the processor and the storage medium may reside in a user terminal as separate components.

[0062] Figures 2 to 6 is a diagram for illustrating the operation of the learning device. Figure 2 and Figure 3 , the first student neural network 1100 can apply the 1_1st neural network operation to the training image to generate first prediction information corresponding to the specific task.

[0063] At this point, the first student neural network 1100 may be in a state where it has been trained to have a certain level of performance using a small amount of training data sets (eg, second training images) including label information.

[0064] For example, the specific task may be a task of outputting a predicted value for each pixel, and may be any one of a segmentation task, a depth estimation task, or an image restoration task, for example. For reference, for ease of description, Figures 2 to 10B The segmentation task is shown as an example of the specific task, but this is merely an example to help understanding. The specific task is not limited to the segmentation task.

[0065] Furthermore, the training images may be unlabeled images.

[0066] For example, the first student neural network 1100 may be a Deeplab v3+ network, in which ResNet-50 is the backbone network. However, this is just an example, and any network with a structure capable of performing a specific task (e.g., a segmentation task) may be the first student neural network 1100.

[0067] Additionally, the 1_1st neural network operation applied to the training image by the first student neural network 1100 may include a convolution operation.

[0068] At the same time, the second student neural network 1200 may apply a 2_1st neural network operation to the training image to generate second prediction information corresponding to a specific task.

[0069] For example, the first student neural network 1100 and the second student neural network 1200 may be Siamese networks, and they may share each other's parameters.

[0070] Additionally, the 2_1st neural network operations applied to the training images by the second student neural network 1200 may include convolution operations.

[0071] Furthermore, the second student neural network 1200 may be in a state where it is trained to have a certain level of performance using a small amount of training data set including label information.

[0072] In addition, the first student neural network 1100 and the second student neural network 1200 may be lightweight neural networks that are simpler in structure than the first teacher neural network 1400 and the second teacher neural network 1500 to be described below, and have fewer parameters than the first teacher neural network 1400 and the second teacher neural network 1500.

[0073] At the same time, reference Figure 2 and Figure 4A, the error identification network (EIN) 1300 may apply an error identification operation to the first integrated image based on the first prediction information from the first student neural network 1100 and the training image to generate first error identification prediction information for predicting the accuracy of the first prediction information.

[0074] Additionally, the EIN 1300 may apply an error identification operation to the second integrated image based on the second prediction information from the second student neural network 1200 and the training image to generate second error identification prediction information for predicting the accuracy of the second prediction information.

[0075] For example, when a concatenation operation is applied to the first prediction information and the training image, a first integrated image may be generated, and when a concatenation operation is applied to the second prediction information and the training image, a second integrated image may be generated.

[0076] For example, EIN 1300 may be a network in which a pre-trained ResNet34 network is a backbone network in ImageNet, and in which a sigmoid layer is included in an output stage.

[0077] For example, when the first prediction information is determined to be accurate, EIN 1300 may generate a first specific value (e.g., 0.7) within a first specific range corresponding to a first identification value (e.g., 1) as the first error identification prediction information, and when the first prediction information is determined to be inaccurate, a second specific value (e.g., 0.2) within a second specific range corresponding to a second identification value (e.g., 0) that is smaller than the first identification value may be generated as the first error identification prediction information.

[0078] Similarly, when the second prediction information is determined to be accurate, EIN 1300 may generate a third specific value (e.g., 0.8) within the first specific range as the second error identification prediction information, and when the second prediction information is determined to be inaccurate, a fourth specific value (e.g., 0.1) within the second specific range may be generated as the second error identification prediction information.

[0079] For reference, EIN 1300 may be a network that has been trained using a training data set that includes error identification label information.

[0080] refer to Figure 4B , the learning device 1000 can identify (i) the error identification prediction information 401 output from the EIN 1300 and (ii) the error identification label information 402 used to train the EIN 1300.

[0081] For example, the EIN 1300 may determine the accuracy (or confidence) of the classification information of each pixel as the prediction information output by the neural network (e.g., the student neural network). When the classification information of the neural network of a particular pixel is determined to be the correct answer, the error identification prediction information (or the error identification prediction information of each pixel) may have a first identification value (e.g., 1) or a value close to the first identification value (e.g., 0.8). When the classification information of the neural network of a particular pixel is determined to be an incorrect answer, the error identification prediction information (or the error identification prediction information of each pixel) may have a second identification value (e.g., 0) or a value close to the second identification value (e.g., 0.1). Therefore, the error identification prediction information including the probability value of each of all pixels may be output.

[0082] In addition, the error identification label information may be used to train the EIN 1300. The error identification label information may include information about whether the classification information of each pixel as the prediction information output by the neural network (e.g., the student neural network) is actually accurate, and may be, for example, Figure 4B Binary mask diagram 402 is shown. For example, when the classification information of the neural network of pixel (10, 20) is incorrect, the wrong identification label information of the pixel may have a value of "0". When the classification information of the neural network (e.g., the student neural network) of pixel (50, 40) is correct, the wrong identification label information of the pixel may have a value of "1".

[0083] At the same time, if Figure 5 As shown, the first teacher neural network 1400 can apply the 1_2nd neural network operation to the training image to generate the first pseudo label information corresponding to the specific task. For reference, although for convenience, in Figure 5 It can be easily understood that the first integrated image based on the first prediction information and the training image is input to the EIN 1300, and the first error identification prediction information is output.

[0084] For reference, since label information is not included in the training image input to the first teacher neural network 1400, the learning device 1000 may regard the prediction information output from the first teacher neural network 1400 (or the prediction information of a specific task corresponding to the training image) as label information (i.e., first pseudo-label information), and may use the prediction information to train the first student neural network 1100, which is pre-trained to have a high level of performance based on a large amount of training data.

[0085] However, when the learning device 1000 relies only on the first teacher neural network 1400 to train the first student neural network 1100, because there is a problem of vulnerability of the first student neural network 1100 trained to the first teacher neural network 1400, it can use the above-mentioned EIN 1300 to prevent this problem.

[0086] In other words, the EIN 1300 may reflect only the information determined to be accurately predicted (or the predicted value corresponding to a specific pixel area) in the first prediction information of the first student neural network 1100 (e.g., classification information of each pixel as segmented prediction information), which receives the knowledge of the first teacher neural network 1400 in the training of the first student neural network 1100. As a result, the learning device 1000 may improve the quality of the first pseudo label information from the first teacher neural network 1400. Only the advantage of each teacher neural network can be transferred to each student neural network through the above process. The detailed learning (e.g., machine learning) process will be described below.

[0087] At the same time, the second teacher neural network 1500 may apply a 2_2nd neural network operation to the training image to generate second pseudo label information corresponding to a specific task.

[0088] For example, the first teacher neural network 1400 may be a segmenter network that can focus on global context information compared to convolution-based segmentation schemes that focus on local windows. In addition, the second teacher neural network 1500 may be an encoder-decoder network that applies depth-wise separable convolutions to an atrous spatial pyramid pooling (ASPP) module and a decoder, which may be a deeplabv3+ network with less spatial information loss because a larger receptive field can be obtained even without performing aggregation. However, this is just an example, and the first teacher neural network 1400 and the second teacher neural network 1500 may be networks of different structures that can perform specific tasks (e.g., segmentation tasks).

[0089] For reference, since the illustration of the second teacher neural network 1500 is the same / similar to that of the first teacher neural network 1400, its repeated illustration will be omitted.

[0090] At the same time, the parameter updating device 1600 may back-propagate the first loss based on at least some of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo label information, or the second pseudo label information, and may update the parameters of the first student neural network 1100 and the second student neural network 1200.

[0091] At this time, the first loss may include a 1_1st loss, a 1_2nd loss, and a 1_3rd loss.

[0092] For example, the parameter updating device 1600 may back-propagate the 1_1st loss generated based on the first prediction information, the first error identification prediction information, and the first pseudo label information, and may update the parameters of the first student neural network 1100.

[0093] In addition, the parameter updating device 1600 may back-propagate the 1_2nd loss generated based on the second prediction information, the second error identification prediction information, and the second pseudo label information, and may update the parameters of the second student neural network 1200.

[0094] In addition, if Figure 6 As shown, the parameter updating device 1600 can back-propagate the 1_3rd loss L generated based on the first prediction information of the first student neural network 1100 and the second prediction information of the second student neural network 1200. MD , and the parameters (one or more) of the first student neural network 1100 and / or the second student neural network 1200 may be updated. In other words, consistent information between the various neural networks may be shared through the 1_3rd loss.

[0095] The configuration of the learning device 1000 has been described. Next, a description will be given of the operation of the learning device 1000.

[0096] Figure 7 is a flowchart for illustrating a learning method. Figure 7 In operation 701, at least one specific student neural network between the first student neural network 1100 and the second student neural network 1200 may apply a specific neural network operation to the second training image to generate specific prediction information corresponding to a specific task (e.g., a segmentation task, a depth estimation task, an image restoration task, and the like).

[0097] For reference, the second training image may include specific label information corresponding to a specific task. For example, the second training image may include label information corresponding to a segmentation task.

[0098] In operation 702, the parameter updating device 1600 may back-propagate the second loss based on the specific prediction information and the specific label information, and may update the parameters of the specific student neural network. In other words, the parameter updating device 1600 may perform supervised learning on the student neural network using a small amount of labeled data.

[0099] Put this into the formula as shown below.

[0100] [Equation 1]

[0101]

[0102] [Equation 2]

[0103]

[0104] For reference, D L may represent a training data set including a second training image, L ce , which may represent the cross entropy of each standard pixel between the specific prediction information P and the specific label information Y corresponding thereto, i may represent the index of each pixel indicating the input, and Y i The one-hot vector representing the ground truth (GT) of pixel i, T represents the transpose, L sup The second loss can be expressed as a cross entropy loss.

[0105] In operation 703 , the EIN 1300 may apply an error identification operation to the specific integrated image based on the specific prediction information and the second training image to generate specific error identification prediction information for predicting the accuracy of the specific prediction information.

[0106] In operation 704 , the parameter updating apparatus 1600 may back-propagate the third loss based on the specific error identification prediction information and the specific label information corresponding to the specific error identification prediction information, and may update the parameters of the EIN 1300 .

[0107] Thus, the EIN 1300 may be trained to determine the accuracy of specific prediction information on a pixel-by-pixel basis.

[0108] Put this into the formula as shown below.

[0109] [Equation 3]

[0110]

[0111]

[0112] For reference, B may be specific error identification prediction information of EIN 1300, which may be the result of applying the error identification operation to the result of concatenating the second training image X with the specific prediction information P. In addition, L ce It can represent the cross entropy of each standard pixel between the specific error identification prediction information B and the specific error identification label information M corresponding to the specific error identification prediction information B, and L EI The third loss can be expressed as a cross entropy loss.

[0113] In operation 705_1, the first student neural network 1100 may apply a 1_1st neural network operation to the first training image to generate first prediction information corresponding to the specific task.

[0114] In operation 705_2, the second student neural network 1200 may apply a 2_1st neural network operation to the first training image to generate second prediction information corresponding to the specific task.

[0115] In operation 706, the EIN 1300 may apply an error identification operation to the first integrated image based on the first prediction information and the first training image to generate first error identification prediction information for predicting the accuracy of the first prediction information, and may apply the error identification operation to the second integrated image based on the second prediction information and the first training image to generate second error identification prediction information for predicting the accuracy of the second prediction information.

[0116] In operation 707_1, the first teacher neural network 1400 may apply a 1_2nd neural network operation to the first training image to generate first pseudo label information corresponding to a specific task.

[0117] In operation 707_2, the second teacher neural network 1500 may apply a 2_2nd neural network operation to the first training image to generate second pseudo label information corresponding to the specific task.

[0118] For ease of description, operations 707_1 and 707_2 are described after operations 705_1 and 705_2, but the operation order of the learning device 1000 is not limited to this example. For example, operations 707_1 and 707_2 may be performed simultaneously with operations 705_1 and 705_2, but may be performed earlier than operations 705_1 and 705_2.

[0119] In operation 708, the parameter updating device 1600 may back-propagate the first loss based on at least some of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo label information, or the second pseudo label information, and may update the parameters of the first student neural network 1100 and the second student neural network 1200.

[0120] For example, the parameter updating device 1600 may back-propagate i) a 1_1st loss generated based on the first prediction information, the first error identification prediction information, and the first pseudo label information, (ii) a 1_2nd loss generated based on the second prediction information, the second error identification prediction information, and the second pseudo label information, and (iii) a 1_3rd loss generated based on the first prediction information and the second prediction information as the first loss, and may update the parameters of the first student neural network 1100 and the second student neural network 1200.

[0121] Put this into the formula as shown below.

[0122] [Equation 5]

[0123]

[0124] [Equation 6]

[0125]

[0126] [Equation 7]

[0127]

[0128] For reference, D U may represent a training dataset (or unlabeled dataset) including the first training image, L pseudo The 1_1 loss can be expressed as similar to L sup The loss of each pixel, B can represent the first error identification prediction information of EIN, The unique code vector representing the first pseudo label information, i represents the index of each pixel, can represent a function that rounds to the nearest integer, and L MD It can be represented based on the first prediction information s 1 and the second prediction information s 2 Generated 1_3rd loss. For example, L MD The first prediction information s can be represented 1 and the second prediction information s 2 For reference, KLD (or relative entropy, information gain, or information divergence) may be a function for calculating the difference between two probability distributions, which may calculate the information entropy difference that can be generated when sampling is performed using another distribution that approximates a certain ideal distribution.

[0129] When the number of teacher neural networks is multiple, they can be arranged according to Equation 6.

[0130] As shown in Equation 5 and Equation 6, the above-mentioned learning device 1000 can use the error identification prediction information of EIN 1300 and only use the error-free areas in the student neural network prediction information for learning.

[0131] Meanwhile, the configuration of the test device 2000 will be described below. The test device 2000 includes a specific student neural network. The parameters of the specific student neural network are as described above. Figure 8 The learning device 1000 is shown being updated.

[0132] Figure 8is a block diagram illustrating the configuration of the test device 2000 . Fig. 9 is a diagram for illustrating the operation of the test device. Fig. 10A and Fig. 10B A graph comparing the output of the test equipment with the output of the equipment.

[0133] refer to Figure 8 , the test device 2000 may include a test image input device 2100 and a task-specific neural network 2200 (i.e., a specific student neural network whose parameters are updated using a learning device).

[0134] First, the test image input apparatus 2100 may receive a test image that does not include tag information.

[0135] At the same time, if Fig. 9 As shown, the task-specific neural network 2200 may apply specific neural network operations to a test image to generate test prediction information corresponding to a specific task.

[0136] For reference, the task-specific neural network 2200 may be any one of the first student neural network or the second student neural network trained through the above training process.

[0137] For ease of description, two student neural networks and two teacher neural networks are described above as examples, but the present disclosure is not limited thereto. For example, the same / similar description may be applied to three or more student neural networks and three or more teacher neural networks corresponding to the three or more student neural networks.

[0138] The outputs of (i) the existing single teacher networks (teachers A and B) and (ii) the outputs of the student network based on knowledge extraction using multiple teacher neural networks will be referred to below. Fig. 10A and Fig. 10B Make a comparison.

[0139] First, refer to Fig. 10A , it can be determined that for the same test image 101, the segmentation accuracy of the output of teacher A (e.g., segmenter) and the output of teacher B (e.g., deeplabv3+) is lower. This can be identified in detail by the error identification information 105 of teacher A and the error identification information 106 of teacher B. However, it is shown that the ambulance and car areas are identified with lower relative errors because teacher A has the advantage of identifying global context information. On the other hand, it can be determined that relatively more errors are found in teacher B, but the road behind the ambulance is relatively better identified in teacher B than in teacher A.

[0140] Using the relatively advantageous pieces of information from each teacher neural network can significantly improve the performance of the student neural network. Therefore, it can be identified by the output 104 of the student neural network that the training of the student neural network has been completed.

[0141] For reference Fig. 10B To identify.

[0142] refer to Fig. 10B , it can be determined that the output 107 of the teacher (A) neural network and the output 108 of the teacher (B) neural network are inaccurate for the same test image, while the output 109 of the specific task neural network included in the test device is very accurate.

[0143] The present technology can provide a learning device that uses a plurality of teacher neural networks to train a student neural network based on knowledge extraction, a learning method thereof, a testing device using the learning device, and a testing method using the learning device.

[0144] The present technology can provide a learning device for transferring the advantages of each of a plurality of teacher neural networks to a student neural network, a learning method thereof, a testing device using the learning device, and a testing method using the learning device.

[0145] The present technology can provide a learning device for reducing the cost consumed in constructing a training data set, a learning method thereof, a testing device using the learning device, and a testing method using the learning device.

[0146] The present technology can provide a learning device for improving the performance of a student neural network based on knowledge extraction between multiple student neural networks, a learning method thereof, a testing device using the learning device, and a testing method using the learning device.

[0147] Furthermore, various effects directly or indirectly determined by the present disclosure can be provided.

[0148] Although the present disclosure has been described above with reference to exemplary embodiments and the accompanying drawings, the present disclosure is not limited thereto, and those skilled in the art to which the present disclosure belongs may make various modifications and changes to the present disclosure without departing from the spirit and scope of the present disclosure as claimed in the following claims.

[0149] Therefore, the exemplary embodiments of the present disclosure are provided to explain the spirit and scope of the present disclosure, rather than to limit them, and therefore the spirit and scope of the present disclosure are not limited by the embodiments. The scope of the present disclosure should be interpreted based on the appended claims, and all technical ideas within the scope equivalent to the claims should be included within the scope of the present disclosure.

Claims

1. A learning device, comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the learning device to: applying a neural network operation of a first student neural network to the training image to generate first prediction information corresponding to a specific task; applying a neural network operation of a second student neural network to the training image to generate second prediction information corresponding to the specific task; Based on the first prediction information and the training image, applying an error identification operation of an error identification network to a first integrated image to generate first error identification prediction information for predicting the accuracy of the first prediction information; Based on the second prediction information and the training image, applying the error identification operation of the error identification network to a second integrated image to generate second error identification prediction information for predicting the accuracy of the second prediction information; applying a neural network operation of a first teacher neural network associated with the first student neural network to the training image to generate first pseudo label information corresponding to the specific task; applying a neural network operation of a second teacher neural network associated with the second student neural network to the training image to generate second pseudo label information corresponding to the specific task; Back-propagating a loss based on at least one of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo-label information, or the second pseudo-label information using a parameter updating device; as well as Using the parameter update device and based on the loss, parameters of the first student neural network and the second student neural network are updated.

2. The learning device according to claim 1, wherein the loss comprises a first loss, a second loss and a third loss, and The instructions, when executed by the one or more processors, further cause the learning device to: generating the first loss based on the first prediction information, the first error identification prediction information, and the first pseudo label information; generating the second loss based on the second prediction information, the second error identification prediction information, and the second pseudo label information; as well as The third loss is generated based on the first prediction information and the second prediction information.

3. The learning device according to claim 1, When the instructions are executed by the one or more processors, the learning device further causes the learning device to perform at least one of the following: generating the first error identification information by determining a first specific value within a first specific range corresponding to a first identification value using the error identification network and based on the accurate determination of the first prediction information, or generating the first error identification prediction information by determining a second specific value within a second specific range corresponding to a second identification value less than the first identification value using the error identification network and based on a determination that the first prediction information is inaccurate, and The instructions, when executed by the one or more processors, further cause the learning device to perform at least one of the following: generating the second error identification prediction information by determining a third specific value within the first specific range using the error identification network and based on the accurate determination of the second prediction information, or The second error identification prediction information is generated by determining a fourth specific value within the second specific range using the error identification network and based on a determination that the second prediction information is inaccurate.

4. The learning device of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the learning device to: generating the first integrated image by concatenating the first prediction information and the training image; and The second integrated image is generated by concatenating the second prediction information and the training image. 5 . The learning device according to claim 1 , wherein the specific task comprises one of the following tasks: a segmentation task, a depth estimation task, or an image restoration task.

6. A testing device, comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the learning device to: applying a first neural network operation of a first student neural network to the training image to generate first prediction information corresponding to the specific task, wherein the first student neural network comprises a task-specific neural network; applying a neural network operation of a second student neural network to the training image to generate second prediction information corresponding to the specific task; Based on the first prediction information and the training image, applying an error identification operation of an error identification network to a first integrated image to generate first error identification prediction information for predicting the accuracy of the first prediction information; Based on the second prediction information and the training image, applying the error identification operation of the error identification network to a second integrated image to generate second error identification prediction information for predicting the accuracy of the second prediction information; applying a neural network operation of a first teacher neural network associated with the first student neural network to the training image to generate first pseudo label information corresponding to the specific task; applying a neural network operation of a second teacher neural network associated with the second student neural network to the training image to generate second pseudo label information corresponding to the specific task; Back-propagating a first loss based on at least one of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo-label information, or the second pseudo-label information using a parameter updating device; Using the parameter updating device and based on the first loss, updating parameters of the first student neural network and the second student neural network; receiving a test image using a test image input device; as well as Applying a specific neural network operation of the specific task neural network to the test image to generate prediction information for testing, the prediction information corresponding to the specific task.

7. The test device according to claim 6, wherein the loss comprises a first loss, a second loss and a third loss, and The instructions, when executed by the one or more processors, further cause the learning device to: generating the first loss based on the first prediction information, the first error identification prediction information, and the first pseudo label information; generating the second loss based on the second prediction information, the second error identification prediction information, and the second pseudo label information; as well as The third loss is generated based on the first prediction information and the second prediction information.

8. The test device according to claim 6, When the instructions are executed by the one or more processors, the learning device further causes the learning device to perform at least one of the following: generating the first error identification information by determining a first specific value within a first specific range corresponding to a first identification value using the error identification network and based on the accurate determination of the first prediction information, or generating the first error identification prediction information by determining a second specific value within a second specific range corresponding to a second identification value less than the first identification value using the error identification network and based on a determination that the first prediction information is inaccurate, and When the instructions are executed by the one or more processors, the learning device further causes the learning device to perform at least one of the following: generating the second error identification prediction information by determining a third specific value within the first specific range using the error identification network and based on the accurate determination of the second prediction information, or The second error identification prediction information is generated by determining a fourth specific value within the second specific range using the error identification network and based on a determination that the second prediction information is inaccurate.

9. The testing device of claim 6, wherein the instructions, when executed by one or more processors, further cause the learning device to: generating the first integrated image by concatenating the first prediction information and the training image; and The second integrated image is generated by concatenating the second prediction information and the training image. 10 . The test device according to claim 6 , wherein the specific task comprises one of the following tasks: a segmentation task, a depth estimation task, or an image restoration task.

11. A method comprising: applying a neural network operation of a first student neural network to a first training image to generate first prediction information corresponding to a specific task; applying a neural network operation of a second student neural network to the first training image to generate second prediction information corresponding to the specific task; Based on the first prediction information and the training image, applying an error identification operation of an error identification network to a first integrated image to generate first error identification prediction information for predicting the accuracy of the first prediction information; Based on the second prediction information and the training image, applying the error identification operation of the error identification network to a second integrated image to generate second error identification prediction information for predicting the accuracy of the second prediction information; applying a neural network operation of a first teacher neural network associated with the first student neural network to the first training image to generate first pseudo label information corresponding to the specific task; applying a neural network operation of a second teacher neural network associated with the second student neural network to the first training image to generate second pseudo-label information corresponding to the specific task; Back-propagating a loss based on at least one of the first prediction information, the second prediction information, the first error identification prediction information, the second error identification prediction information, the first pseudo-label information, or the second pseudo-label information using a parameter updating device; as well as Using the parameter update device and based on the loss, parameters of the first student neural network and the second student neural network are updated.

12. The method of claim 11, wherein updating the parameter comprises: Using the parameters to update the device, backpropagate the loss as follows: generating a first loss based on the first prediction information, the first error identification prediction information, and the first pseudo label information; generating a second loss based on the second prediction information, the second error identification prediction information, and the second pseudo label information; as well as A third loss is generated based on the first prediction information and the second prediction information.

13. The method of claim 11, further comprising one of the following: generating the first error identification information by determining a first specific value within a first specific range corresponding to a first identification value using the error identification network and based on the accurate determination of the first prediction information, or generating the first error identification prediction information by determining a second specific value within a second specific range corresponding to a second identification value less than the first identification value using the error identification network and based on a determination that the first prediction information is inaccurate, The method further comprises one of the following: generating the second error identification prediction information by determining a third specific value within the first specific range using the error identification network and based on the accurate determination of the second prediction information, or The second error identification prediction information is generated by determining a fourth specific value within the second specific range using the error identification network and based on a determination that the second prediction information is inaccurate.

14. The method of claim 11, wherein one of the first student neural network or the second student neural network is a specific student neural network, and Before generating the first prediction information and the second prediction information, the method further includes: Using the specific student neural network, applying a specific neural network operation to a second training image including specific label information corresponding to the specific task to generate specific prediction information corresponding to the specific task; Back-propagating a second loss based on the specific prediction information and the specific label information using the parameter updating device; as well as The parameter updating device is used to update the parameters of the specific student neural network based on the second loss.

15. The method according to claim 14, after updating the parameters of the specific student neural network, further comprising: applying the error identification operation to a specific integrated image using the error identification network and based on the specific prediction information and the second training image to generate specific error identification prediction information for predicting the accuracy of the specific prediction information; as well as Using the parameter updating device, back-propagating a third loss based on the specific error identification prediction information and the specific error identification label information corresponding to the specific error identification prediction information; as well as The parameters of the error identification network are updated using the parameter updating device.

Images