Learning device, learning method, and program

Abstract

To generate learning data suitable to improvement of the generalization performance of a neural network used for estimation.SOLUTION: A learning device comprises: a conversion unit which creates a second conversion image obtained by converting a first image into a second domain image in a pseudo manner by using a first neural network, converts the second conversion image into a first reconstructed image reconstructed to the first domain image by using a second neural network different from the first neural network, creates a first conversion image obtained by converting a second image into the first domain image in a pseudo manner by using the second neural network, and converts the first conversion image into a second reconstructed image reconstructed to the second domain image by using the first neural network; and an update unit which calculates a loss due to image conversion to update to parameters of the first neural network and the second neural network in which the loss becomes minimum.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a learning method, and a program. [Background technology]

[0002] There are currently known techniques for generating training data used in machine learning, such as neural networks that perform estimations of class classification, object detection, and position regression. For example, techniques that generate data similar to training data using deep learning, such as variational autoencoders (VAEs) and generalized adversarial networks (GANs), are used to augment or replace training data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-86497 [Non-patent literature]

[0004] [Non-Patent Document 1] Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks” ICCV 2017 [Non-patent document 2] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks.In ICLR,2016 [Non-patent document 3] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, University of California, Berkeley, “Image-to-Image Translation with Conditional Adversarial Nets,” CVPR2017 Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional techniques aim at data conversion itself, and it has been difficult to generate training data suitable for improving the discrimination performance of the neural network used for estimation. [Means for solving the problem]

[0006] A learning device according to one embodiment of the present disclosure is a learning device for a first neural network that transforms a first image in a first domain into a second image in a second domain. The learning device includes a transformation unit and an update unit. The transformation unit uses a first neural network to create a second transformed image by pseudo-transforming the first image into an image in the second domain, uses a second neural network different from the first neural network to transform the second transformed image into a first reconstructed image by reconstructing the image in the first domain, uses the second neural network to create a first transformed image by pseudo-transforming the second image into an image in the first domain, and transforms the first transformed image into a second reconstructed image by reconstructing the image in the second domain by using the first neural network. The update unit calculates a first difference between the first image and the first reconstructed image, calculates a second difference between the second image and the second reconstructed image, calculates a third difference between the first image and the second transformed image from structural information based on a first correct answer label of the first domain, calculates a fourth difference between the second image and the first transformed image from structural information based on a second correct answer label of the second domain, calculates a loss due to image transformation based on the first difference, the second difference, the third difference, and the fourth difference, and updates the parameters of the first neural network and the second neural network to minimize the loss.

[0007] A training method according to one embodiment of the present disclosure is a training method for a first neural network that transforms a first image in a first domain into a second image in a second domain using a training device. The training method is performed by a processing unit of the training device. The processing unit of the training device uses the first neural network to create a second transformed image by pseudo-transforming the first image into an image in the second domain, uses a second neural network different from the first neural network to transform the second transformed image into a first reconstructed image by reconstructing the image in the first domain, uses the second neural network to create a first transformed image by pseudo-transforming the second image into an image in the first domain, and uses the first neural network to transform the first transformed image into a second reconstructed image by reconstructing the image in the second domain. The processing unit of the learning device calculates a first difference between the first image and the first reconstructed image, calculates a second difference between the second image and the second reconstructed image, calculates a third difference between the first image and the second transformed image from structural information based on a first correct answer label of the first domain, calculates a fourth difference between the second image and the first transformed image from structural information based on a second correct answer label of the second domain, calculates a loss due to image transformation based on the first difference, the second difference, the third difference, and the fourth difference, and updates the parameters of the first neural network and the second neural network to minimize the loss. [Effects of the Invention]

[0008] The learning device, learning method, and program disclosed herein can augment training data that is more suitable for improving the discrimination performance of the neural network used for estimation than conventional techniques, thereby enabling high-performance estimation of class discrimination, object detection, position regression, and the like using machine learning such as neural networks. [Brief explanation of the drawings]

[0009] [Figure 1] Schematic diagram of image conversion according to an embodiment [Figure 2] FIG. 1 is a diagram illustrating an example of a functional configuration of a learning device according to an embodiment. [Figure 3] FIG. 1 is a diagram showing an example of the configuration of a neural network according to an embodiment; [Figure 4] 1 is a flowchart illustrating an example of a learning method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of a learning device, a learning method, and a program will be described in detail with reference to the accompanying drawings. However, more detailed explanations than necessary may be omitted. For example, detailed explanations of well-known matters or redundant explanations of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following explanations and to facilitate understanding by those skilled in the art.

[0011] The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment) A learning device according to an embodiment of the present invention is a device that trains a first neural network that accepts input of a first image as data of a first domain and converts it to resemble a second image, which is data of a second domain (to resemble data of the second domain), and a second neural network that accepts input of a second image as data of the second domain and converts it to resemble a first image, which is data of the first domain (to resemble data of the first domain).

[0013] FIG. 1 is a schematic diagram of image conversion according to an embodiment. For example, as shown in FIG. 1, the input to a first neural network is a visible light image of the front view captured by an on-board camera, and the input to a second neural network is an infrared image of the front view captured by the on-board camera. The learning device then accepts the input of the visible light image as data of the first domain and converts it into a pseudo-infrared image similar to the infrared image, which is data of the second domain, and accepts the input of the pseudo-infrared image as data of the second domain and converts it into a reconstructed visible light image similar to the data of the first domain. A case where the first neural network and the second neural network are trained by performing such conversion will be described.

[0014] To learn such a transformation, a learning device according to an embodiment of the present invention stores visible light images (data of the first domain) and infrared images (data of the second domain) captured in advance by an on-board camera, correct label data for the first domain, and correct label data for the second domain. The correct labels describe the positions within the image of the circumscribing rectangle of another vehicle contained in the image and the vertices of the circumscribing rectangular parallelepiped of the vehicle that are in contact with the ground.

[0015] Furthermore, the learning device according to an embodiment of the present invention further trains the first neural network and the second neural network to improve the conversion quality achieved by training the first neural network with data from the first domain and the second neural network with data from the second domain. The first neural network converts the data from the first domain to resemble data from the second domain (to resemble data from the second domain), and the second neural network converts the data from the second domain to resemble data from the first domain (to resemble data from the first domain).

[0016] The data in the second domain is, for example, infrared images and correct label data.

[0017] In this way, the learning infrared images (data of the second domain) and their corresponding correct labels are stored in the learning device of the embodiment according to the present invention.

[0018] The data in the second domain is not limited to infrared images. The data in the second domain and its correct label may be any combination of correct data and data that can be used to augment or substitute for the data in the first domain.

[0019] Regarding the correct labels, there may be data that is included in the correct labels of the data in the first domain but not in the correct labels of the data in the second domain, and conversely, there may be data that is included in the correct labels of the data in the second domain but not in the correct labels of the data in the first domain.

[0020] Furthermore, if the second neural network can generate data corresponding to the data in the first domain from the correct answer label of the data in the first domain, there is no need to prepare a correct answer label for the data in the second domain (it may be the same as the correct answer label for the data in the first domain).

[0021] The first neural network may be any neural network capable of converting data in a first domain to resemble data in a second domain. The second neural network may be any neural network capable of converting data in a second domain to resemble data in a first domain. The first and second neural networks may employ the most appropriate conversion technology based on the data format of the second domain and the data format of the first domain. Examples of conversion technologies applied to the first and second neural networks include CycleGAN (Non-Patent Document 1), DCGAN (Non-Patent Document 2), and Pix2Pix (Non-Patent Document 3).

[0022] [Configuration example] 2 is a block diagram showing an example of the configuration of a learning device 1 according to an embodiment of the present invention. This learning device 1 is configured using, for example, a dedicated or general-purpose computer. As shown in FIG. 2, the learning device 1 according to an embodiment of the present invention includes a processing unit 10, a storage unit 20, a communication unit 30, and a bus 40 connecting these components.

[0023] The processing unit 10 includes an acquisition unit 11, a conversion unit 12, and an update unit 13. Specific explanations of the processing of each unit will be given later. Note that while Fig. 2 illustrates major functional blocks related to the embodiment of the present invention, the functions of the processing unit 10 are not limited to these.

[0024] The processing of each function executed by the learning device 1 is stored in the storage unit 20, for example, in the form of a program executable by a computer. The processing unit 10 is a processor that realizes the function corresponding to each program by reading and executing the program from the storage unit 20. When each program is read, the processing unit 10 has the functional blocks shown in FIG.

[0025] 2 illustrates each functional block being realized by a single processing unit 10, but multiple independent processors may be combined to form the processing unit 10. In this case, each processing function may be configured as a program, or a specific function may be implemented in a dedicated, independent program execution circuit.

[0026] The above-mentioned "processor" refers to, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit), or a circuit such as an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), or a Field Programmable Gate Array (FPGA)).

[0027] The processing unit 10 realizes its functions by reading and executing a program stored in the storage unit 20. Note that instead of storing a program in the storage unit 20, the program may be directly embedded in the circuit of the processing unit 10. In this case, the processing unit 10 realizes its functions by reading and executing the program embedded in the circuit.

[0028] The storage unit 20 stores data associated with each functional block of the processing unit 10 as necessary. The storage unit 20 stores neural network programs and weight parameters. The storage unit 20 according to the embodiment of the present invention stores programs and data used for various processes. For example, the storage unit 20 may be a random access memory (RAM), a semiconductor memory device such as a flash memory, a hard disk, or an optical disk. The storage unit 20 may also be replaced by a storage device external to the learning device 1. The storage unit 20 may also be a storage medium that stores or temporarily stores a program downloaded via a local area network (LAN) or the Internet. The storage medium is not limited to one, and may be composed of multiple media.

[0029] The memory unit 20 may store in advance data of the first domain used for learning and the corresponding correct labels for that data, and data of the second domain and the corresponding correct labels for that data, or the aforementioned data stored in advance in a device such as another server may be read out in portions via a LAN or the like and stored in the memory unit 20.

[0030] The communication unit 30 is an interface for inputting and outputting information to and from an external device connected by wire or wirelessly. The communication unit 30 may also be connected to a network for communication.

[0031] Next, the processing of each functional block of the processing unit 10 will be described.

[0032] The acquisition unit 11 reads out, as learning data, data of the first domain and a correct label of the data of the first domain from the storage unit 20. The acquisition unit 11 also reads out, as learning data, data of the second domain and a correct label of the data of the second domain from the storage unit 20.

[0033] As shown in Fig. 1, the conversion unit 12 uses a neural network to accept data of a first domain (visible light image), convert the data of the first domain into second converted data (pseudo-infrared image) similar to the data of a second domain, and accept data of the second domain and convert the data of the second domain into first converted data (reconstructed visible image) similar to the data of the first domain. Details of the configuration of the neural network used for conversion will be described later. The conversion unit 12 using a neural network means that it uses a neural network program and weight parameters stored in the storage unit 20.

[0034] The update unit 13 updates the parameters of the neural network based on the learning data (the data of the first domain and its correct label, or the data of the second domain and its correct label) read out by the acquisition unit 11. Details of the update method will be described later.

[0035] [Neural network configuration example] Fig. 3 is a diagram showing an example of the configuration of a neural network according to an embodiment of the present invention. In the example of Fig. 3, a visible light image is used as data for the first domain, and an infrared image is used as data for the second domain.

[0036] The data in the first and second domains may be R, G, and B color images, or may be color images with converted color spaces (e.g., Y, U, and V color images). The data in the first and second domains may also be single-channel images obtained by converting a color image to monochrome. The data in the first and second domains may not be the original image, but may be, for example, an R, G, and B color image, and may be an image obtained by subtracting the average pixel value of each channel. For example, the data in the first and second domains may be normalized images in which the pixel value of each pixel falls within the range of 0 to 1 or -1 to 1. Normalization is performed, for example, by subtracting the average value from the pixel value of each pixel, and then dividing each pixel value by the variance or the dynamic range of the pixel values ​​of the image.

[0037] As shown in FIG. 3, when a first neural network (hereinafter, "neural network" will be referred to as "NN" in the drawings) 101 receives input of data D1D of a first domain, it converts the data D1D of the first domain into second converted data D2CD that is similar to the data of a second domain and outputs the converted data.

[0038] When the second neural network 102 receives the input of the second domain data D2D, it converts the second domain data D2D into first converted data D1CD similar to the data in the first domain and outputs the converted data.

[0039] When the first neural network 101 receives the input of the first transformed data D1CD, it converts it into second reconstructed data D2RD that is similar to the data of the second domain and outputs it.

[0040] When the second neural network 102 receives the input of the second transformed data D2CD, it transforms it into first reconstructed data D1RD that is similar to the data of the first domain and outputs it.

[0041] When the third neural network 103 receives the input of the second domain data D2D or the second converted data D2CD, it identifies whether the image is true or false. Then, the third neural network 103 transmits the identification result of the second domain data D2D or the second converted data D2CD to the third update unit 133.

[0042] When the fourth neural network 104 receives the input of the first domain data D1D or the first converted data D1CD, it identifies whether the first domain data D1D or the first converted data D1CD is a true image or a false image. Then, the fourth neural network 104 transmits the true / false identification result to the fourth update unit 134.

[0043] The first neural network 101, the second neural network 102, the third neural network 103, and the fourth neural network 104 are used by the conversion unit 12 described above.

[0044] The parameters of first neural network 101, second neural network 102, third neural network 103, and fourth neural network 104 are updated by update unit 13. Update unit 13 includes first update unit 131, second update unit 132, third update unit 133, fourth update unit 134, fifth update unit 135, and sixth update unit 136.

[0045] The first update unit 131 receives the data D1D of the first domain and receives the first reconstructed data D1RD from the second neural network 102. Then, the first update unit 131 updates the parameters of the first neural network 101 and the parameters of the second neural network 102 so that the first reconstructed data D1RD approaches the data D1D of the first domain.

[0046] The second update unit 132 receives the data D2D of the second domain and receives the second reconstructed data D2RD from the first neural network 101. Then, the second update unit 132 updates the parameters of the second neural network 102 and the parameters of the first neural network 101 so that the second reconstructed data D2RD approaches the data D2D of the second domain.

[0047] The third update unit 133 receives the authenticity discrimination result from the third neural network 103. Then, the third update unit 133 updates the parameters of the third neural network 103 based on the authenticity discrimination result of the image so that the third neural network 103 can correctly discriminate between authentic and fake images.

[0048] The fourth update unit 134 receives the authenticity discrimination result from the fourth neural network 104. Then, the fourth update unit 134 updates the parameters of the fourth neural network 104 based on the authenticity discrimination result of the image so that the fourth neural network 104 can correctly discriminate between authentic and fake images.

[0049] The fifth update unit 135 receives the first domain data D1D, the second transformed data D2CD, and the correct label D1CL of the first domain data from the acquisition unit 11. Then, the fifth update unit 135 updates the parameters of the first neural network 101 based on the first domain data D1D, the second transformed data D2CD, and the correct label D1CL of the first domain data.

[0050] The sixth update unit 136 receives the second domain data D2D, the first transformed data D1CD, and the correct label D2CL of the second domain data from the acquisition unit 11. Then, the sixth update unit 136 updates the parameters of the second neural network 102 based on the second domain data D2D, the first transformed data D1CD, and the correct label D2CL of the second domain data.

[0051] Specifically, the first update unit 131 calculates the loss L from the difference between the first reconstructed data D1RD and the data D1D of the first domain. cycle1 Similarly, the second update unit 132 calculates the loss L from the difference between the second reconstructed data D2RD and the second domain data D2D. cycle2 The third update unit 133 calculates the loss L from the difference between the numerical value output as a probability value exceeding 0 and less than 1 and the "true value: 1" and the "false value: 0". gan1 Similarly, the fourth update unit 134 calculates the loss L from the difference between the numerical value output as a probability value exceeding 0 and less than 1 and the "true value: 1" and the "false value: 0". gan2 The fifth update unit 135 calculates the loss L from the difference between the data D1D of the first domain and the second converted data D2CD from the region structure information based on the correct label D1CL of the data of the first domain. edge1 Similarly, the sixth update unit 136 calculates the loss L from the difference between the second domain data D2D and the first converted data D1CD from the region structure information based on the correct label D2CL of the second domain data. edge2 Then, the update unit 13 calculates L cycle1 and L cycle2 and L edge1 and L edge2 The loss L is calculated by adding weighted sums using the following formula (1). g Ask for. L g = a × L cycle1 +b×L cycle2 +c×L edge1 +d×L edge2 ···(1) Here, a, b, c, and d are predetermined constants.

[0052] Then, the first update unit 131, the second update unit 132, the fifth update unit 135, and the sixth update unit 136 calculate the loss L g The parameters of the first neural network 101 and the second neural network 102 are updated so that is minimized.

[0053] It should be noted that the method of updating the parameters of the first neural network 101 and the second neural network 102 is not limited to the method described here; other methods may be used as long as they reduce the difference between the first and second domain data D1D, D2D and the first and second reconstructed data D1RD, D2RD in the outputs of the first neural network 101 and the second neural network 102.

[0054] Furthermore, any method for calculating the loss may be used as long as it can be applied retroactively to the neural network to update the parameters, and a loss calculation method may be selected depending on the task. For example, the loss may be calculated using L1Loss, L2Loss, SSIMLoss, PerceptualLoss, or a combination of these methods. Furthermore, the constants a, b, c, and d may be adaptively changed depending on the progress of learning.

[0055] Furthermore, the third update unit 133 and the fourth update unit 134 update the loss L gan1 , L gan2 The parameters of the third neural network 103 and the fourth neural network 104 are updated so that is minimized.

[0056] It should be noted that the method of updating the parameters of the third neural network 103 and the fourth neural network 104 is not limited to the method described here; any other method may be used as long as it reduces the difference between the "true value: 1" and the "false value: 0" in the outputs of the third neural network 103 and the fourth neural network 104.

[0057] Furthermore, any method for calculating the loss can be used as long as it can be applied retroactively to the neural network to update the parameters, and the loss calculation method can be selected according to the task. For example, CrossEntropyLoss, L1Loss, L2Loss, etc. can be selected as the loss calculation method.

[0058] In addition, L edge1 and L edge2 The calculation method of (D1D, D2D) may be any method that reduces the difference between the first and second domain data D1D, D2D and the first and second converted data D1CD, D2CD from region structure information based on the correct labels D1CL, D2CL of the first and second domains. For example, an arbitrary calculation method may be selected from the circumscribing rectangle of the object region included in the image based on the correct labels D1CL, D2CL of the first and second domains, and the pixel information within each region in the first and second domain data D1D, D2D and the first and second converted data D1CD, D2CD. For example, the loss calculation method may involve edge extraction, and the error in edge strength, position, or direction, or the result of a matching process such as Chamfer, may be selected as the loss. An arbitrary filter process such as Laplacian or Difference of Gaussian may be selected as the edge extraction method. Alternatively, a calculation method based on the difference in the frequency band may be selected by converting the pixel information within the region into the frequency domain.

[0059] [Example of learning method] 4 is a flowchart showing an example of a learning method according to an embodiment of the present invention. First, the acquisition unit 11 reads out learning data (data of the first domain, a correct answer label for the data of the first domain, data of the second domain, and a correct answer label for the data of the second domain) from the storage unit 20 (step S31).

[0060] For example, when the data of the first domain is a visible light image and the data of the second domain is an infrared image, the acquisition unit 11 may read one visible light image and its correct label, and one infrared image and its correct label. Alternatively, for example, the acquisition unit 11 may read a set of a visible light image and its correct label, and a set of an infrared image and its correct label, for example, two, four, or eight images each. Alternatively, for example, the acquisition unit 11 may change the number of images read of the first domain data D1D and the second domain data D2D.

[0061] These sets of inputs (units of data to be processed at one time) will be referred to as batches below. The number of parameter update processes for one batch of inputs will be referred to as the number of iterations.

[0062] Next, the conversion unit 12 executes a conversion process using the first neural network 101 (step S32). Specifically, the conversion unit 12 inputs the first domain data D1D in the read batch to the first neural network 101, and generates second converted data D2CD.

[0063] Next, the conversion unit 12 executes a conversion process using the second neural network 102 (step S33). Specifically, the conversion unit 12 inputs the data D2D of the second domain in the read batch to the second neural network 102, and generates first converted data D1CD.

[0064] Next, the conversion unit 12 executes a conversion process using the second neural network 102 (step S34). Specifically, the conversion unit 12 inputs the generated second converted data to the second neural network 102 to generate first reconstructed data D1RD.

[0065] Next, the conversion unit 12 executes a conversion process using the first neural network 101 (step S35). Specifically, the conversion unit 12 inputs the generated first converted data to the first neural network 101 to generate second reconstructed data D2RD.

[0066] Next, the first update unit 131, the second update unit 132, the fifth update unit 135, and the sixth update unit 136 calculate the loss defined by the above-mentioned equation (1) based on the processing results of steps S32 to S35 (step S36).

[0067] Next, the first update unit 131, the second update unit 132, the fifth update unit 135, and the sixth update unit 136 update the parameters of the first neural network 101 and the second neural network 102 based on the loss calculated by the processing of step S36 (step S37).

[0068] Next, the conversion unit 12 executes a discrimination process using the third neural network 103 (step S38). Specifically, the conversion unit 12 inputs the second domain data D2D and the generated second converted data D2CD to the third neural network 103, and discriminates whether the image is a true image or a false image.

[0069] Next, the conversion unit 12 executes a discrimination process using the fourth neural network 104 (step S39). Specifically, the conversion unit 12 inputs the first domain data D1D and the generated first converted data D1CD to the fourth neural network 104, and discriminates whether the image is a true image or a false image.

[0070] Next, the third update unit 133 and the fourth update unit 134 calculate the loss based on the processing results of steps S38 and S39 (step S40).

[0071] Next, third update unit 133 and fourth update unit 134 update the parameters of third neural network 103 and fourth neural network 104 based on the loss calculated by the process of step S40 (step S41).

[0072] Next, the update unit 13 determines whether the update process has been repeated a predetermined number of times (iteration count) (step S42). If the update process has not been repeated the predetermined number of times (step S42, No), the process returns to step S31. If the update process has been repeated the predetermined number of times (step S42, Yes), the process ends.

[0073] As described above, in the learning device 1 according to the embodiment of the present invention, the conversion unit 12 receives data of a first domain and converts the data of the first domain into second converted data using the first neural network 101 that converts the data of the first domain into second converted data similar to the data of the second domain, receives data of the second domain and converts the data of the second domain into first converted data using the second neural network 102 that converts the data of the second domain into first converted data similar to the data of the first domain, receives the second converted data and converts the second converted data into first reconstructed data using the second neural network 102, and converts the first converted data into second reconstructed data using the first neural network 101 that receives the first converted data and converts the first converted data into second reconstructed data. When the third neural network 103 receives input of data of the second domain or the second converted data, it identifies whether the image is true or false. The fourth neural network 104 receives the input of the first domain data or the first transformed data, and distinguishes between a true image and a false image.

[0074] The first update unit 131 updates the parameters of the first neural network 101 so that the first reconstructed data approaches the data of the first domain. The second update unit 132 updates the parameters of the second neural network 102 so that the second reconstructed data approaches the data of the second domain. The third update unit 133 updates the parameters of the third neural network 103 so that it can correctly distinguish between true and false images. The fourth update unit 134 updates the parameters of the fourth neural network 104 so that it can correctly distinguish between true and false images. The fifth update unit 135 updates the parameters of the first neural network 101 based on the data of the first domain, the second transformed data, and the correct label of the data of the first domain. The sixth update unit 136 updates the parameters of the second neural network 102 based on the data of the second domain, the first transformed data, and the correct label of the data of the second domain.

[0075] As a result, the learning device 1 according to the embodiment of the present invention can perform learning suitable for improving the generation quality of neural networks (first neural network 101 and second neural network 102) used for image conversion. Specifically, the learning device 1 according to the embodiment of the present invention can train, for example, the first neural network 101 that receives a visible light image as input and generates an infrared image, and the second neural network 102 that receives an infrared image as input and generates a visible light image. This can improve the generation quality of the generative network (first neural network 101) that generates an image of the first domain (such as a visible light image) and the generative network (second neural network 102) that generates an image of the second domain (such as an infrared image). This can improve the discrimination performance of the networks used for estimation.

[0076] As described above, the processing functions of the learning device 1 according to the embodiment of the present invention are realized by the learning device 1, which is configured using a computer, executing a program. In this case, the program executed by the learning device 1 according to the embodiment of the present invention may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. The program executed by the learning device 1 according to the embodiment of the present invention may also be provided or distributed via a network such as the Internet. Furthermore, the program executed by the learning device 1 according to the embodiment of the present invention may be provided by being pre-installed on a non-volatile recording medium such as a ROM.

[0077] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Industrial Applicability]

[0078] The learning device, learning method, and program disclosed herein are capable of augmenting learning data better than conventional techniques, and are therefore useful as data analysis processing using machine learning such as neural networks for class identification, object detection, and position regression. [Explanation of symbols]

[0079] 1 Learning device 10 Processing section 11 Acquisition Department 12 Conversion unit 13 Update section 20 Memory section 30 Communications Department 101 The First Neural Network 102 Second Neural Network 103 The Third Neural Network 104 The Fourth Neural Network 131 First Update 132 Second Update 133 Third Update 134 Fourth Update 135 Fifth Update 136 Sixth Update

Claims

1. 1. A first neural network trainer configured to transform a first image in a first domain into a second image in a second domain, the trainer comprising: a conversion unit that uses the first neural network to create a second converted image by pseudo-converting the first image into an image of the second domain, that uses a second neural network different from the first neural network to convert the second converted image into a first reconstructed image by reconstructing the image of the first domain, that uses the second neural network to create a first converted image by pseudo-converting the second image into the image of the first domain, and that converts the first converted image into a second reconstructed image by reconstructing the image of the second domain by using the first neural network; an updating unit that calculates a first difference between the first image and the first reconstructed image, calculates a second difference between the second image and the second reconstructed image, calculates a third difference between the first image and the second transformed image from structural information based on a correct label for the first image of the first domain, calculates a fourth difference between the second image and the first transformed image from structural information based on a correct label for the second image of the second domain, calculates a loss due to image transformation based on the first difference, the second difference, the third difference, and the fourth difference, and updates parameters of the first neural network and the second neural network so as to minimize the loss. Learning device.

2. The conversion unit uses a third neural network different from either the first or second neural network to identify whether the second image and the second converted image are true or false, and uses a fourth neural network different from either the first to third neural networks to identify whether the first image and the first converted image are true or false. The learning device according to claim 1 .

3. the updating unit updates parameters of the third neural network or the fourth neural network based on a result of authenticity discrimination of the image output by the third neural network or the fourth neural network. The learning device according to claim 2 .

4. the first image is a visible light image and the second image is an infrared image; The learning device according to any one of claims 1 to 3.

5. 1. A method of training a first neural network configured to transform a first image in a first domain into a second image in a second domain using a training device, the method comprising: a processing unit of the learning device that uses the first neural network to create a second converted image by pseudo-converting the first image into an image of the second domain, uses a second neural network different from the first neural network to convert the second converted image into a first reconstructed image by reconstructing the image of the first domain, uses the second neural network to create a first converted image by pseudo-converting the second image into an image of the first domain, and uses the first neural network to convert the first converted image into a second reconstructed image by reconstructing the image of the second domain; the processing unit of the learning device calculates a first difference between the first image and the first reconstructed image, calculates a second difference between the second image and the second reconstructed image, calculates a third difference between the first image and the second transformed image from structural information based on a correct label for the first image of the first domain, calculates a fourth difference between the second image and the first transformed image from structural information based on a correct label for the second image of the second domain, calculates a loss due to image transformation based on the first difference, the second difference, the third difference, and the fourth difference, and updates parameters of the first neural network and the second neural network so as to minimize the loss. How to learn.

6. A program for causing a processing unit of the learning device to execute the learning method according to claim 5.

7. The structural information is based on a ground truth label that identifies an object region in the first or second image, The learning device calculating the structural information from pixel information within the object region corresponding to the correct label in the first or second image; Based on the third and fourth differences, parameters of the first and second neural networks are updated so as to minimize loss due to image transformation for the object region. The learning device according to claim 1 .

Images