Learning device, image generation device and corresponding method and computer readable storage medium

Abstract

A learning device, an image generating device, a learning method, an image generating method and a program are provided, which are capable of improving the estimation accuracy of a high brightness area in a wide-angle image to be generated. A second training data acquisition unit (64) acquires an input image. A wide-angle image generating unit (28) generates a generated wide-angle image based on an input of the input image, and the generated wide-angle image has a wider viewing angle than the input image. The second training data acquisition unit (64) acquires a comparison wide-angle image to be compared with the generated wide-angle image. A second learning unit (66) trains the wide-angle image generating unit (28) by updating a parameter value of the wide-angle image generating unit (28) based on a comparison result between the generated wide-angle image and the comparison wide-angle image, so as to increase the update amplitude of the parameter value of the pixel according to the brightness of the pixel in the comparison wide-angle image or the brightness of the pixel in the generated wide-angle image.

Description

Technical Field

[0001] The present invention relates to a learning device, an image generating device, a learning method, an image generating method and a program. Background Art

[0002] There is known a technology for IBL (Image Based Lighting) for generating a real CG (Computer Graphics) image and a composite image of the CG image and the real live-action image using a light source set based on a real live-action image.

[0003] In order to enhance the realism of the generated image, in IBL, it is desirable to set the light source based on a wide-angle image such as an HDR (High Dynamic Range) 360-degree image. However, in order to obtain such a wide-angle image, it is necessary to shoot using dedicated equipment such as a full sky camera or a half sky camera, and professional knowledge is also required.

[0004] Therefore, instead of generating such a wide-angle image by shooting, a wide-angle image is generated using an image generation unit implemented by a learned machine learning model, etc. In this case, in response to an image shot by a common camera being input into the image generation unit, a wide-angle image is generated in which an estimation result of an environment outside the field of view of the relevant image (e.g., an object, scene, etc. outside the field of view of the relevant image) is supplemented. Summary of the invention

[0005] [Technical issues]

[0006] However, this related art for generating a wide-angle image in the manner described above cannot properly estimate the high-brightness portion, and therefore, in some cases, a situation where the high-brightness portion is arranged at an unnatural position in the wide-angle image, or conversely, a situation where the high-brightness portion is not arranged at a position where it is to be arranged occurs. As a result, in some cases, for example, a situation where the light source setting in the IBL is unnatural occurs.

[0007] The present invention is made in view of the above-mentioned problems, and one of the objects of the present invention is to provide a learning device, an image generating device, a learning method, an image generating method and a program, which can improve the accuracy of estimation of high-brightness parts in the generated wide-angle image.

[0008] [Problem Solution]

[0009] In order to solve the above-mentioned problems, a learning device according to the present invention includes: an input image acquisition part, which acquires an input image; a wide-angle image generation part, which generates a generated wide-angle image in response to the input of the input image, and the generated wide-angle image is an image with a wider viewing angle than the input image; a comparison wide-angle image acquisition part, which acquires a comparison wide-angle image as an image to be compared with the generated wide-angle image; and a wide-angle image generation learning part, which performs learning on the wide-angle image generation part by updating the parameter value of the wide-angle image generation part based on the comparison result between the generated wide-angle image and the comparison wide-angle image, so that the update amount of the parameter value regarding the pixel is increased according to the brightness level of the pixel in the comparison wide-angle image or the brightness level of the pixel in the generated wide-angle image.

[0010] In one aspect of the present invention, the learning device also includes: an HDR image generating part, which generates a generated HDR image as a high dynamic range image in response to an input of a standard dynamic range image; a comparison HDR image acquiring part, which acquires a comparison HDR image as a high dynamic range image to be compared with the generated HDR image; and an HDR image generation learning part, which performs learning on the HDR image generating part by updating a parameter value of the HDR image generating part based on a comparison result between the generated HDR image and the comparison HDR image, wherein the input image acquiring part acquires the generated HDR image generated by the learned HDR image generating part as the input image.

[0011] In addition, the image generating device according to the present invention includes: an image acquiring part, which acquires a standard dynamic range image; an HDR image generating part, which generates a high dynamic range image in response to the input of the standard dynamic range image; and a wide-angle image generating part, which generates a wide-angle image having a wider viewing angle than the high dynamic range image in response to the input of the high dynamic range image.

[0012] In one aspect of the present invention, the wide-angle image generating part is a learned machine learning model, and learning is performed by updating the parameter value of the wide-angle image generating part based on the comparison result between the generated wide-angle image and the comparative wide-angle image, wherein the generated wide-angle image is generated by the wide-angle image generating part in response to the input of the input image, and the comparative wide-angle image is to be compared with the generated wide-angle image.

[0013] In this regard, the wide-angle image generation section is a learned machine learning model, and learning is performed by updating the parameter value of the wide-angle image generation section so that the update amount of the parameter value about the pixel is increased according to the brightness level of the pixel in the comparative wide-angle image or the brightness level of the pixel in the generated wide-angle image.

[0014] In addition, in one aspect of the present invention, the HDR image generating part is a learned machine learning model, and learning is performed by updating the parameter value of the HDR image generating part based on the comparison result between the high dynamic range image and the comparative high dynamic range image, wherein the high dynamic range image is generated by the HDR image generating part in response to the input of the standard dynamic range image, and the high dynamic range image is to be compared with the comparative high dynamic range image.

[0015] In addition, the learning method according to the present invention includes: a step of acquiring an input image; a step of generating a generated wide-angle image using a wide-angle image generating part in response to the input of the input image, wherein the generated wide-angle image is an image having a wider viewing angle than the input image; a step of acquiring a comparison wide-angle image as an image to be compared with the generated wide-angle image; and a step of performing learning on the wide-angle image generating part by updating a parameter value of the wide-angle image generating part based on a comparison result between the generated wide-angle image and the comparison wide-angle image, so that an update amount of the parameter value regarding the pixel is increased according to the brightness level of the pixel in the comparison wide-angle image or the brightness level of the pixel in the generated wide-angle image.

[0016] In addition, the image generation method according to the present invention includes: a step of acquiring a standard dynamic range image; a step of generating a high dynamic range image in response to the input of the standard dynamic range image; and a step of generating a wide-angle image having a wider viewing angle than the high dynamic range image in response to the input of the high dynamic range image.

[0017] In addition, the program according to the present invention causes a computer to execute: a process of acquiring an input image; a process of generating a generated wide-angle image using a wide-angle image generating part in response to the input of the input image, the generated wide-angle image being an image having a wider viewing angle than the input image; a process of acquiring a comparison wide-angle image as an image to be compared with the generated wide-angle image; and a process of performing learning on the wide-angle image generating part by updating a parameter value of the wide-angle image generating part based on a comparison result between the generated wide-angle image and the comparison wide-angle image, so that the update amount of the parameter value regarding the pixel is increased according to the brightness level of the pixel in the comparison wide-angle image or the brightness level of the pixel in the generated wide-angle image.

[0018] In addition, another program according to the present invention causes a computer to execute a process of acquiring a standard dynamic range image; a process of generating a high dynamic range image in response to input of the standard dynamic range image; and a process of generating a wide-angle image having a wider viewing angle than the high dynamic range image in response to input of the high dynamic range image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a configuration diagram of an image processing apparatus according to an embodiment of the present invention.

[0020] Figure 2 is a diagram showing an example of the first learning according to the embodiment of the present invention.

[0021] Figure 3 is a diagram showing an example of second learning according to an embodiment of the present invention.

[0022] Figure 4 is a diagram schematically showing an example of learning an input HDR image.

[0023] Figure 5 is a diagram schematically showing an example of contrasting wide-angle images.

[0024] Figure 6 is a diagram schematically showing an example of a generated wide-angle image.

[0025] Figure 7 is a diagram showing an example of a weight map.

[0026] Figure 8 is a diagram showing an example of image generation according to an embodiment of the present invention.

[0027] Fig. 9 is a functional block diagram showing an example of functions implemented by the image processing apparatus according to the embodiment of the present invention.

[0028] Fig.10 : is a flowchart showing an example of the flow of a learning process performed by the image processing apparatus according to the embodiment of the present invention.

[0029] Fig.11 : is a flowchart showing an example of the flow of a learning process performed by the image processing apparatus according to the embodiment of the present invention.

[0030] Fig.12 : is a flowchart showing an example of the flow of a learning process performed by the image processing apparatus according to the embodiment of the present invention.

[0031] Fig.13 : is a flowchart showing an example of the flow of a process for generating a target wide angle performed by the image processing apparatus according to the embodiment of the present invention. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present invention will be described in detail based on the accompanying drawings.

[0033] Figure 1 1 is a configuration diagram of an image processing device 10 according to the present embodiment. The image processing device 10 according to the present embodiment is a computer such as a game machine, a personal computer, or the like. Figure 1 As shown, the image processing apparatus 10 according to the present embodiment includes, for example, a processor 12 , a storage section 14 , an operation section 16 , and a display section 18 .

[0034] The processor 12 is a program-controlled device such as a CPU (Central Processing Unit), which operates according to a program installed in the image processing apparatus 10, for example.

[0035] The storage section 14 is a storage element such as a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk drive, etc. The storage section 14 stores programs executed by the processor 12 and the like.

[0036] The operation section 16 is a user interface including a keyboard, a mouse device, a controller of a game machine, and the like, and upon receiving an operation input by a user, outputs a signal indicating the content of the operation input to the processor 12 .

[0037] The display section 18 is a display device such as a liquid crystal display, and displays various images according to instructions from the processor 12 .

[0038] Note that the image processing apparatus 10 may include a communication interface such as a network board, an optical disk drive for reading a disk such as a DVD (Digital Versatile Disc)-ROM or a Blu-ray (registered trademark) disk, and a USB (Universal Serial Bus) port.

[0039] The image processing device 10 according to the present embodiment implements a learned machine learning model. In addition, an image is generated using a machine learning model, in which objects and scenes outside the perspective of the plane image are supplemented based on a plane image (two-dimensional image). Here, an HDR (high dynamic range) image with 32 bits per channel can be generated, for example, based on an SDR (standard dynamic range) plane image (two-dimensional image) with 8 bits per channel. In addition, the SDR plane image can be, for example, an image taken by a common camera. In addition, the generated HDR image can be, for example, a full sky image (360-degree image), a half sky image (180-degree image), etc. In addition, the generated HDR image can be a panoramic image.

[0040] Next, an example of a machine learning model implemented in the image processing device 10 will be described.

[0041] In the learning of the machine learning model implemented in the image processing device 10 according to the present embodiment, first, as Figure 2 As shown, the learning of the HDR image generating section 20 is performed. Here, the learning data to be used in the learning of the HDR image generating section 20 will be referred to as first learning data. Furthermore, hereinafter, the learning of the HDR image generating section 20 will be referred to as first learning.

[0042] The HDR image generation section 20 is a machine learning model implemented by, for example, a CNN (Convolutional Neural Network). For example, in response to an input of an 8-bit SDR image for each channel, the HDR image generation section 20 generates and outputs an HDR image associated with the SDR image and using 32 bits for each channel.

[0043] The first learning data includes, for example, a combination of a learning input SDR image 22 and a comparison HDR image 24 .

[0044] The learning input SDR image 22 included in the first learning data is, for example, an image input to the HDR image generation section 20 in the first learning. The learning input SDR image 22 is, for example, an SDR plane image such as an image captured by a common camera.

[0045] The comparison HDR image 24 included in the first learning data is, for example, an HDR plane image such as an image not generated by the HDR image generating section 20 but photographed by a general camera or the like.

[0046] In the present embodiment, for example, the learning input SDR image 22 is input to the HDR image generation section 20. Then, in response to the input of the learning input SDR image 22, the HDR image generation section 20 generates and outputs an HDR plane image associated with the learning input SDR image 22. The image generated in this manner will be referred to as a generated HDR image 26.

[0047] In addition, in the present embodiment, for example, an error (comparison result) between a generated HDR image 26 and a comparison HDR image 24 included in the first learning data is specified, and the generated HDR image 26 is output when the learning input SDR image 22 included in the first learning data is input to the HDR image generation section 20. Here, for example, an inter-pixel error such as an L1 error can be specified. In addition, based on the specified error, the parameter value of the HDR image generation section 20 is updated, for example, by an error back propagation method.

[0048] In this embodiment, for example, Figure 3As shown, the learning of the wide-angle image generating section 28 is performed separately from the above-mentioned first learning. Here, the learning data to be used in the learning of the wide-angle image generating section 28 will be referred to as second learning data. In addition, hereinafter, the learning of the wide-angle image generating section 28 will be referred to as second learning.

[0049] Next, an example of second learning will be described in which, using the conditional GAN ​​(Generative Adversarial Network) technique, not only the learning of the wide-angle image generating section 28 but also the learning of the domain identifying section 30 is performed together therewith.

[0050] The wide-angle image generation section 28 is a machine learning model implemented by CNN, for example. For example, an HDR plane image is input to the wide-angle image generation section 28 .

[0051] In addition, the wide-angle image generation part 28 estimates, for example, an environment outside the viewing angle of the input HDR plane image, such as an object and a scene outside the viewing angle of the image. Then, for example, the wide-angle image generation part 28 generates an image reflecting the estimation result, which has a wider viewing angle than any image included in the input plane image. Here, for example, a full sky image or a half sky image can be generated. In addition, for example, a panoramic image can be generated. In addition, the wide-angle image generation part 28 outputs the generated image.

[0052] For example, the domain recognition section 30 is a machine learning model implemented by CNN. For example, an image generated by the wide-angle image generation section 28 or an image different from the image generated by the wide-angle image generation section 28 is input to the domain recognition section 30. Then, the domain recognition section 30 outputs, for example, a recognition result indicating whether the image that has been input to the domain recognition section 30 is an image generated by the wide-angle image generation section 28. Here, the domain recognition section 30 may output data indicating the degree of possibility that the image input to the domain recognition section 30 is an image generated by the wide-angle image generation section 28.

[0053] The second learning data includes a learning input HDR image 32 and a comparison wide-angle image 34 .

[0054] The learning input HDR image 32 included in the second learning data is, for example, an HDR plane image such as an image captured by a general-purpose camera. Figure 4 3 is a diagram schematically showing an example of learning an input HDR image 32 according to the present embodiment. Here, Figure 4An image obtained by binarizing an HDR image is shown. Note that an HDR plane image generated by the learned HDR image generation section 20 can be used as the learning input HDR image 32. Here, for example, an HDR plane image generated by the learned HDR image generation section 20 in response to an input of a learning input SDR image 22 can be used as the learning input HDR image 32. In addition, an HDR plane image generated by the learned HDR image generation section 20 in response to an input of an SDR image different from the learning input SDR image 22 can be used as the learning input HDR image 32.

[0055] The comparison wide-angle image 34 included in the second learning data is, for example, an HDR wide-angle image that is not an image generated by the wide-angle image generating section 28 but is a full-sky image captured by a full-sky camera, a half-sky image captured by a half-sky camera, etc. In addition, the comparison wide-angle image 34 may be a panoramic image captured by a panoramic camera. Figure 5 3 is a diagram schematically showing an example of a comparative wide-angle image 34 according to the present embodiment. Here, Figure 5 An image obtained by binarizing the HDR image is shown.

[0056] In addition, an image associated with the learning input HDR image 32 included in the second learning data can be used as the comparison wide-angle image 34 included in the second learning data. The comparison wide-angle image 34 included in the second learning data can be, for example, an image captured at the same position as the learning input HDR image 32 included in the second learning data.

[0057] In the second learning, the learning input HDR image 32 included in the second learning data is input to the wide-angle image generating section 28. Then, in response to the input of the learning input HDR image 32, the wide-angle image generating section 28 generates an image having a wider viewing angle than the learning input HDR image 32. The image generated in this manner will be referred to as a generated wide-angle image 36. Figure 6 Schematically shows an example of a wide-angle image 36 generated according to the present embodiment. Figure 6 An image obtained by binarizing the HDR image is shown.

[0058] Furthermore, the generated wide-angle image 36 generated based on the learning input HDR image 32 included in the second learning data or the comparative wide-angle image 34 included in the second learning data is input to the domain identification portion 30 .

[0059] In addition, the domain identification section 30 outputs, for example, generation possibility data 38 indicating the degree of possibility that the image that has been input to the domain identification section 30 is the generated wide-angle image 36. Then, adversarial loss, which is an error (comparison result) between the generation possibility data 38 and the domain identification data 40 indicating which of the generated wide-angle image 36 and the comparison wide-angle image 34 has been input to the domain identification section 30, is specified.

[0060] Here, for example, as the value of the domain identification data 40 , a value associated with the generated wide-angle image 36 or the comparative wide-angle image 34 may be set.

[0061] For example, the generation possibility data 38 may be data taking a value greater than or equal to 0 and less than or equal to 1. In this case, the domain identification data 40 may be, for example, data taking a value of 1 in the case where the image that has been input to the domain identification section 30 is the generated wide-angle image 36 , and data taking a value of 0 in the case where the image that has been input to the domain identification section 30 is the comparative wide-angle image 34 .

[0062] In addition, the generation possibility data 38 and the domain identification data 40 may each be represented as a vector including, for example, two elements. For example, the value of the first element of the generation possibility data 38 may be a value indicating the degree of possibility that the image input to the domain identification part 30 is the generated wide-angle image 36. In addition, the value of the second element of the generation possibility data 38 may be a value indicating the degree of possibility that the image input to the domain identification part 30 is the comparative wide-angle image 34. In this case, the value of the domain identification data 40 associated with the generated wide-angle image 36 may be represented by a two-dimensional vector (1, 0), and the value of the domain identification data 40 associated with the comparative wide-angle image 34 may be represented by a two-dimensional vector (0, 1).

[0063] Furthermore, in the present embodiment, for example, the parameter values ​​of the wide-angle image generating section 28 or the domain identifying section 30 are updated by an error back propagation method based on the adversarial loss.

[0064] Here, in the second learning according to the present embodiment, the updating of the parameter values ​​of the wide-angle image generating section 28 using a predetermined number of second learning data sets and the updating of the parameter values ​​of the domain identifying section 30 using a predetermined number of second learning data sets may be repeated alternately. In this case, when the parameter values ​​of the wide-angle image generating section 28 are updated, the parameter values ​​of the wide-angle image generating section 28 are updated in a state where the parameter values ​​of the domain identifying section 30 are fixed. Furthermore, when the parameter values ​​of the domain identifying section 30 are updated, the parameter values ​​of the domain identifying section 30 are updated in a state where the parameter values ​​of the wide-angle image generating section 28 are fixed.

[0065] In addition, as described above, the second learning according to the present embodiment is performed using the conditional GAN ​​technique. Therefore, in the update of the parameter value of the wide-angle image generation part 28, not only the adversarial loss is used, but also the inter-pixel error between the generated wide-angle image 36 and the comparison wide-angle image 34, such as L1 loss, etc., is additionally used.

[0066] Furthermore, in the present embodiment, for example, for each pixel, the weight 42 is determined according to the size of the brightness value in the contrast wide-angle image 34. Furthermore, the corrected inter-pixel error is determined by weighting the aforementioned inter-pixel error using the weight 42. For example, for each pixel, the value of the corrected inter-pixel error is determined by multiplying the value of the inter-pixel error by the value indicating the weight 42. Furthermore, based on the adversarial loss determined for each pixel and the corrected inter-pixel error, the parameter value of the wide-angle image generation section 28 is updated, for example, by the error back propagation method.

[0067] Figure 7 4 is a diagram showing an example of a weight map 44 according to the present embodiment, in which an example of a weight 42 according to a luminance value is depicted. Figure 7 In the weight graph 44, the larger the value of the weight 42 indicating the pixel, the whiter the tone of the pixel depicted, and the smaller the value of the weight 42 indicating the pixel, the darker the tone of the pixel depicted. Here, Figure 7 An image obtained by binarization processing is shown.

[0068] Here, the weight 42 can be represented, for example, by an increasing function of the brightness value. For example, the weight 42 of each pixel can be set so that when the brightness value is less than a predetermined value, the weight 42 is set to 1, and when the brightness value is greater than or equal to the predetermined value, the larger the brightness value, the larger the weight 42. In addition, for example, the weight 42 of each pixel can be set so that when the brightness value is less than a predetermined value, the weight 42 is set to 1, and when the brightness value is greater than or equal to the predetermined value, the weight 42 is set to 2. Note that the relationship between the brightness value and the weight 42 is not limited to the above relationship.

[0069] In the present embodiment, the learning of the wide-angle image generating part 28 is performed in the manner as described above, i.e., by updating the parameter values ​​of the wide-angle image generating part 28 so that the update amount of the parameter values ​​related to the relevant pixels is increased according to the brightness level of the pixels in the comparison wide-angle image 34.

[0070] Note that, for each pixel, the weight 42 may be determined according to the magnitude of the brightness value not in the comparative wide-angle image 34 but in the generated wide-angle image 36. In this case, the learning of the wide-angle image generating section 28 is performed by updating the parameter value of the wide-angle image generating section 28 so that, according to the brightness level of the pixel in the generated wide-angle image 36, the update amount of the parameter value related to the relevant pixel is increased.

[0071] Furthermore, for each pixel, the weight 42 may be determined according to the size of the larger one of the brightness value of the relevant pixel in the comparison wide-angle image 34 and the brightness value of the relevant pixel in the generated wide-angle image 36. In this case, the parameter value of the wide-angle image generation section 28 is updated so that the update amount of the parameter value related to the relevant pixel is increased according to the brightness level of the pixel in the comparison wide-angle image 34 or the brightness level of the pixel in the generated wide-angle image 36.

[0072] Note that the learning method according to the present embodiment is not limited to the above-described learning method. For example, instead of using, for example, the conditional GAN ​​technique, the second learning may be performed based on, for example, only the inter-pixel error and the corrected inter-pixel error. In this case, as a result, the learning of the wide-angle image generation section 28 is performed without using the domain identification section 30.

[0073] Next, we will refer to Figure 8 The generation of an image using the learned HDR image generation section 20 and the learned wide-angle image generation section 28 is described.

[0074] First, an SDR plane image (eg, an image captured by a common camera or the like) is input to the learned HDR image generation section 20. Hereinafter, the image input in this manner will be referred to as a target input image 46.

[0075] Then, the HDR image generation section 20 generates and outputs an HDR plane image based on the target input image 46. Hereinafter, the HDR image output in this manner will be referred to as a target intermediate image 48.

[0076] Furthermore, the target intermediate image 48 is input to the learned wide-angle image generation section 28. Then, the wide-angle image generation section 28 generates and outputs an HDR image based on the target intermediate image 48. The image output here is, for example, an image supplemented with objects and scenes outside the angle of view of the target intermediate image 48, and the image has a wider angle of view than the target intermediate image 48. Hereinafter, the relevant image is referred to as a target wide-angle image 50.

[0077] A pixel with high brightness (e.g., a pixel with a large brightness value) is highly likely to be a pixel depicting a so-called light source (e.g., the sun or a lighting lamp). Furthermore, in the case where such a high brightness pixel is improperly estimated by the wide-angle image generation section 28, the high brightness portion is arranged at an unnatural position in the generated wide-angle image 36, or the high brightness portion is not arranged at a portion to be arranged. As a result, for example, a situation occurs in some cases where the light source setting in the IBL is unnatural.

[0078] In the present embodiment, as described above, the learning of the wide-angle image generation section 28 is performed by updating the parameter values ​​of the wide-angle image generation section 28 so that the update amount of the parameter values ​​about the relevant pixels is increased according to the brightness level of the pixels. Therefore, this configuration makes it possible to improve the accuracy of the estimation of high-brightness pixels by the wide-angle image generation section 28. Therefore, for example, the above configuration makes it possible to make the setting of the light source in IBL natural. In addition, the above configuration makes it possible to finely describe, for example, the reflection of a highlight object using the technology for IBL.

[0079] Here, the learning of the wide-angle image generating section 28 can be performed by, for example, updating the parameter values ​​of the wide-angle image generating section 28 so that the update amount of the parameter values ​​regarding the relevant pixels is increased according to the brightness levels of the pixels in the comparative wide-angle image 34. In this case, the possibility that a high brightness portion is not arranged at a portion to be arranged in the generated wide-angle image 36 can be reduced.

[0080] Furthermore, the learning of the wide-angle image generating section 28 may be performed by, for example, updating the parameter values ​​of the wide-angle image generating section 28 so that the update amount of the parameter values ​​with respect to the relevant pixels is increased according to the brightness levels of the pixels in the generated wide-angle image 36. In this case, the possibility that a high brightness portion is arranged at an unnatural position in the generated wide-angle image 36 can be reduced.

[0081] Furthermore, in the present embodiment, the weight 42 of each pixel is determined according to its brightness value based on an HDR image having a wide dynamic range, and thus, this configuration enables accurate weighting to be performed according to brightness, such as allowing particularly large weights 42 to be assigned to parts of particularly high brightness, etc.

[0082] Furthermore, in the case where the conditional GAN ​​technique is used in the second learning, the HDR image generated by the wide-angle image generating section 28 can be made a high-frequency image.

[0083] Furthermore, in the present embodiment, as described above, for example, the learning of the HDR image generation section 20 is performed separately from the learning of the wide-angle image generation section 28. Furthermore, the HDR image generation section 20 generates an HDR image based on the SDR image, and the wide-angle image generation section 28 generates an HDR wide-angle image based on the associated HDR image. In this way, the HDR wide-angle image is generated in two stages, and therefore, as a result, according to the present embodiment, an HDR wide-angle image with a high contrast is generated.

[0084] Note that the light source in IBL (Image Based Lighting) can be set based on the target wide-angle image 50 generated by the image processing device 10 according to the present embodiment. Note that the use of the target wide-angle image 50 generated by the image processing device 10 according to the present embodiment is of course not limited to IBL.

[0085] Next, the functions of the image processing apparatus 10 and the processing performed by the image processing apparatus 10 according to the present embodiment will be further described.

[0086] Fig. 9 1 is a functional block diagram showing an example of functions implemented by the image processing device 10 according to the present embodiment. Note that in the image processing device 10 according to the present embodiment, it is not necessary to implement Fig. 9 All functions shown in Fig. 9 One or more features other than those shown.

[0087] like Fig. 9 As shown, the image processing device 10 according to the present embodiment functionally includes, for example, an HDR image generating section 20, a wide-angle image generating section 28, a domain identifying section 30, a first learning data acquiring section 60, a first learning section 62, a second learning data acquiring section 64, a second learning section 66, a target input image acquiring section 68, a target intermediate image acquiring section 70, and a target wide-angle image acquiring section 72. The above elements are mainly implemented as a processor 12 and a storage section 14.

[0088] The image processing device 10 according to the present embodiment plays the role of a learning device for performing learning of a machine learning model implemented in the image processing device 10, and plays the role of an image generating device for performing image generation using the learned machine learning model. Fig. 9In the example of , the HDR image generating section 20, the wide-angle image generating section 28, the domain identifying section 30, the first learning data acquiring section 60, the first learning section 62, the second learning data acquiring section 64, and the second learning section 66 correspond to the roles as the learning device. In addition, the HDR image generating section 20, the wide-angle image generating section 28, the target input image acquiring section 68, the target intermediate image acquiring section 70, and the target wide-angle image acquiring section 72 correspond to the roles as the image generating device.

[0089] The above functions can be realized by allowing the processor 12 to execute a program installed in the image processing apparatus 10 as a computer and including instructions corresponding to the above functions. The program can be provided to the image processing apparatus 10 via a computer-readable information storage medium such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, or a flash memory, or via the Internet or the like.

[0090] As described above, in the present embodiment, for example, the HDR image generation section 20 generates and outputs an HDR plane image in response to an input of an SDR plane image. In addition, the HDR image generation section 20 may be implemented by a machine learning model such as CNN.

[0091] As described above, in the present embodiment, for example, the wide-angle image generation section 28 generates an image having a wider viewing angle than the relevant image in response to the input of an image. In addition, as described above, in the present embodiment, for example, the wide-angle image generation section 28 outputs the generated image, and may also output an HDR image in response to the input of an HDR image. In addition, the wide-angle image generation section 28 may be implemented by a machine learning model such as CNN.

[0092] As described above, in the present embodiment, for example, the domain identification section 30 outputs an identification result indicating whether the image input to the domain identification section 30 is an image generated by the wide-angle image generation section 28. In addition, the domain identification section 30 may be implemented by a machine learning model such as CNN or the like.

[0093] In the present embodiment, for example, the first learning data acquisition section 60 acquires first learning data including a combination of the learning input SDR image 22 and the comparison HDR image 24 .

[0094] In the present embodiment, for example, the first learning section 62 performs the first learning using the first learning data acquired by the first learning data acquisition section 60. Here, for example, based on the learning input SDR image 22 included in the first learning data, the generated HDR image 26 may be generated by the HDR image generation section 20. In addition, the learning of the HDR image generation section 20 may be performed by updating the parameter value of the HDR image generation section 20 based on the comparison result between the generated HDR image 26 and the comparison HDR image 24 included in the first learning data.

[0095] In the present embodiment, for example, the second learning data acquisition section 64 acquires second learning data including a combination of the learning input HDR image 32 and the comparison wide-angle image 34. Here, the second learning data acquisition section 64 may acquire, as the learning input HDR image 32, second learning data including the generated HDR image 26 generated by the learned HDR image generation section 20.

[0096] In the present embodiment, for example, the second learning portion 66 performs second learning using the second learning data acquired by the second learning data acquisition portion 64 .

[0097] Here, for example, based on the learning input HDR image 32 included in the second learning data, a generated wide-angle image 36 can be generated by the wide-angle image generation section 28. In addition, for each pixel in the generated wide-angle image 36 or the comparison wide-angle image 34 included in the second learning data, a weight 42 according to the brightness value of the pixel can be determined.

[0098] In addition, the pixels included in the generated wide-angle image 36 and the pixels included in the comparative wide-angle image 34 may be associated with each other in a one-to-one manner. In addition, for each pixel included in the comparative wide-angle image 34, a weight 42 according to a larger one of a brightness value of the relevant pixel and a brightness value of a pixel associated with the relevant pixel and included in the generated wide-angle image 36 may be determined as a weight 42 of the relevant pixel.

[0099] Furthermore, the learning of the wide-angle image generating section 28 may be performed by updating the parameter value of the wide-angle image generating section 28, thereby obtaining an update amount according to the determined weight 42 and the inter-pixel error between the generated wide-angle image 36 and the comparative wide-angle image 34. For example, for a certain pixel, the update amount of the parameter value related to the relevant pixel may be determined based on the value of the corrected inter-pixel error obtained by multiplying the value of the inter-pixel error between the generated wide-angle image 36 and the comparative wide-angle image 34 by the value of the weight 42 indicating the relevant pixel.

[0100] Here, learning of the wide-angle image generating section 28 may be performed by, for example, updating parameter values ​​of the wide-angle image generating section 28 such that the update amount of the parameter values ​​regarding relevant pixels is increased according to the brightness level of the pixels in the comparative wide-angle image 34 .

[0101] Furthermore, learning of the wide-angle image generating section 28 may be performed by, for example, updating parameter values ​​of the wide-angle image generating section 28 such that the update amount of the parameter values ​​regarding relevant pixels is increased according to the brightness level of the pixels in the generated wide-angle image 36 .

[0102] Furthermore, the parameter values ​​of the wide-angle image generation section 28 may be updated so that the update amount of the parameter values ​​related to the relevant pixels is increased, for example, according to the brightness level of the pixels in the comparative wide-angle image 34 or the brightness level of the pixels in the generated wide-angle image 36 .

[0103] In the present embodiment, for example, the target input image acquisition section 68 acquires the aforementioned target input image 46 , and inputs the image to the learned HDR image generation section 20 .

[0104] In the present embodiment, for example, the target intermediate image acquisition section 70 acquires the target intermediate image 48, which the HDR image generation section 20 generates and outputs based on the target input image 46 as described above. Furthermore, in the present embodiment, for example, the target intermediate image acquisition section 70 inputs the target intermediate image 48 to the wide-angle image generation section 28.

[0105] In the present embodiment, for example, the target wide-angle image acquisition section 72 acquires the target wide-angle image 50 , and the wide-angle image generation section 28 generates and outputs the target wide-angle image 50 based on the target intermediate image 48 as described above.

[0106] Here, the reference is Fig.10 An example of the flow of learning processing in the HDR image generating section 20, the wide-angle image generating section 28, and the domain identifying section 30 performed by the image processing device 10 according to the present embodiment is described with reference to the flowchart of the example in FIG.

[0107] First, the first learning data acquisition section 60 acquires a plurality of first learning data sets ( S101 ).

[0108] Then, the first learning section 62 performs first learning using the plurality of first learning data sets acquired in the process indicated in S101 (S102). Here, learning of the HDR image generation section 20 is performed.

[0109] In addition, the second learning data acquisition part 64 acquires a plurality of second learning data sets (S103). Here, for example, it is assumed that (m×2n) second learning data sets are acquired (m and n are each integers). It is assumed that the (m×2n) second learning data sets are divided into data groups consisting of 2n second learning data sets. Hereinafter, the i-th order data group will be referred to as the i-th data group (1￡i￡m).

[0110] Furthermore, the second learning section 66 sets the value of the variable i to 1 ( S104 ).

[0111] Then, the second learning section 66 performs second learning on the wide-angle image generating section 28 using the n second learning data sets included in the i-th data group (S105). In the process shown in S105, the parameter value of the wide-angle image generating section 28 is updated in a state where the parameter value of the domain identifying section 30 is fixed.

[0112] Then, the second learning section 66 performs second learning on the domain identification section 30 using the n remaining second learning data sets included in the i-th data set and not used in the process indicated by S105 (S106). In the process shown in S106, the parameter value of the domain identification section 30 is updated in a state where the parameter value of the wide-angle image generation section 28 is fixed.

[0113] In addition, the second learning section 66 confirms whether the value of the variable i is equal to m (S107). In the case where the value is not equal to m (S107: N), the second learning section 66 increases the value of the variable i by 1 (S108), and the flow returns to the processing indicated in S105. In the case where the value is equal to m (S107: Y), the processing indicated in this processing example ends.

[0114] Next, we will refer to Fig.11 The flowchart shown as an example in is used to describe an example of the flow of the processing indicated in S105 described above, that is, the second learning of the wide-angle image generating section 28 is performed using the n second learning data sets included in the i-th data group.

[0115] First, the second learning section 66 acquires a second learning data set on which the following processing indicated in S202 to S212 has not yet been performed, from among n second learning data sets included in the i-th data group ( S201 ).

[0116] Then, the second learning section 66 specifies the comparative wide-angle image 34 included in the second learning data set obtained in the process indicated in S201 ( S202 ).

[0117] Then, the second learning section 66 specifies the learning input HDR image 32 included in the second learning data set obtained in the process indicated in S201 ( S203 ).

[0118] Then, the second learning section 66 inputs the learning input HDR image 32 specified in the process indicated in S203 to the wide-angle image generating section 28 ( S204 ).

[0119] Then, the second learning section 66 acquires the generated wide-angle image 36 which is the output when the learning input HDR image 32 is input to the wide-angle image generating section 28 in the process indicated by S204 ( S205 ).

[0120] Then, the second learning section 66 inputs the comparison wide-angle image 34 designated in the process indicated by S202 or the generated wide-angle image 36 obtained in the process indicated by S205 to the domain identification section 30 ( S206 ).

[0121] Then, the second learning section 66 acquires the generation possibility data 38 which is the output according to the input of the domain identifying section 30 in the process indicated in S206 ( S207 ).

[0122] Then, the second learning section 66 calculates the adversarial loss based on the generation possibility data 38 obtained in the process indicated in S207 and the domain identification data 40 associated with the input in the process indicated in S206 ( S208 ).

[0123] Then, the second learning section 66 specifies the inter-pixel error between the comparison wide-angle image 34 specified in the process indicated by S202 and the generated wide-angle image 36 obtained in the process indicated by S205 ( S209 ).

[0124] Then, the second learning section 66 determines the weights 42 associated with the respective pixels included in the comparison wide-angle image 34 specified in the process indicated in S202 (S210). Here, for example, the weights 42 associated with the respective pixels included in the generated wide-angle image 36 obtained in the process indicated in S205 may be determined. In addition, for example, for each of the pixels, the weight 42 may be determined according to the size of the larger one of the brightness value of the relevant pixel in the comparison wide-angle image 34 or the brightness value of the relevant pixel in the generated wide-angle image 36.

[0125] Furthermore, the second learning section 66 calculates a corrected inter-pixel error based on the inter-pixel error specified in the process indicated by S209 and the weight 42 determined in the process indicated by S210 ( S211 ).

[0126] Then, the second learning section 66 updates the parameter value of the wide-angle image generation section 28 based on the adversarial loss calculated in the process indicated by S208 and the corrected inter-pixel error calculated in the process indicated by S211 (S212). Here, as described above, the parameter value of the wide-angle image generation section 28 is updated so that the update amount of the parameter value related to the relevant pixel is increased according to the brightness level of the pixel in the contrast wide-angle image 34. In addition, the parameter value of the wide-angle image generation section 28 can be updated in the direction in which the adversarial loss increases and in the direction in which the corrected inter-pixel error decreases, for example, by the error back propagation method.

[0127] In addition, the second learning section 66 confirms whether the processing indicated in S202 to S212 has been performed on the n second learning data sets included in the i-th data group (S213). In the case where the processing indicated in S202 to S212 has not been performed on all the second learning data (S213: N), the flow returns to the processing indicated in S201. In the case where the processing indicated in S202 to S212 has been performed on all the second learning data (S213: Y), the processing indicated in S105 ends.

[0128] Note that during one cycle of the processing indicated by S201 to S213, the processing indicated by S206 to S212 can be repeated twice for the corresponding cases (i.e., the case where the comparison wide-angle image 34 is input to the domain identification part 30, and the case where the generated wide-angle image 36 is input to the domain identification part 30).

[0129] Next, we will refer to Fig.12 The flowchart of the example shown in the figure is used to describe an example of the process of the processing indicated in S106 above, that is, the second learning of the domain identification part 30 using the n remaining second learning data sets included in the i-th data group and not used in the processing indicated in S105.

[0130] First, the second learning section 66 acquires one second learning data set on which the processing indicated in S302 to S312 has not been performed from among the n remaining second learning data sets included in the i-th data group and not used in the processing indicated in S105 (S301).

[0131] Then, the second learning section 66 specifies the comparative wide-angle image 34 included in the second learning data set, which has been obtained in the process indicated in S301 ( S302 ).

[0132] Furthermore, the second learning section 66 specifies the learning input HDR image 32 included in the second learning data set which has been obtained in the process indicated in S301 ( S303 ).

[0133] Then, the second learning section 66 inputs the learning input HDR image 32 designated in the process indicated in S303 to the wide-angle image generating section 28 ( S304 ).

[0134] Then, the second learning section 66 acquires the generated wide-angle image 36 which is the output when the learning input HDR image 32 is input to the wide-angle image generating section 28 in the process indicated by S304 ( S305 ).

[0135] Then, the second learning section 66 inputs the comparison wide-angle image 34 designated in the process indicated by S302 or the generated wide-angle image 36 obtained in the process indicated by S305 to the domain identification section 30 ( S306 ).

[0136] Then, the second learning section 66 acquires the generation possibility data 38 which is the output according to the input of the domain identifying section 30 in the process indicated by S306 ( S307 ).

[0137] Then, the second learning section 66 calculates the adversarial loss based on the generation possibility data 38 obtained in the process indicated in S307 and the domain identification data 44 associated with the input in the process indicated in S306 ( S308 ).

[0138] Then, the second learning section 66 updates the parameter value of the domain identification section 30 based on the adversarial loss calculated in the process indicated by S308 (S309). Here, the parameter value of the domain identification section 30 can be updated in the direction of reducing the adversarial loss, for example, by the error back propagation method.

[0139] In addition, the second learning section 66 confirms whether the processing indicated in S302 to S309 has been performed on all the remaining n second learning data sets included in the i-th data group and not used in the processing indicated in S105 (S310). In the case where the processing indicated in S302 to S309 has not been performed on all the second learning data (S310:N), the flow returns to the processing indicated in S301. In the case where the processing indicated in S302 to S309 has been performed on all the second learning data (S310:Y), the processing indicated in S106 ends.

[0140] Note that during one cycle of the processing shown in S301 to S313, the processing indicated in S306 to S309 can be repeated twice for the corresponding cases (i.e., the case where the comparison wide-angle image 34 is input to the domain identification part 30, and the case where the generated wide-angle image 36 is input to the domain identification part 30).

[0141] In addition, despite the Figure 10 to Figure 12 In the illustrated processing example, a different second learning data set is used for each of the processing indicated in S105 and the processing indicated in S106 , but the same second learning data set may be used in the processing indicated in S105 and the processing indicated in S106 .

[0142] Next, reference as Fig.13 An example of a flow of processing for generating the target wide-angle image 50 performed by the image processing device 10 according to the present embodiment will be described with reference to the flowchart shown by way of example in FIG.

[0143] First, the target input image acquisition section 68 acquires the target input image 46 ( S401 ).

[0144] Then, the target input image acquisition section 68 inputs the target input image 46 obtained in the process indicated in S401 to the HDR image generation section 20 ( S402 ).

[0145] Then, the HDR image generation section 20 generates the target intermediate image 48 from the target input image 46 input in the process indicated in S402, and outputs the target intermediate image 48 (S403).

[0146] Then, the target intermediate image acquisition section 70 acquires the target intermediate image 48 output in the process indicated in S403 ( S404 ).

[0147] Furthermore, the target intermediate image acquisition section 70 inputs the target intermediate image 48 acquired in the process indicated in S404 to the wide-angle image generation section 28 ( S405 ).

[0148] Then, the wide-angle image generation section 28 generates the target wide-angle image 50 based on the target intermediate image 48 input in the process indicated at S405, and outputs the target wide-angle image 50 (S406).

[0149] Then, the target wide-angle image acquisition section 72 acquires the target wide-angle image 50 output in the process indicated in S406 ( S407 ), and the process indicated in the present processing example ends.

[0150] Note that the present invention is not limited to the above-described embodiments.

[0151] Furthermore, the specific character strings and numerical values ​​described above and those in the drawings are exemplary representations, and the present invention is not limited to these character strings and numerical values.

Claims

1. A learning device, comprising: An input image acquisition part, which acquires an input image; a wide-angle image generating section that generates a generated wide-angle image in response to input of the input image, the generated wide-angle image being an image having a wider viewing angle than the input image; a comparison wide-angle image acquisition section that acquires a comparison wide-angle image as an image to be compared with the generated wide-angle image; as well as A wide-angle image generation learning section performs learning on the wide-angle image generation section by updating a parameter value of the wide-angle image generation section based on a comparison result between the generated wide-angle image and the comparison wide-angle image, so that an update amount of the parameter value with respect to a pixel in the comparison wide-angle image or a pixel in the generated wide-angle image is increased according to a brightness level of a pixel in the comparison wide-angle image or a brightness level of a pixel in the generated wide-angle image.

2. The learning device according to claim 1, further comprising: an HDR image generating section that generates a generated HDR image as a high dynamic range image in response to an input of a standard dynamic range image; a comparison HDR image acquisition section, which acquires a comparison HDR image as a high dynamic range image to be compared with the generated HDR image; as well as an HDR image generation learning section that performs learning on the HDR image generation section by updating a parameter value of the HDR image generation section based on a comparison result between the generated HDR image and the comparison HDR image, The input image acquisition section acquires the generated HDR image generated by the learned HDR image generation section as the input image.

3. An image generating device, comprising: An image acquisition part, which acquires a standard dynamic range image; An HDR image generating section, which generates a high dynamic range image in response to the input of the standard dynamic range image; as well as a wide-angle image generating section which generates a wide-angle image having a wider viewing angle than the high dynamic range image in response to the input of the high dynamic range image, In which, the wide-angle image generating part is a learned machine learning model, and learning is performed by updating the parameter value of the wide-angle image generating part based on the comparison result between the generated wide-angle image and the comparison wide-angle image. The generated wide-angle image is generated by the wide-angle image generating part in response to the input of the input image, and the comparison wide-angle image is to be compared with the generated wide-angle image.

4. An image generating device according to claim 3, wherein the wide-angle image generating part is a learned machine learning model, and learning is performed by updating the parameter value of the wide-angle image generating part, so that the update amount of the parameter value of the pixel in the comparison wide-angle image or the pixel in the generated wide-angle image is increased according to the brightness level of the pixel in the comparison wide-angle image or the brightness level of the pixel in the generated wide-angle image.

5. The image generating device according to claim 3 or 4, in, The HDR image generating part is a learned machine learning model, and learning is performed by updating the parameter value of the HDR image generating part based on the comparison result between the high dynamic range image and the comparative high dynamic range image. The high dynamic range image is generated by the HDR image generating part in response to the input of the standard dynamic range image, and the high dynamic range image is to be compared with the comparative high dynamic range image.

6. A learning method comprising: Steps to obtain input image; a step of generating a generated wide-angle image using a wide-angle image generating section in response to input of the input image, the generated wide-angle image being an image having a wider viewing angle than the input image; a step of acquiring a comparison wide-angle image as an image to be compared with the generated wide-angle image; as well as Based on the comparison result between the generated wide-angle image and the comparative wide-angle image, a step of performing learning on the wide-angle image generating part by updating the parameter value of the wide-angle image generating part makes the update amount of the parameter value regarding the pixel in the comparative wide-angle image or the pixel in the generated wide-angle image increase according to the brightness level of the pixel in the comparative wide-angle image or the brightness level of the pixel in the generated wide-angle image.

7. A method for generating an image, comprising: Steps to obtain a standard dynamic range image; In response to the input of the standard dynamic range image, a step of generating a high dynamic range image; as well as In response to the input of the high dynamic range image, a step of generating a wide-angle image having a wider viewing angle than the high dynamic range image using the learned machine learning model, In which, the learned machine learning model performs learning by updating the parameter value of the wide-angle image generating part based on the comparison result between the generated wide-angle image and the comparative wide-angle image, and the generated wide-angle image is generated by the wide-angle image generating part in response to the input of the input image, and the comparative wide-angle image is to be compared with the generated wide-angle image.

8. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a computer, causing the computer to perform: The process of obtaining the input image; a process of generating a generated wide-angle image using a wide-angle image generating section in response to input of the input image, the generated wide-angle image being an image having a wider viewing angle than the input image; a process of acquiring a comparison wide-angle image as an image to be compared with the generated wide-angle image; as well as Based on the comparison result between the generated wide-angle image and the comparative wide-angle image, a learning process is performed on the wide-angle image generating part by updating the parameter value of the wide-angle image generating part, so that the update amount of the parameter value regarding the pixel in the comparative wide-angle image or the pixel in the generated wide-angle image is increased according to the brightness level of the pixel in the comparative wide-angle image or the brightness level of the pixel in the generated wide-angle image.

9. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a computer, causing the computer to perform: The process of acquiring a standard dynamic range image; A process of generating a high dynamic range image in response to input of the standard dynamic range image; and In response to the input of the high dynamic range image, a process of generating a wide-angle image having a wider viewing angle than the high dynamic range image using the learned machine learning model, in, The learned machine learning model performs learning by updating the parameter value of the wide-angle image generating part based on the comparison result between the generated wide-angle image and the comparative wide-angle image, wherein the generated wide-angle image is generated by the wide-angle image generating part in response to the input of the input image, and the comparative wide-angle image is to be compared with the generated wide-angle image.

Images