Image processing apparatus and method, electronic apparatus, program, and storage medium

Abstract

To convert an image superimposing an object on the generated gain map into a desired image by using the gain map.SOLUTION: An image processing apparatus executes object superimposition processing on first image data in a first dynamic range associated with a first gain map for converting the first dynamic range into a second dynamic range, to generate superimposition image data. The image processing apparatus acquires conversion characteristics for converting image data of the object into image data in the second dynamic range on the basis of the luminance of the object, and generates a second gain map for converting the superimposition image data into second image data in the second dynamic range by using the conversion characteristics. The image processing apparatus updates an image file comprising the first image data and the first gain map by using the superimposition image data and the second gain map.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and method, an electronic device, a program, and a storage medium. [Background technology]

[0002] Conventionally, a technique is known in which a gain map describing the gain corresponding to each region is used to apply a different gain to each region of an image. For example, Patent Document 1 discloses a technique relating to a gain map that associates different gains with different regions of an image.

[0003] In recent years, as the brightness of displays has increased, High Dynamic Range (HDR) camera systems have been proposed that capture images that can reproduce the high-brightness gradations that were previously compressed with gradations closer to what the eye sees. HDR can express a wider dynamic range than Standard Dynamic Range (SDR).

[0004] Here, when an HDR image is to be displayed on an SDR display device (hereinafter referred to as an "SDR display device"), it is necessary to convert the HDR image to an SDR image, and a gain map can be applied to this conversion. For example, by generating and linking a gain map for conversion to SDR to an HDR image, it is possible to store HDR to SDR conversion information for each region of the HDR image. Furthermore, when an SDR image is displayed on an HDR display device (hereinafter referred to as an "HDR display device"), it is possible to use a gain map to convert the image into an image compatible with the HDR display device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-4675 Summary of the Invention [Problem to be solved by the invention]

[0006] Here, it is possible to superimpose an object such as text on an HDR image associated with a gain map. In this case, if the HDR image with the superimposed object is converted into an SDR image using the gain map associated with the HDR image, the image may not be as expected.

[0007] However, Patent Document 1 does not mention a gain map when a change occurs in the original image, and is unable to address the above-mentioned problem that occurs when an object is superimposed on an image.

[0008] The present invention has been made in consideration of the above problems, and has as its object to enable an image on which an object is superimposed after a gain map is generated to be converted into a desired image using the gain map. [Means for solving the problem]

[0009] In order to achieve the above object, the image processing device of the present invention has a superimposition means that performs object superimposition processing on first image data of a first dynamic range, which is associated with a first gain map for converting the first dynamic range to a second dynamic range, and generates superimposed image data; a generation means that acquires conversion characteristics for converting the image data of the object into image data of the second dynamic range based on the luminance of the object, and generates a second gain map using the conversion characteristics to convert the superimposed image data into second image data of the second dynamic range; and an update means that updates an image file consisting of the first image data and the first gain map using the superimposed image data and the second gain map. [Effects of the Invention]

[0010] According to the present invention, after the gain map is generated, an image on which an object is superimposed can be converted into a desired image by using the gain map. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the arrangement of an image processing apparatus according to a first embodiment of the present invention. [Figure 2] FIG. 1A is a diagram showing an example of a scene to be photographed in the first embodiment, FIG. 1B is a diagram showing an example of an HDR image, FIG. 1C is a diagram showing an example of an SDR image, and FIG. 1D is a diagram showing an example of a gain map. [Figure 3] FIG. 10 is a diagram showing the relationship between the luminance of a subject and the luminance values ​​of an HDR image and an SDR image. [Figure 4] 5 is a flowchart showing a process for superimposing objects in the first embodiment. [Figure 5] FIG. 3 is a conceptual diagram of object overlay processing in the first embodiment. [Figure 6] 3A and 3B are conceptual diagrams showing the structure of a superimposed image file and superimposition information in the first embodiment. [Figure 7] 5A and 5B are diagrams showing conversion characteristics according to the luminance of an object region in the first embodiment. [Figure 8] FIG. 4 is a conceptual diagram of a gain map synthesis method according to the first embodiment. [Figure 9] FIG. 10 is a block diagram showing an example of the arrangement of an image processing apparatus according to a second embodiment. [Figure 10] 10 is a flowchart showing a process for superimposing objects in the second embodiment. [Figure 11] FIG. 10 is a diagram for explaining a superposition gain map in the second embodiment. [Figure 12] FIG. 10 is a block diagram showing an example of the arrangement of an image processing device according to a third embodiment. [Figure 13] 11 is a flowchart showing a process for superimposing an object in the third embodiment. [Figure 14] FIG. 11 is a conceptual diagram of object superimposition processing according to the third embodiment. [Figure 15]FIG. 11 is a conceptual diagram of a gain map updated superimposed image file in the third embodiment. [Figure 16] FIG. 10 is a block diagram showing a schematic configuration of an imaging apparatus according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] First Embodiment FIG. 1 is a block diagram showing an example of the configuration of an image processing apparatus 10 according to the first embodiment of the present invention. The image processing device 10 includes an image input unit 101 , a superimposition information input unit 102 , an image superimposition unit 103 , a gain map generation unit 104 , a gain map synthesis unit 105 , a gain map application unit 106 , and an image output unit 107 .

[0014] The image input unit 101 inputs an image file 1000 that is configured from HDR image data, which is image data of an HDR image, and a gain map used in processing to convert the HDR image into an SDR image.

[0015] Here, HDR images, SDR images, and gain maps will be described in detail. In this embodiment, an HDR image is an image with a wider dynamic range than an SDR image, such as a YUV image to which the Opto-Electronic Transfer Function (OETF) characteristics described in ST2084, an HDR standard used in HDR display devices, are applied. An SDR image is a YUV image with a narrower dynamic range than an HDR image, such as a YUV image to which the sRGB gamma characteristics are applied. In this embodiment, the gamma of an HDR image is referred to as HDR gamma, and the gamma of an SDR image is referred to as SDR gamma, and the description will be given assuming that the color gamuts are the same.

[0016] Furthermore, the gain map of this embodiment is assumed to be one-channel data having gain information for the same number of pixels as the image to which it is added. The gain map of this embodiment records the ratio of the Y component of each pixel after de-gammaing the HDR image and the SDR image. Therefore, conversion from an HDR image to an SDR image can be performed by de-gammaing the HDR image, changing the Y component of each pixel with reference to the gain in the gain map, and then applying an SDR gamma. Conversely, conversion from an SDR image to an HDR image can be performed by de-gammaing the SDR image, changing the Y component of each pixel with reference to the gain in the gain map, and then applying an HDR gamma.

[0017] Hereinafter, the relationship between the HDR image, the SDR image, and the gain map will be described more specifically with reference to FIGS. FIG. 2(a) is a diagram showing an example of a scene to be photographed, where region 200 is the region with the highest brightness in the scene. FIG. 2(b) is a diagram showing an example of an HDR image generated by performing image processing on image data obtained by photographing the scene shown in FIG. 2(a) with an imaging device. FIG. 2(c) is a diagram showing an example of an SDR image generated by performing image processing on the same image data as FIG. 2(b). FIG. 2(d) is a diagram showing an example of a gain map that enables mutual conversion between the images shown in FIG. 2(b) and FIG. 2(c). An image file 1000 input to the image input unit 101 is made up of the HDR image shown in FIG. 2(b) and the gain map shown in FIG. 2(d).

[0018] FIG. 3 is a graph showing the relationship between the luminance of the subject and the luminance values ​​in the HDR image and the SDR image, with the horizontal axis representing the luminance of the subject and the vertical axis representing the luminance value of the image signal.

[0019] Here, the brightness of the brightest region 200 in the scene shown in Figure 2(a) (maximum subject brightness) is assumed to be lower than the maximum brightness that can be displayed by an HDR display device, but higher than the maximum brightness that can be displayed by an SDR display device. In the case of an HDR image, the brightness of region 200 can be directly assigned as a brightness value in the HDR display device, so each brightness in the scene can be directly converted to a brightness value in the HDR image. On the other hand, because the maximum brightness that can be displayed by an SDR display device is lower than the brightness of region 200, in the case of an SDR image, a brightness value is assigned so that the brightness of region 200 becomes the maximum brightness value of the SDR image.

[0020] Next, the process performed in the first embodiment when an object is superimposed on an image will be described with reference to the flowchart in FIG.

[0021] First, in S101 , the image input unit 101 acquires an image file 1000 and outputs the acquired image file 1000 to the image superimposition unit 103 .

[0022] In S102, the superimposition information input unit 102 receives user instructions 1001, which are instructions for image superimposition processing, from the user. The superimposition information input unit 102 is configured, for example, with a display that shows the user the details of the image and object superimposition processing, and a mouse, touch panel, or the like that receives user instructions. The user uses the superimposition information input unit 102 to instruct the type of superimposition processing to be performed on the image. The superimposition information input unit 102 outputs the received user instructions 1001 to the image superimposition unit 103.

[0023] The image superimposition unit 103 is a processing unit that performs superimposition processing on an image, and performs object superimposition processing on the HDR image data contained in the input image file 1000 based on the input user instructions 1001, thereby generating superimposed image data.

[0024] Fig. 5 shows an example of an image obtained when input HDR image data is subjected to a process of overlaying an object 501. Fig. 5(a) shows an example of the image before the object is overlaid, and Fig. 5(b) shows an example of the image after the object is overlaid.

[0025] Furthermore, the image superimposition unit 103 generates a superimposed image file 1030 composed of the generated superimposed image data and a gain map included in the image file 1000, and also generates superimposition information 1031 indicating what type of superimposition processing has been performed on the image. Here, the superimposition information 1031 includes, for example, area information of the area where the object is superimposed on the image. Fig. 6(a) is a conceptual diagram showing a superimposed image file 1030, and Fig. 6(b) is a conceptual diagram showing superimposition information 1031. Fig. 6(a) shows an example of superimposed image data 601 and a gain map 602 included in the superimposed image file 1030. Note that the gain map 602 is the same as the gain map shown in Fig. 2(d).

[0026] The image superimposing unit 103 outputs the generated superimposed image file 1030 and superimposition information 1031 to the gain map generating unit 104 . The gain map generation unit 104 generates a new gain map from the input superimposed image file 1030 and superimposition information 1031. More specifically, the gain map generation unit 104 de-gammas the superimposed image data included in the input superimposed image file 1030, and calculates which luminance value of the HDR image and the SDR image the luminance of each pixel corresponds to.

[0027] At this time, first in S103, when calculating the allocation of brightness values ​​of the SDR image, gain map generation unit 104 detects the maximum brightness Obj_max of the area where the object is superimposed (object area) using superimposition area information included in superimposition information 1031. Then, it is determined whether the maximum brightness Obj_max of the detected object area is equal to or less than the maximum brightness SDR_max that can be represented by the SDR display device, greater than the maximum brightness SDR_max that can be represented by the SDR display device and less than the maximum brightness Sbj_max of the subject (brightness of area 200), or equal to or greater than the maximum brightness Sbj_max of the subject.

[0028] If the maximum luminance Obj_max of the object region is equal to or less than the maximum luminance SDR_max of the SDR display device, the process proceeds to S104, where a first conversion characteristic such as that shown in graph 701 of FIG. 7 is selected. Then, a luminance value is assigned using the first conversion characteristic to the luminance obtained by de-gamma-ing superimposed image data 601 included in the input superimposed image file 1030. If the maximum luminance Obj_max of the object region is greater than the maximum luminance SDR_max of the SDR display device and less than the maximum luminance Sbj_max of the subject, the process proceeds to S105, where a luminance value is assigned using a second conversion characteristic such as that shown in graph 702. If the maximum luminance Obj_max of the object region is equal to or greater than the maximum luminance Sbj_max of the subject, the process proceeds to S106, where a luminance value is assigned using a third conversion characteristic such as that shown in graph 703.

[0029] Here, a case where the maximum luminance Obj_max of the object region is greater than the maximum luminance SDR_max of the SDR display device and less than the maximum luminance Sbj_max of the subject will be described in more detail. When the maximum luminance Sbj_max of the subject is assigned to be the maximum luminance value of the SDR image (graph 703), even though the maximum luminance Obj_max of the object region is greater than the maximum luminance SDR_max of the SDR display device, a luminance less than the maximum luminance of the SDR image is assigned due to the influence of the luminance of other regions (here, the maximum luminance Sbj_max of the subject). In this case, for example, when white text such as a caption or the shooting date and time is superimposed on an image as an object, the luminance of the superimposed region may decrease due to the influence of higher-luminance regions outside the superimposed region, causing the white text to appear gray in the SDR image.

[0030] Therefore, in this embodiment, a conversion characteristic is used that assigns the maximum luminance Obj_max of the object region to the maximum luminance value of the SDR image, as shown in graph 702. By assigning in this manner, the maximum luminance of the object region can be assigned to the luminance value of the SDR image without being affected by the luminance of other regions. That is, the slope of the graph 702 up to the maximum luminance Obj_max of the object region varies between a slope in which the maximum luminance Obj_max is assigned to the maximum luminance value of the SDR display device and a slope in which the maximum luminance Sbj_max of the subject is assigned to the maximum luminance value of the SDR display device, depending on the magnitude of the maximum luminance Obj_max.

[0031] In S107, the gain map generation unit 104 converts the luminance obtained by de-gamma-filtering the superimposed image data 601 included in the input superimposed image file 1030 into an HDR image using the conversion characteristics shown in Fig. 3. Then, the gain map generation unit 104 generates a superimposed gain map 1040 by recording the ratio of the Y component of each corresponding pixel between the converted HDR image and an image converted in any of S104 to S106. Then, the gain map generation unit 104 outputs the superimposed image file 1030, superimposition information 1031, and superimposed gain map 1040 to the gain map synthesis unit 105.

[0032] Then, in S 108 , the gain map synthesis unit 105 generates a synthesis gain map 1050 based on the input superimposed image file 1030 , superimposition information 1031 , and superimposition gain map 1040 . In this embodiment, the synthesis method involves obtaining the area where the object is superimposed from the superimposition information 1031, and replacing the gain of the object superimposition area, of the gains in the gain map 602 included in the superimposed image file 1030, with the gain in the superimposition gain map 1040. In this way, a synthesized gain map 1050 is generated. Fig. 8 is a conceptual diagram showing the method of synthesizing gain maps in this embodiment.

[0033] In this embodiment, the method of combining gain maps has been described in which the gain of the object overlap region in the original gain map is replaced with the gain of the gain map for the object overlap region, but the method of combining gain maps is not limited to this.

[0034] For example, the gain map 602 and the superimposition gain map 1040 may be compared for each pixel, and the larger gain may be used as the gain of the composite gain map. Alternatively, for example, if an object is semi-transparent and the original image and the object are to be composited at a certain composite ratio, a composite method may be used in which the gain of the gain map 602 and the gain of the superimposition gain map 1040 are composited according to the composite ratio.

[0035] The gain map synthesis unit 105 outputs the superimposed image file 1030 and the generated synthesis gain map 1050 to the gain map application unit 106.

[0036] In S109, the gain map assigning unit 106 assigns the composite gain map 1050 to the superimposed image data 601 included in the input superimposed image file 1030, and generates an updated superimposed image file 1060 made up of the superimposed image data 601 and the composite gain map 1050. Then, the gain map assigning unit 106 outputs the generated updated superimposed image file 1060 to the image output unit 107.

[0037] The image output unit 107 is a processing unit that outputs an image, and outputs the input updated superimposed image file 1060 to the outside.

[0038] As described above, according to the first embodiment, when an object is superimposed on an image to which a gain map has been applied, it is possible to generate a gain map corresponding to that processing. As a result, when converting between an HDR image and an SDR image using a gain map, it is possible to obtain the image that the user envisions.

[0039] Second Embodiment Next, a second embodiment of the present invention will be described. FIG. 9 is a block diagram showing an example of the configuration of an image processing device 20 according to the second embodiment. The image processing device 20 has an image input unit 101, a superimposition information input unit 102, an image superimposition unit 103, a gain map generation unit 201, a gain map assignment unit 202, and an image output unit 107. In Fig. 9, the same components as those shown in Fig. 1 described in the first embodiment are denoted by the same reference numerals, and their description will be omitted.

[0040] The process of superimposing an object on an image, which is performed in the second embodiment, will be described below with reference to the flowchart in Fig. 10. Note that in the process shown in Fig. 10, the same steps as those described in the first embodiment with reference to Fig. 4 are assigned the same step numbers, and their description will be omitted.

[0041] In S201, the gain map generation unit 201 generates a superimposition gain map that records the ratio of the Y component of each corresponding pixel between an image converted in any of S104 to S106 and an HDR image converted using the conversion characteristics shown in Fig. 3, with respect to the luminance obtained by de-gamma-ing the superimposition image data 601 included in the input superimposition image file 1030. Then, in order to reduce the data size of the superimposition gain map, the superimposition gain map is divided into blocks (30 divisions horizontally and 20 divisions vertically in this embodiment), and a partial gain map is generated that leaves only the blocks corresponding to the area where the object is superimposed, based on the superimposition information 1031. Then, the generated partial gain map is combined with gain map meta information that is information indicating whether or not a partial gain map corresponding to the divided block area exists, to generate a partial superimposition gain map 2010.

[0042] Fig. 11 is a conceptual diagram for explaining the partial superimposition gain map 2010. Fig. 11(a) is a diagram showing the superimposition gain map divided into blocks, Fig. 11(b) is a diagram showing the partial gain maps included in the partial superimposition gain map 2010, and Fig. 11(c) is a diagram showing gain map meta information. Here, the gain map meta information shown in Fig. 11(c) indicates that for each region obtained by dividing the image into blocks, there is information of 1 if there is a corresponding partial gain map, and 0 if there is not.

[0043] The gain map generation unit 201 outputs the superimposed image file 1030, the superimposition information 1031, and the generated partial superimposition gain map 2010 to the gain map application unit 202.

[0044] In S202, the gain map assigning unit 202 assigns a partial superimposition gain map to the image file to generate a new image file. Specifically, the gain map assigning unit 202 adds a partial superimposition gain map 2010 to the input superimposed image file 1030 to generate an updated superimposed image file 2020. The updated superimposed image file generated here in this embodiment is a file including multiple gain maps. The gain map assigning unit 2022 outputs the generated updated superimposed image file 2020 to the image output unit 107.

[0045] As described above, according to the second embodiment, when an object is superimposed on an image to which a gain map has been added, it is possible to generate a gain map corresponding to the processing. Furthermore, by excluding the gain map for areas other than the area where the object is superimposed, it is possible to reduce the file size.

[0046] <Third embodiment> Next, a third embodiment of the present invention will be described. FIG. 12 is a block diagram showing an example of the configuration of an image processing device 30 according to the third embodiment. The image processing device 30 has an image input unit 101, a superimposition information input unit 102, an image superimposition unit 301, a gain map generation unit 302, and an image output unit 107. In Fig. 12, the same components as those shown in Fig. 1 described in the first embodiment are denoted by the same reference numerals, and their description will be omitted.

[0047] The process of superimposing an object on an image, which is performed in the third embodiment, will be described below with reference to the flowchart in Fig. 13. In the process shown in Fig. 13, the same steps as those described in the first embodiment with reference to Fig. 4 are assigned the same step numbers, and their description will be omitted.

[0048] In S301, the image superimposition unit 301 performs object superimposition processing on HDR image data included in the input image file 1000 based on a user instruction 1001 input from the superimposition information input unit 102. In the third embodiment, the image file has a layer structure made up of multiple layers, and when processing such as object superimposition, the image superimposition unit 301 adds a layer for recording the processing content.

[0049] Fig. 14 is a conceptual diagram showing the process of superimposing an object on an original image in the object superimposition process. Fig. 14(a) shows an image file 1000 before the object superimposition process, and the image file 1000 consists of an HDR image 1401, which is the original image, and a gain map 1402 added to the HDR image 1401. Fig. 14(b) shows a superimposed image file 3010 generated after the object superimposition process, and consists of a layer consisting of the HDR image 1401 and the gain map 1402, and a layer recording the contents of the object superimposition process 1403c. The image superimposing unit 301 outputs the image file on which the object superimposing process has been performed as a superimposed image file 3010 to the gain map generating unit 302 .

[0050] Then, in S103, the magnitude of the maximum brightness Obj_max of the object region is calculated based on the contents of the object superimposition process 1403, and compared with the maximum brightness SDR_max of the SDR display device and the maximum brightness Sbj_max of the subject.

[0051] If the maximum luminance Obj_max of the object region is equal to or less than the maximum luminance SDR_max of the SDR display device, the process proceeds to S302, where a first conversion characteristic such as that shown in graph 701 of FIG. 7 is selected. Then, a luminance value is assigned using the first conversion characteristic to the luminance obtained by de-gamma-ing layers in the input superimposed image file 3010 to which no gain map has been assigned. If the maximum luminance Obj_max of the object region is greater than the maximum luminance SDR_max of the SDR display device and less than the maximum luminance Sbj_max of the subject, the process proceeds to S303, where a luminance value is assigned using a second conversion characteristic such as that shown in graph 702. If the maximum luminance Obj_max of the object region is equal to or greater than the maximum luminance Sbj_max of the subject, the process proceeds to S304, where a third conversion characteristic such as that shown in graph 703 is assigned.

[0052] Next, in S305, the gain map generation unit 302 generates a gain map that records the ratio of the Y component of each corresponding pixel between the HDR image 1401 included in the input superimposed image file 3010 and the image converted in any of S302 to S304. Then, in S306, the gain map generation unit 302 records the generated gain map in association with the layer of the object superimposition processing 1403 in the superimposed image file 3010, generates an updated superimposed image file 3020, and outputs it to the image output unit 107.

[0053] 15 shows a conceptual diagram of an updated superimposed image file 3020. The updated superimposed image file 3020 includes a layer in which an HDR image 1401 and a gain map 1402 added to the HDR image 1401 are recorded, and a layer in which the contents of an object superimposition process 1403 and a gain map 1404 corresponding to the contents of the object superimposition process 1403 are recorded.

[0054] By generating a gain map corresponding only to the object overlapping region in this way, it is possible to generate a gain map without being affected by regions other than the object overlapping region.

[0055] The number of layers on which the object overlay processing is recorded is not limited to one, and multiple layers may be used. For example, if an object is to be overlaid on an image file that has multiple layers on which images, each with an associated gain map, are recorded, it is possible to add an additional layer and perform gain map generation processing corresponding to that layer. By performing such processing, even if an object is overlaid on an image that already has an object overlaid, it is possible to generate a gain map suitable for the overlaid object.

[0056] Furthermore, if you later want to remove the overlaid object, you can remove the overlaid layer of that object and the gain map associated with that layer, thereby removing the influence of the overlaid object from the image and gain map.

[0057] As described above, according to the third embodiment, when an object is superimposed on an image to which a gain map has been added, it is possible to generate a corresponding gain map. Furthermore, by using a layer structure, it is possible to easily return to the gain map before processing when canceling the processing.

[0058] In the first to third embodiments described above, the input image file 1000 is configured from an HDR image and a gain map, but it may be an image file configured from an SDR image and a gain map. Furthermore, the gain map in this embodiment has the same number of pixels as the image, but is not limited to this. For example, the image may be reduced, and the gain map may have the same number of pixels as the reduced image. In this case, the image is scaled to the same number of pixels as the image before reduction before use.

[0059] Furthermore, although the gain of each pixel in the gain map of this embodiment is created from the ratio of Y values ​​after de-gammaing the HDR image and SDR image, the creation method is not limited to this. For example, the gain may be created by converting an HDR image into an RGB image to which HDR gamma has been applied, or an SDR image into an RGB image to which SDR gamma has been applied, and then using the ratio of one of the RGB values. Furthermore, although the gain map of this embodiment is one-channel data, the present invention is not limited to this. For example, the gain map may have a three-channel data configuration, with gains for each of the R, G, and B colors of an RGB image.

[0060] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described. 16 is a block diagram showing an example of the basic functional configuration of a digital camera 1600 (hereinafter referred to as camera 1600) according to a fourth embodiment of the present invention, as an example of a device equipped with the image processing device 10, 20, or 30 described in any of the first to third embodiments. Note that the imaging device to which the present invention can be applied may be any electronic device equipped with a camera function. Examples of electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, drive recorders, etc. These are merely examples, and the present invention can also be applied to other electronic devices.

[0061] In the camera 1600, the optical system 1601 is an imaging optical system that includes a lens group, a shutter, an aperture, etc., and forms an optical image of a subject on the imaging plane of the image sensor 1602. The lens group includes fixed lenses and movable lenses, and the movable lenses include a lens for image stabilization, a focus lens, a variable magnification lens, etc. The aperture may also function as a mechanical shutter. The operations of the movable lenses, aperture, and shutter are controlled by a CPU 1603, which is the main control unit of the camera 1600. The optical system 1601 may be configured as an integral part of the camera 1600, or may be configured to be replaceable.

[0062] The image sensor 1602 is, for example, a CMOS image sensor, and has a plurality of pixels, each having a photoelectric conversion region, arranged two-dimensionally. The image sensor 1602 also has a color filter with a specific color pattern, and each pixel is provided with a filter of one color corresponding to the color pattern. The image sensor 1602 photoelectrically converts the optical image formed on the light receiving surface by the optical system 1601 at each pixel, and converts it into an analog image signal indicating the luminance information for each pixel.

[0063] An analog image signal generated by the image sensor 1602 is converted into a digital image signal by an A / D converter (not shown). The image sensor 1602 may have the A / D converter, or the CPU 1603 may perform the A / D conversion. The pixel signals constituting the digital image signal obtained by the A / D conversion are RAW data having only the luminance components of the colors of the color filters provided in the pixels that generated the signals. The CPU 1603 stores this RAW data in the primary storage device 1604. The imaging sensitivity of the image sensor 1602 is set by the CPU 1603, for example, in accordance with a user instruction via the operation unit 1609, photometry results, etc.

[0064] The CPU 1603 controls each unit of the camera 1600 and realizes various functions of the camera 1600 by transferring a program stored in the secondary storage device 1607 to the primary storage device 1604 and executing the program. Note that in the following description, at least some of the functions realized by the CPU 1603 executing the program may be realized by dedicated hardware such as an ASIC.

[0065] The primary storage device 1604 is a volatile storage device such as a RAM, etc. The primary storage device 1604 is used by the CPU 1603 to execute programs, and is also used as a buffer memory for image data, a work area for image processing, a video memory for display, etc.

[0066] The secondary storage device 1607 is a rewritable nonvolatile storage device such as an EEPROM, etc. The secondary storage device 1607 stores programs (instructions) that can be executed by the CPU 1603, settings for the camera 1600, GUI data, etc.

[0067] The recording medium 1606 is a rewritable nonvolatile storage device such as a semiconductor memory card. The recording medium 1606 may be built into the camera 1600 or may be configured to be detachable from the camera 1600. Data (still image data, video data, audio data, etc.) generated by the camera 1600 can be recorded on the recording medium 1606. That is, the camera 1600 has a function for reading and writing from and to the recording medium 1606, and, if the recording medium 1606 is detachable, an attachment / detachment mechanism. Note that the recording destination of the data generated by the camera 1600 is not limited to the recording medium 1606; for example, the data may be transmitted to an external device via a communication interface of the camera 1600 and recorded on a recording device accessible by the external device.

[0068] The display unit 1608 is configured by, for example, a liquid crystal display. The CPU 1603 functions as a display control device for the display unit 1608. In a shooting standby state or while recording moving images, the display unit 1608 displays captured moving images in real time, and the display unit 1608 functions as an electronic viewfinder. The display unit 1608 also displays image data recorded on the recording medium 1606 and GUI images such as menu screens.

[0069] The operation unit 1609 is a general term for a group of input devices that accept user operations, and may include, for example, buttons, levers, and touch panels. The operation unit 1609 may also include input devices that do not require physical operation, such as voice or line of sight. The input devices included in the operation unit 1609 are given names according to the functions assigned to them, and examples of such input devices include a shutter button, menu button, directional keys, a decision (set) button, and a mode switching dial. Different functions may be selectively assigned to one input device. The operation unit 1609 may constitute a part of the superimposition information input unit 102.

[0070] The image processing unit 1605 includes the configuration of any of the image processing devices 10, 20, and 30 described in the first to third embodiments, and applies predetermined image processing to image data (which may be RAW data or image data after development processing), generates image data in different formats, and acquires and / or generates various types of information. The image processing unit 1605 may be, for example, a dedicated hardware circuit such as an ASIC designed to achieve a specific function, or may be configured to achieve a specific function by a programmable processor such as a DSP executing software.

[0071] The image processing applied by the image processing unit 1605 may include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on the imaging element, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data.It is also called demosaic processing. The correction processing may include white balance adjustment, tone correction, correction of image degradation caused by optical aberration of the optical system 1601 (image restoration), correction of the effects of vignetting of the optical system 1601, color correction, and the like.

[0072] The detection process may include detection of characteristic regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing may include superimposition, gain map generation, area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generating signals and evaluation values ​​used in autofocus (AF) detection, and generating evaluation values ​​used in automatic exposure control (AE). The special effect processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are examples of processing that can be applied by the image processing unit 1605, and do not limit the processing that can be applied by the image processing unit 1605.

[0073] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0074] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] <Summary> The disclosure of this embodiment includes the following configuration.

[0076] (Item 1) a superimposing unit that performs a superimposing process of an object on first image data of a first dynamic range, the first image data being associated with a first gain map for converting the first dynamic range into a second dynamic range, to generate superimposed image data; a generation means for acquiring a conversion characteristic for converting image data of the object into image data of the second dynamic range based on the luminance of the object, and generating a second gain map for converting the superimposed image data into second image data of the second dynamic range using the conversion characteristic; an updating means for updating an image file consisting of the first image data and the first gain map using the superimposed image data and the second gain map; 1. An image processing device comprising: (Item 2) the update means has a synthesis means for synthesizing the first gain map and the second gain map to generate a synthetic gain map; 2. The image processing device according to item 1, wherein the update means replaces the first image data and the first gain map of the image file with the superimposed image data and the composite gain map, respectively. (Item 3) 3. The image processing device according to item 2, wherein the synthesizing means replaces the gain of the area of ​​the first gain map corresponding to the area of ​​the object with the gain of the second gain map. (Item 4) 3. The image processing device according to item 2, wherein the synthesis means generates a synthetic gain map by selecting the larger gain from among the gains constituting the first gain map and the second gain map. (Item 5) The image processing device described in item 2 is characterized in that the synthesis means replaces the gain of the area of ​​the first gain map corresponding to the area of ​​the object with a gain obtained by synthesizing the gain of the first gain map and the gain of the second gain map using a synthesis ratio of the first image data and the image data of the object used in the superposition process. (Item 6) The image processing device described in item 1 is characterized in that the update means generates a partial gain map of the second gain map that corresponds to the region of the object and meta information that represents the region of the first gain map that corresponds to the region of the object, replaces the first image data of the image file with the superimposed image data, and adds the partial gain map and the meta information. (Item 7) the superimposing means generates image data of the object in the first dynamic range as the superimposed image data; 2. The image processing device according to item 1, wherein the update means adds the superimposed image data and the second gain map to the image file. (Item 8) The generating means converting the superimposed image data into the second image data in the second dynamic range using the conversion characteristic; A gain for converting the superimposed image data into the second image data is calculated for each predetermined region to generate the second gain map. 8. The image processing device according to any one of items 1 to 7, characterized in that: (Item 9) 9. The image processing device according to item 8, wherein the predetermined region is each pixel. (Item 10) 9. The image processing device according to item 8, wherein the predetermined areas are areas obtained by dividing a plurality of pixels that make up the second image data. (Item 11) an imaging means for capturing an image and generating and outputting image data in the first dynamic range; An image processing device according to any one of items 1 to 10; An electronic device comprising: (Item 12) An image processing device according to any one of items 1 to 10, an output means for converting the image data of the first dynamic range included in the image file output from the image processing device using a gain map included in the image file and outputting the converted image data to the display device of the second dynamic range; An electronic device comprising: (Item 13) a superimposing step of performing an object superimposing process on first image data of a first dynamic range, the first image data being associated with a first gain map for converting the first dynamic range into a second dynamic range, to generate superimposed image data; a generating step of acquiring a conversion characteristic for converting image data of the object into image data of the second dynamic range based on the luminance of the object, and generating a second gain map using the conversion characteristic for converting the superimposed image data into second image data of the second dynamic range; an updating step of updating an image file consisting of the first image data and the first gain map using the superimposed image data and the second gain map; An image processing method comprising: (Item 14) A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 10. (Item 15) Item 15. A computer-readable storage medium storing the program described in item 14.

[0077] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0078] 10, 20, 30... image processing device, 101... image input unit, 102... superimposition information input unit, 103, 301... image superimposition unit, 104, 201, 302... gain map generation unit, 105... gain map synthesis unit, 106, 202... gain map assignment unit, 107... image output unit

Claims

1. a superimposing unit that performs a superimposing process of an object on first image data of a first dynamic range, the first image data being associated with a first gain map for converting the first dynamic range into a second dynamic range, to generate superimposed image data; a generating means for acquiring a conversion characteristic for converting image data of the object into image data of the second dynamic range based on the luminance of the object, and generating a second gain map for converting the superimposed image data into second image data of the second dynamic range using the conversion characteristic; an updating means for updating an image file consisting of the first image data and the first gain map using the superimposed image data and the second gain map; 1. An image processing device comprising:

2. the update means includes a synthesis means for synthesizing the first gain map and the second gain map to generate a synthetic gain map; 2. The image processing apparatus according to claim 1, wherein the updating means replaces the first image data and the first gain map of the image file with the superimposed image data and the composite gain map, respectively.

3. 3. The image processing apparatus according to claim 2, wherein the synthesizing means replaces the gain of the area of ​​the first gain map corresponding to the area of ​​the object with the gain of the second gain map.

4. 3. The image processing device according to claim 2, wherein the combining means generates a combined gain map by selecting a larger gain from among the gains constituting the first gain map and the second gain map.

5. 3. The image processing device according to claim 2, wherein the combining means replaces the gain of a region of the first gain map corresponding to the region of the object with a gain obtained by combining the gain of the first gain map and the gain of the second gain map using a combination ratio of the first image data and the image data of the object used in the superposition process.

6. The image processing device according to claim 1, characterized in that the updating means generates a partial gain map of the second gain map corresponding to the region of the object and meta information representing the region of the first gain map corresponding to the region of the object, replaces the first image data of the image file with the superimposed image data, and adds the partial gain map and the meta information.

7. the superimposing means generates image data of the object in the first dynamic range as the superimposed image data; 2. The image processing apparatus according to claim 1, wherein the updating means adds the superimposed image data and the second gain map to the image file.

8. The generating means converting the superimposed image data into the second image data in the second dynamic range using the conversion characteristic; A gain for converting the superimposed image data into the second image data is calculated for each predetermined region to generate the second gain map.

2. The image processing device according to claim 1, wherein:

9. 9. The image processing device according to claim 8, wherein the predetermined region is each pixel.

10. 9. The image processing apparatus according to claim 8, wherein the predetermined area is each area obtained by dividing a plurality of pixels that constitute the second image data.

11. an imaging means for capturing an image and generating and outputting image data in the first dynamic range; The image processing device according to any one of claims 1 to 10, An electronic device comprising:

12. An image processing device according to any one of claims 1 to 10; an output means for converting the image data of the first dynamic range included in the image file output from the image processing device using a gain map included in the image file and outputting the converted image data to the display device of the second dynamic range; An electronic device comprising:

13. a superimposing step of performing an object superimposing process on first image data of a first dynamic range, the first image data being associated with a first gain map for converting the first dynamic range into a second dynamic range, to generate superimposed image data; a generating step of acquiring a conversion characteristic for converting image data of the object into image data of the second dynamic range based on the luminance of the object, and generating a second gain map using the conversion characteristic for converting the superimposed image data into second image data of the second dynamic range; an updating step of updating an image file consisting of the first image data and the first gain map using the superimposed image data and the second gain map; An image processing method comprising:

14. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 10.

15. A computer-readable storage medium storing the program according to claim 14.

Images