Image processing device, control method, and program

Abstract

In an augmented image, image processing is performed while mitigating the impact of the augmented region on the original region, and while also taking the augmented region into consideration. [Solution] The image processing device has an image extension function that expands image data. The image processing device generates region identification information that identifies the original image region before expansion and the expanded region included in the expanded image data expanded by the image extension function. Furthermore, the parameters for correction processing of the expanded image data are determined based on the characteristic information of each region including the original image region before expansion and the expanded region included in the expanded image data, and the expanded image data is corrected using the determined parameters.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method, and a program, and particularly to image processing for an extended image.

Background Art

[0002] In recent years, with the advancement of generative AI technology, it has become possible to expand image data. As a result, it has become possible to generate areas that did not exist in the original image and create an image with a wider field of view, or to generate a subject that did not exist in the original image in a specific area of the image. Consider applying such a technique to an image that requires post-processing, such as a RAW image. In such a case, the extended area may have characteristics different from those of the original area, and as a result, an extended image may contain areas with different characteristics. In such an image, due to the difference in characteristics between the original area and the extended area, differences in color tone and brightness between those areas may be visualized by post-processing, resulting in an unnatural image.

[0003] As a conventional technique related to white balance, which is one of the post-processings of an image, there are, for example, the techniques described in Patent Document 1 and Patent Document 2. In Patent Document 1, when an image is cut out at a set aspect ratio during image recording, a configuration is disclosed in which an evaluation value for white balance control is calculated by changing the usage ratio between the entire cut-out image and the common area of the image used regardless of the aspect ratio. In Patent Document 1, when the main subject is included in the non-common area that is used or not used depending on the aspect ratio setting, the usage ratio of the common area is increased to calculate the color level for white balance control.

[0004] Furthermore, Patent Document 2 discloses a configuration in which, when a specific area is cropped and saved as an electronic zoom area during image recording, color extraction for white balance is also performed from outside the cropped area. In Patent Document 2, if a light source different from the light source illuminating the subject is located outside the electronic zoom area, the weighting of the colors extracted from the outside is reduced to perform white balance control. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2012-039256 [Patent Document 2] Japanese Patent Publication No. 2004-064676 [Overview of the project] [Problems that the invention aims to solve]

[0006] Patent Document 1 suppresses changes in brightness and color balance of main subjects, such as a person's face, due to changes in the aspect ratio setting, and controls the difference in white balance between images with different aspect ratios. Patent Document 2, on the other hand, assumes the cropping and saving of a specific field of view, and aims to appropriately adjust the color balance and brightness of the cropped image by effectively utilizing the information of the area outside the cropped area. For example, if a light source different from the light source illuminating the subject is outside the electronic zoom area, the weighting of the colors extracted from the outside is reduced, but if the light source is inside the electronic zoom area, the weighting is not changed.

[0007] Even if such conventional technologies are applied to image data augmented by AI generation or the like, only the effects described in each patent document will be applied to the augmented image data. Therefore, it is difficult to improve image quality, such as the unnaturalness of the image after post-processing, which is caused by the difference in characteristics between the original region and the augmented region.

[0008] This invention has been made in view of the above-mentioned problems, and aims to enable image processing that takes the expanded region into consideration while reducing the influence of the expanded region on the original region in an expanded image. [Means for solving the problem]

[0009] To achieve the above objective, according to one aspect of the present invention, a generation means for generating region identification information that identifies the original image region before expansion and the expanded region included in the expanded image data expanded by the image expansion function, A determination means for determining the parameters of the correction process for the extended image data based on feature information for each region, including the original image region before extension and the extended region, which are included in the extended image data generated by the image extension function. The system includes a correction means for correcting the extended image data using the determined parameters. An image processing device characterized by the above is provided. [Effects of the Invention]

[0010] According to the present invention, it is possible to perform image processing that takes the expanded region into account while reducing the influence of the expanded region on the original region in the expanded image. [Brief explanation of the drawing]

[0011] [Figure 1] Block diagram illustrating the hardware configuration of the imaging device 100 according to embodiments and modified examples of the present invention. [Figure 2] Flowchart illustrating the extended image storage process and extended image development process performed in the imaging device 100 according to embodiments and modifications of the present invention. [Figure 3] A diagram illustrating the extended image storage process performed in the imaging device 100 according to embodiments and modified examples of the present invention. [Figure 4] A diagram illustrating the development parameter calculation process performed in the imaging device 100 according to Modification 1 of the present invention. [Figure 5] A diagram illustrating the histogram calculation process performed in the imaging device 100 according to a modified example 2 of the present invention. [Figure 6] A diagram illustrating the extended region acquisition process performed in the imaging device 100 according to Modification 3 of the present invention. [Figure 7] A diagram illustrating an example of user operation according to Embodiment 2 of the present invention. [Figure 8] A flowchart illustrating the extended image development process performed in the imaging device 100 according to Embodiment 2 of the present invention. [Modes for carrying out the invention]

[0012] [Embodiment 1] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0013] One embodiment described below illustrates an example of applying the present invention to an imaging device, which is an example of an image processing device, and has a function that performs image processing that takes into account the expanded region while reducing the influence of the expanded region on the original region in an expanded image. However, the present invention is applicable to any device that can generate an image that has undergone image processing that takes into account the expanded region while reducing the influence of the expanded region on the original region in an expanded image.

[0014] 《Imaging Device Hardware Configuration》 The hardware configuration of the imaging device 100 according to this embodiment will be illustrated below using the block diagram in Figure 1. The imaging device 100 may include, for example, devices provided for imaging purposes such as digital cameras and digital video cameras, or electronic devices equipped with imaging functions such as camera-equipped mobile phones and camera-equipped computers.

[0015] The optical system 101 is an imaging optical system including a lens group, a shutter, an aperture, etc. The lens group can include a correction lens for correcting camera shake and the like, a focus lens, and the like. The optical system 101 forms an image of subject light on the imaging surface of the imaging device 102 based on a control signal received from the CPU 103 described later. The imaging device 102 is an imaging sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging device 102 converts an optical image formed on the imaging surface by the optical system 101 into an image signal by photoelectric conversion. The image signal is digitized to become digital image data (referred to as image data), and is stored or subjected to image processing.

[0016] The CPU 103 is a control device that controls the operations of each block of the imaging device 100. The CPU 103 reads out the operation programs of each block stored in the secondary storage unit 107, expands them in the primary storage unit 104, and executes them to control the operations of each block. In controlling the operations of each block, the CPU 103 appropriately sends a control signal corresponding to the applicable operation to each block. The CPU 103 may also be referred to as a processor.

[0017] The secondary storage unit 107 is a non-volatile storage device such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), for example. The secondary storage unit 107 stores various setting information necessary for the operations of each block in addition to the operation programs of each block of the imaging device 100 and the firmware of the imaging device 100. The primary storage unit 104 is a volatile storage device such as a RAM (Random Access Memory), for example. The primary storage unit 104 is used not only as an expansion area for the operation programs of each block but also as a work memory for storing temporary data output by the operations of each block. The information stored in the primary storage unit 104 can be used by the image processing device 105 or recorded on the recording medium 106.

[0018] The recording medium 106 is a recording device configured to be detachable from the imaging device 100, such as a semiconductor memory card. When imaging is performed by the imaging device 100, the recording medium 106 is used to record the image data stored in the primary storage unit 104 as a result of the imaging. The data recorded on the recording medium 106 can be used on an external device such as a PC (personal computer) by attaching the recording medium 106 to the external device.

[0019] The display unit 108 is a display device such as an LCD. The display unit 108 is used to display the viewfinder image during shooting, the captured image, and GUI (Graphical User Interface) images for interactive operation. When shooting with the imaging device 100, the display unit 108 can display a live view. In addition, selected image data from the captured image data (image files) stored on the recording medium 106 can be displayed on the display unit 108.

[0020] The operation unit 109 is a user interface such as a button, lever, or touch panel. When an operation input is received for any of the operation components, the operation unit 109 transmits a control signal corresponding to that operation input to the CPU 103. In addition, the operation unit 109 may also include input devices that use voice or eye gaze. Furthermore, the operation unit 109 includes a touch panel integrated with the display unit 108. The user can perform operations on the display of the display unit 108 using the touch panel.

[0021] The image processing unit 105 is a device that performs various image processing on an image. In this embodiment, the image processing unit 105 is configured to be able to apply multiple types of image processing to an image. Which image processing the image processing unit 105 performs may be defined as a pattern for each shooting mode, for example. In this embodiment, the pattern of image processing that the image processing unit 105 applies to the captured image is controlled based on the shooting mode information set by the user. The image processing performed on the captured image includes so-called development processing, as well as processing such as color tone adjustment. In this embodiment, the image processing unit 105 also performs AI enhancement processing on the image data. The image processing unit 105 may also be equipped with a graphics processing unit (GPU) as the processor that performs the processing.

[0022] In the example shown in Figure 1, the imaging device 100 has an image processing unit 105 as a single piece of hardware, and the image processing unit 105 is described as performing image processing. However, the implementation of the present invention is not limited to this. Some image processing may be realized by the CPU 103 executing a corresponding processing program. Alternatively, image processing may be performed by an information processing device such as a server connected via a communication unit (not shown). In that case, image data may be transmitted to the information processing device via the communication unit, and the processed image data, which has undergone image processing there, may be transmitted to the imaging device 100.

[0023] Overview of the development parameter calculation process The following describes the extended image development process performed by the imaging device 100 of this embodiment, using the flowchart in Figure 2. The process corresponding to the flowchart is divided into an extended image saving process (Figure 2(a), S201-S203) for saving the AI-enhanced image and an extended image development process (Figure 2(b), S204-S205) for developing the saved image. The former extended image saving process can be realized, for example, by the CPU 103 reading the corresponding processing program stored in the secondary storage unit 107, expanding it in the temporary storage unit 104, and executing it. This extended image saving process is described as starting, for example, when an operation input related to the AI ​​enhancement process is made. The latter extended image development process can be executed by the CPU 103, similar to the extended image saving process, or it can be executed by an external application, for example, not shown. Both processes are realized by the CPU 103 executing a program loaded into memory, but in this embodiment, the CPU 103 controls the image processing unit 105 to perform the process.

[0024] In the extended image storage process of this embodiment, the image processing unit 105 performs AI extension on captured image data. For example, a trained model, which has been trained by machine learning and stored in the secondary storage unit 107, is loaded into the primary storage unit 104 and used. This trained model takes captured image data (RAW image data in this example) and extension parameters as input and outputs extended image data. Extension parameters may include, for example, the subject of the image to be added by extension, or the position where the extension region is superimposed or added (for example, a position within the image or a position outside the image). Alternatively, they may include the subject in the original image to be deleted from the original image. The trained model may be trained by an information processing device other than the imaging device 100. Training may be performed using known methods. For example, the original image data and extension parameters may be used as input data, and the output extended image data or its feature data may be used as ground truth data for training. Of course, the administrator may view the output image and provide feedback to the machine learning program. Training may also be performed by the imaging device 100, in which case the training process may be performed by the image processing unit 105.

[0025] The process shown in Figure 2(a) may be executed when the user selects image data and inputs instructions, including image enhancement parameters, on the operating unit 109 of the imaging device 100.

[0026] In S201, the image processing unit 105 performs AI augmentation under the control of the CPU 103. This augmentation process can be performed using augmentation by various generating AIs. AI augmentation is performed with target image data (target image data) and augmentation parameters (augmentation parameters) as input, and augmented image data (augmented image data) is output. Here, the target image data may be image data selected from captured image data stored on the recording medium 106. The augmentation parameters may be gestures or strings entered by the user from the operation unit 109 or the like. An example of a gesture will be explained with reference to Figure 7 in Embodiment 2, and may include operations such as pinch-to-zoom or area setting by specifying a frame. Alternatively, it may be a parameter selected by the user from among the set parameters. The augmentation parameters may be grouped into a set of parameters as an augmentation profile, for example, in which case the user may select the desired augmentation profile and give an augmentation instruction. In this embodiment, both the target image data and the augmented image data are assumed to be RAW data.

[0027] In S201, for example, the CPU 103 loads the target image data selected by the user from the recording medium 106 into the primary storage unit 104 and gives an instruction to the image processing unit 105 to perform AI extension of the extended parameters. The image processing unit 105 may perform the AI ​​extension process according to the instruction. The extended image generated by the image processing unit 105 is stored in the primary storage unit 104.

[0028] In image augmentation, for example, it is possible to create an image with a wider field of view by generating areas in the top, bottom, left, and right directions of the image that were not present in the original image, or to generate subjects in specific areas of the image that were not present in the original image. How the target image data is augmented is specified by augmentation parameters. However, the specific augmentation method corresponding to those parameters depends on the trained model. This makes it possible, for example, to obtain an augmented image 302 from the captured image 301 in Figure 3, with the field of view expanded in the upward and leftward directions.

[0029] In S202, the image processing unit 105, under the control of the CPU 103, acquires the extended region. Acquiring the extended region may mean acquiring information that identifies the extended region (referred to as extended region identification information). Note that the extended region identification information is also called region identification information because it distinguishes the extended region from the original image region before expansion. Here, it is possible to generate mask data corresponding to the extended region, such as the mask data 303 in Figure 3, where the original region is 1 and the expanded region is 0. That is, here, mask data corresponding to the extended region is generated as information that identifies the extended region. For example, the correlation between the image data of the target image 301 and the image data of the extended image 303 can be determined to determine the region in the extended image that corresponds to the target image, and mask data with the corresponding region set to 1 and the non-corresponding region set to 0 can be generated as extended region identification information. Alternatively, the extended region identification information may be stored as coordinate information indicating the extended region. For example, it may be identified by positional information such as the coordinates of the boundary (or contour) between the original image region and the extended region, and the positional information, or the vector connecting them, may be used as extended region identification information. The generated mask data and other extended region identification information are stored in the primary storage unit 104. In S202, as in S201, the CPU 103 may instruct the image processing unit 105 to generate extended region identification information, and the image processing unit 105 may generate the extended region identification information accordingly and store it in the primary storage unit 104.

[0030] In S203, CPU103 saves an image file. Here, the extended image 302 and the extended region identification information, such as the mask data 303 acquired in S202, are combined (i.e., associated) and saved as an image file 304.

[0031] In the extended image saving process, the CPU 103 may perform all steps without using the image processing unit 105.

[0032] From S204 shown in Figure 2(b), the process becomes extended image development. This extended image development process is described as being started, for example, when an input for developing an AI-enhanced image is received. In the extended image development process, the RAW data saved as extended image data is developed. In the RAW data development process, post-processing such as white balance, brightness adjustment, hue adjustment, and noise reduction is performed to generate developed image data (developed image data), which is saved in a different format from the RAW data, for example, as JPEG data. In this embodiment, the parameters for these post-processing steps are also determined. In this embodiment, the extended image development process is also described as being realized by the imaging device 100, particularly its CPU 103, executing a program loaded into memory such as the primary storage unit 104. This program includes processing that the CPU 103 controls and executes in the image processing unit 105. The process in Figure 2(b) may be started when the user selects the image data to be developed and specifies the development in the imaging device 100.

[0033] In S204, the CPU 103 reads the extended image and information related to the extended area. Here, as shown in Figure 4, the CPU reads the extended image data 402 from the selected image file 401 among the image files stored on the recording medium 106, and the extended area identification information 403 stored together with the extended image, and loads it into the temporary storage device 104.

[0034] In S205, the image processing unit 105 performs development parameter calculations under the control of the CPU 103. Development parameters may be parameters for correction processing of image data. Here, for example, according to the mask data which is the extended region identification information 403, the region 404 corresponding to the region where the value of the mask data is 1 is extracted from the extended image data 402. In other words, the region corresponding to the original image is extracted from the extended image data. Then, feature quantities are extracted only from that region, that is, from the original image region excluding the extended region, and development parameters are calculated from those feature quantities. Development parameters are parameters for post-processing performed in S206, and are determined for each type of post-processing. The calculation of development parameters may be performed, for example, by a predetermined procedure, and a procedure used in conventional digital cameras may be used. However, the target region is not the entire captured image, but is limited to the original image region identified by extended region identification information such as mask data.

[0035] For example, when performing white balance adjustment as a post-processing step, the color distribution on the image is obtained as a feature length from the area to be targeted for parameter determination, in this case the original image area corresponding to the mask data. Then, the color temperature of the light source is estimated from these features, and the coefficients for each color channel to correct it are determined as parameters. Similarly, when performing exposure correction, a histogram of luminance features is created from the original image area corresponding to the mask data, and parameters for correcting luminance are determined based on this histogram. Of course, these are just examples, and parameters for other post-processing steps may be determined, or they may be determined by other methods.

[0036] In S206, the image processing unit 105 performs post-processing by applying development parameters under the control of the CPU 103. Here, the development parameters calculated in S205 are applied to the entire extended image data 402, and post-processing is performed to generate the final image. The generated image data may be saved as is, or it may be compressed and saved as a JPEG file or the like.

[0037] In the extended image development process, the CPU 103 may perform all the steps shown in Figure 2(b) without using the image processing unit 105.

[0038] As explained above, the image processing apparatus of this embodiment makes it possible to reduce the influence of the expanded region on the original region. For example, if post-processing is performed on the entire image region including the expanded region (extended region), excessive adjustment (over-adjustment) may occur in the region corresponding to the original image (original image region), but this embodiment can prevent over-adjustment of the original image region. Conversely, if post-processing is performed on the entire image region including the expanded region, insufficient adjustment may occur in the original image region, but this embodiment can prevent insufficient adjustment of the original image region. In this way, even with post-processing of an expanded image, it is possible to perform post-processing appropriate to the original image region, thereby improving the quality of the developed image.

[0039] [Example 1] In the embodiments described above, a method for calculating development parameters from only the area represented by the mask data was explained. However, the present invention is not limited to this. For example, it is also possible to calculate the white balance color temperature from the original image area 404 represented by the mask data (for example, the color temperature x of the color temperature curve 405 in Figure 4) and the white balance color temperature (color temperature y of the color temperature curve 406) calculated from the entire extended image data. The calculation can be performed, for example, by the following formula.

[0040] Color temperature=x×0.8+y×0.2 In other words, while the weight (or contribution rate) of the color temperature of the original image region was set to 1 in the above embodiment, in this modified example, the color temperature is determined by combining a weight of 0.8 for the color temperature of the original image region and a weight of 0.2 for the color temperature of the entire extended image. That is, the feature information of the entire extended image data is determined by assigning region-specific weights to the feature information generated from each region of the extended image data and combining them.

[0041] By applying development parameters calculated by taking into account the characteristics of the original image region to the characteristics of the entire extended image in this way, it becomes possible to obtain a development result that takes into account the extended region itself, while mitigating the influence of the extended region on the original region.

[0042] [Differentiation 2] In the embodiments described above, a method was explained in which development parameters calculated from the original image region and the extended region, respectively, are combined and applied. However, the implementation of the present invention is not limited to this. For example, as shown in Figure 5, the original image region 504 and the extended region 505 are obtained based on the extended image data 502 and mask data 503 read from the image file 501. Here, the extended image data 502 has a dark face generated in the extended region due to underexposure. Therefore, if the histogram showing the brightness distribution in the extended image data 502 is obtained as is, a histogram distributed in the dark areas (left side of the graph) is obtained, as shown in histogram 506. Normally, if dark area correction is performed as is, a strong dark area correction is applied to the entire extended image in an attempt to brighten the dark areas, which may result in excessive correction including the original image region. Therefore, when the histogram 507 obtained from the original image region 504 is taken as h1(L) and the histogram 508 obtained from the extended region 505 is taken as h2(L), it is possible to calculate the histogram 509 as follows. Histogram 509 will be represented as h3(L).

[0043] h3(L) = h1(L) × 1.0 + h2(L) × 0.3 Based on the histogram calculated in this way, the amount of correction for dark areas is calculated. In other words, in this modified example as well, the feature information generated from each region of the extended image data is weighted for each region and combined to determine the feature information of the entire extended image data. By doing so, it is possible to obtain a development result that takes into account the extended region itself, which is ultimately saved, while reducing the influence of the extended region on the original image region. In other words, by performing post-processing with development parameters calculated by adding the characteristics of the extended region to the image characteristics of the original image region, it is possible to obtain a development result that takes the extended region into account while reducing the influence of the extended region on the original image region.

[0044] [Difference 3] In the embodiments described above, a process for separating and handling the original image region and the extended region using mask data and coordinate information indicating the extended region was explained. However, the implementation of the present invention is not limited to this. For example, it is possible to include multiple mask values ​​in the mask data.

[0045] In generative AI, in addition to the process of expanding the field of view of image data as described in Embodiment 1, it is also possible to add a subject to a specific area within the image. Furthermore, it is possible to remove a subject by compositing a background image. For example, in Figure 6, it is possible not only to expand the field of view from the captured image 601 to the expanded image 602, but also to add a tree to area 603. In such cases, as mentioned above, the generated area does not necessarily have the same characteristics as the original area, but since the subject was added intentionally by the user, it is desirable that the development parameters take this area into consideration to some extent. Note that generating an image with other objects such as subjects added to the original image, or removing a subject, is also called image expansion, and even if the field of view of the original image does not change, the area containing the added object is called the expanded area. Therefore, image expansion can also be called image modification or compositing, and the expanded area can be called the modified area or the composite area.

[0046] Therefore, in the above case, depending on how each region is expanded, mask data is generated such that the coefficient of the unexpanded original image region 604 is 1, the coefficient of the automatically generated expanded region is 0.3, and the coefficient of the region generated by the user specifying the subject is 0.7. In this case, for example, when calculating a histogram, it is possible to calculate the histogram h(L) based on the mask value m(x,y) and brightness L(x,y) at coordinates (x,y) as follows.

[0047] h(L) = Σ x,y {m(x,y) × δ(LL(x,y)} However, δ is the Dirac delta function, which returns 1 when the argument is zero and 0 otherwise. The histogram h(L) shows the number of pixels with luminance L weighted by the mask value m(x, y) for the entire augmented image. Here, the weighting is achieved by multiplying by the mask value corresponding to the position of the pixels with luminance L. In other words, in this modified example as well, the feature information for the entire augmented image data is determined by combining the feature information generated from each region of the augmented image data with region-specific weights. However, in this modified example, the weights are not fixed values ​​but are given as mask data. It is possible to decide in advance, as in the example above, which regions to assign what weights to, as indicated by the mask data.

[0048] As mentioned above, instead of using mask values ​​for weights, it is also possible to store them as label information or as coefficients corresponding to the coordinate information representing each region. In such cases, it is possible to set the weights (or coefficients) corresponding to the labels or regions again when calculating the development parameters. Furthermore, by configuring the system so that this setting can be specified by the user, weighting can be performed more flexibly.

[0049] In this way, by performing post-processing with development parameters calculated by adding the characteristics of the extended region to the characteristics of the original image region, it is possible to reduce the influence of the extended region on the original image region while still obtaining a development result that takes the extended region into consideration. Furthermore, in this modified example, by setting coefficients as mask values, the weights for each region can be set.

[0050] [Embodiment 2] The embodiments and modifications described above describe a method in which an enhanced image, obtained by applying AI enhancements to a captured image, is saved to an image file. This embodiment describes an alternative method in which the enhancement process is performed simultaneously with the capture.

[0051] An example of a scene in which augmentation processing is performed simultaneously during shooting will be explained using Figure 7. Figure 7 shows a screen displaying an image on a display unit 108 equipped with a touch panel, and an example of its operation. Screen 701 shows an example of live view display when waiting to shoot. If, for example, a tree on the right side of the image is in the way and you want to remove it, as shown in Screen 702, the user can slide the guide at the edge of the screen to limit the shooting area, and by compositing an augmented image outside that area, it is possible to generate image 703 with the tree removed. Also, if you want to shoot at a wider angle than the current live view display, as shown in Screen 704, the user can pinch in on the screen to reduce the shooting area, and then expand the insufficient area to generate image 705. Screen operations are acquired by the CPU 103 as trajectory data of the operations on the screen, and augmentation parameters corresponding to the operation are generated and stored in, for example, the primary storage unit 104, such as erasing a subject in a specified area or expanding the area that became empty due to reduction. In this way, in this embodiment, augmentation parameters for augmenting the target image data are set before the target image data is shot.

[0052] The extended image development process performed by the imaging device 100 of this embodiment will be described below using the flowchart in Figure 8. The process corresponding to the flowchart can be realized by the CPU 103 reading the corresponding processing program stored in, for example, the secondary storage unit 107, expanding it in the temporary storage unit 104, and executing it. The extended image development process in Figure 8 will be described as starting immediately after shooting in response to a shooting instruction by the shutter button, etc., when, for example, the user inputs an operation to perform extended processing shooting (for example, the operation exemplified in screens 702 and 704 in Figure 7). At this start, the extended parameters input in the screen operation before shooting are stored in, for example, the primary storage unit 104, and the image data captured in response to the subsequent shooting operation is also stored in the primary storage unit 104. The image data stored in the primary storage unit 104 becomes the target image data for extended processing, but in the following description, the target image data is also called the captured image. Note that S801 and S803 may be started in parallel or asynchronously. However, using the completion points of S802 and S805 as synchronization points, the system waits for the completion of whichever process is behind before executing S806 and subsequent steps.

[0053] In S801, the image processing unit 105, under the control of the CPU 103, acquires image characteristics from the captured image. In this process, just as in normal shooting without generation and expansion, it is always possible to acquire characteristic information for development parameter calculation from image data equivalent to screen 701 in Figure 7 (i.e., the captured unprocessed RAW data), regardless of the user's generation and expansion specifications. The acquired characteristic information includes, for example, a histogram showing the distribution of white balance and brightness, as shown in Embodiment 1.

[0054] In S802, the image processing unit 105, under the control of the CPU 103, calculates and determines the development parameters. In this process, as in S801, it is possible to calculate the development parameters in the same way as during normal shooting. For example, it can be done in the same manner as the calculation of development parameters for the original image area corresponding to the mask data in S205.

[0055] In S803, the image processing unit 105 performs AI augmentation processing under the control of the CPU 103. This makes it possible to obtain augmented images such as image 703 and image 705. Here, the AI ​​augmentation processing can be performed in the same manner as in S201 in Figure 2, but the target image data is the captured image, and the augmentation parameters can be those stored in the primary storage unit 104.

[0056] In S804, the image processing device 105 acquires the extended region under the control of the CPU 103. Here, as mentioned above, it is possible to generate mask data to distinguish the extended region and acquire coordinate information indicating the extended region. This process may be carried out in the same manner as in S202.

[0057] In step S805, the image processing unit 105, under the control of the CPU 103, acquires the characteristics of the extended region (extended region characteristics). The extended region characteristics may be characteristics that apply only to the region extended by the extension process, or they may be characteristics that apply to the entire extended image data, including the original image region and the extended region. This may be determined according to the weight (i.e., multiplier) assigned to the extended region characteristic information, as explained in modifications 1 to 3 of Embodiment 1. For example, if the characteristic information applies to the entire extended image data as explained in Figure 4, then characteristic information that applies to the entire extended image data should be acquired. Alternatively, if the original image region and the extended region are weighted respectively, as explained in Figures 5 and 6, then characteristic information that applies only to the extended region should be acquired.

[0058] Next, in S806, the image processing unit 105 performs a recalculation determination under the control of the CPU 103. At this time, it synchronizes with the completion of S802, which was running in parallel. That is, if S802 has finished, S806 is executed; otherwise, it waits for S802 to finish before executing S806.

[0059] In determining whether recalculation is necessary, for example, by comparing the histogram distributions of the original image region and the extended region, it is possible to determine if recalculation is unnecessary if the similarity is high. Similarity can be determined by various criteria, such as determining similarity if the mean, variance, or peak distribution is within a predetermined difference. Alternatively, if there is no subject in the extended region, or if the extended region is narrow to begin with, it may be determined that recalculation is unnecessary.

[0060] If S806 determines that recalculation is necessary, in S807, the image processing unit 105 updates the development parameters under the control of the CPU 103. In this process, the same processing as in S205 in Figure 2 can be performed. On the other hand, if S806 determines that recalculation is not necessary, the process proceeds to S808. This is because, at the start of S806, the development parameters for the original image area have already been calculated by S802, and post-processing can be performed using these parameters. This makes it possible to shorten the processing time by skipping the recalculation process for development parameters.

[0061] In S808, the image processing unit 105 performs post-processing by applying development parameters under the control of the CPU 103. Here, the calculated development parameters are applied to the extended image 703 and the entire 705 to generate the final image data. The generated image data corresponds to the developed image and is saved to the recording medium 106. When saving, it may be saved as compressed image data such as JPEG.

[0062] With the configuration and processing procedure described above, this embodiment allows for the parallel execution of AI augmentation processing, acquisition of characteristic information of the original image region, and determination of development parameters. This reduces processing time and enables rapid post-processing (i.e., correction processing or RAW development processing) when it is not necessary to determine development parameters corresponding to the augmented region. Furthermore, if post-processing corresponding to the augmented region is required, performing it improves the quality of the AI-augmented image after development.

[0063] <Other variations> It is also possible to always update the development parameters based on the expanded image using the procedure shown in Figure 8. In this case, steps S801, S802, and S806 are unnecessary, and only steps S803-S805 and S807-S808 need to be executed.

[0064] Although all the embodiments and their modifications described above have explained image enhancement using AI, they can also be applied to images enhanced using image enhancement methods that do not use AI. For example, all embodiments and their modifications can be applied to enhanced image data created by compositing a photographed image onto a pre-prepared background, or to enhanced image data created by combining multiple image data by cutting and pasting. Furthermore, all embodiments and their modifications can also be applied to enhanced data created by manually filling in the color of a specified area with a specified part.

[0065] Furthermore, in the above embodiments and modifications, parameters were generated by weighting the feature information (also called feature quantities) of the regions identified by the region identification information. In contrast, in the case where the original image data before expansion and the expanded image data after expansion are retained, as in Embodiment 2, the original image regions and expanded regions can be identified from those image data, so it is not necessary to generate region identification information. By identifying each region without generating region identification information, it is possible to determine the parameters and perform development processing as in all the embodiments and modifications described above. This eliminates the need for the process of generating region identification information, and also eliminates the need to save the generated region identification information, thereby reducing the processing burden and the amount of resources required for storage.

[0066] Furthermore, the above embodiments and modifications show examples where image augmentation and development of augmented image data are performed using an imaging device such as a digital camera. In contrast, image data captured by an imaging device such as a digital camera may be stored in an information processing device such as a personal computer, and then augmented and developed using that information processing device. In that case, a service using a generation AI provided via the internet may be used as the augmentation process.

[0067] [Other examples] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0068] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0069] ●Summary of Embodiments The above embodiments can be summarized as follows: (Item 1) A determination means for determining the parameters of correction processing for extended image data generated by an image extension function, based on feature information for each region including the original image region before extension and the extended region included in the extended image data generated by the image extension function, The system includes a correction means for correcting the extended image data using the determined parameters. An image processing apparatus characterized by the following: (Item 2) The image processing apparatus described in item 1, The system further comprises a generation means for generating region identification information that identifies the original image region before expansion and the expanded region included in the expanded image data generated by the aforementioned image expansion function, The determination means determines the parameters for the correction process on the extended image data based on the characteristic information of the region identified by the region identification information. An image processing apparatus characterized by the following: (Item 3) The image processing apparatus described in item 2, The system further comprises storage means for storing the extended image data and the region identification information. An image processing apparatus characterized by the following: (Item 4) An image processing apparatus as described in item 2 or 3, The region identification information is mask data corresponding to the original image region and the extended region, respectively. An image processing apparatus characterized by the following: (Item 5) An image processing apparatus described in any one of items 2 to 4, The region identification information is positional information of the boundary between the original image region and the extended region. An image processing apparatus characterized by the following: (Item 6) An image processing apparatus described in any one of items 1 to 5, The determination means determines the parameters based on feature quantities generated from the original image region, excluding the extended region. An image processing apparatus characterized by the following: (Item 7) An image processing apparatus described in any one of items 1 to 6, The determination means determines the parameters based on feature information obtained by combining feature information generated from the extended region and the original image region, and feature information generated from the original image region excluding the extended region. An image processing apparatus characterized by the following: (Item 8) An image processing apparatus described in any one of items 1 to 7, The determination means combines the feature information generated from each of the regions identified by the region identification information by assigning a region-specific weight to each region. An image processing apparatus characterized by the following: (Item 9) An image processing apparatus described in any one of items 1 to 8, The weight for each region is determined according to the method of region expansion by the image expansion means. An image processing apparatus characterized by the following: (Item 10) The image processing apparatus described in item 9, The methods of expanding each of the aforementioned regions include automatic expansion without user specification of a subject, expansion with user specification of a subject, and no expansion at all. An image processing apparatus characterized by the following: (Item 11) An image processing apparatus described in any one of items 1 to 10, The system further comprises image extension means for providing the aforementioned image extension function. An image processing apparatus characterized by the following: (Item 12) The image processing apparatus described in item 11, It further comprises an imaging means for capturing images. An image processing apparatus characterized by the following: (Item 13) A program for causing a computer to function as an image processing device as described in any one of items 1 through 12. (Item 14) A control method for an image processing apparatus comprising a determination means and a correction means, The determination means includes a determination step of determining the parameters for correction processing on extended image data, which is an extended image data generated by the image extension function, based on feature information for each region, which includes the original image region before extension and the generated extended region, included in the extended image data that has been extended by the image extension function. The correction means includes a correction step of correcting the extended image data using the determined parameters. A control method for an image processing apparatus, characterized by the features described above.

[0070] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0071] 100: Imaging device, 103: CPU, 104: Temporary storage device, 105: Image processing device, 107: Secondary storage device

Claims

1. A determination means for determining the parameters of correction processing for extended image data generated by an image extension function, based on feature information for each region including the original image region before extension and the extended region included in the extended image data generated by the image extension function, The system includes a correction means for correcting the extended image data using the determined parameters. An image processing apparatus characterized by the following:

2. An image processing apparatus according to claim 1, The system further comprises a generation means for generating region identification information that identifies the original image region before expansion and the expanded region included in the expanded image data generated by the aforementioned image expansion function, The determination means determines the parameters for the correction process on the extended image data based on the characteristic information of the original image region and the extended region, which are identified by the region identification information. An image processing apparatus characterized by the following:

3. An image processing apparatus according to claim 2, The system further comprises storage means for storing the extended image data and the region identification information. An image processing apparatus characterized by the following:

4. An image processing apparatus according to claim 2, The region identification information is mask data corresponding to the original image region and the extended region, respectively. An image processing apparatus characterized by the following:

5. An image processing apparatus according to claim 2, The region identification information is positional information of the boundary between the original image region and the extended region. An image processing apparatus characterized by the following:

6. An image processing apparatus according to claim 1, The determination means determines the parameters based on feature quantities generated from the original image region, excluding the extended region. An image processing apparatus characterized by the following:

7. An image processing apparatus according to claim 1, The determination means determines the parameters based on feature information obtained by combining feature information generated from the extended region and the original image region, and feature information generated from the original image region excluding the extended region. An image processing apparatus characterized by the following:

8. An image processing apparatus according to claim 2, The determination means combines the feature information generated from each of the regions identified by the region identification information by assigning a region-specific weight to each region. An image processing apparatus characterized by the following:

9. An image processing apparatus according to claim 1, The weight for each region is determined according to how each region is expanded by the image extension function. An image processing apparatus characterized by the following:

10. An image processing apparatus according to claim 9, The methods of expanding each of the aforementioned regions include automatic expansion without user specification of a subject, expansion with user specification of a subject, and no expansion at all. An image processing apparatus characterized by the following:

11. An image processing apparatus according to any one of claims 1 to 10, The system further comprises image extension means for providing the aforementioned image extension function. An image processing apparatus characterized by the following:

12. An image processing apparatus according to claim 11, It further comprises an imaging means for capturing images. An image processing apparatus characterized by the following:

13. A program for causing a computer to function as an image processing device according to any one of claims 1 to 10.

14. A control method for an image processing apparatus comprising a determination means and a correction means, The determination means includes a determination step of determining the parameters for correction processing on extended image data, which is an extended image data generated by the image extension function, based on feature information for each region including the original image region before extension and the extended region included in the extended image data generated by the image extension function, The correction means includes a correction step of correcting the extended image data using the determined parameters. A control method for an image processing apparatus, characterized by the features described above.

Images