Image processing apparatus, control method, computer program product, and storage medium

Abstract

The invention provides an image processing apparatus, a control method, a computer program product, and a storage medium. The image processing apparatus determines a parameter for correction processing of expanded image data generated by an image expansion function based on feature information of each of areas including an original image area before expansion and an expansion area included in the expanded image data generated by the image expansion function, wherein the image data in the expanded image data is expanded; and correcting the extended image data using the determined parameters.

Description

Technical Field

[0001] The technology disclosed herein relates to image processing apparatus, control methods, computer program products and media, and particularly to image processing of extended images. Background Technology

[0002] In recent years, with the development of generative AI technology, image data can be expanded. This enables the generation of regions not present in the original image, creating images with a wider field of view, and allowing the generation of subjects not present in the original image within specific regions of the image. Consider applying this technology to images requiring post-processing, such as RAW images. In this case, the expanded region can have different characteristics than the original region, and as a result, regions with different characteristics can be included in an expanded image. In such images, due to the differences in characteristics between the original and expanded regions, there is a possibility that post-processing could visualize the tonal and brightness differences between these regions, resulting in an unnatural image.

[0003] Known techniques related to white balance, as one aspect of image post-processing, include those described, for example, in Japanese Patent Application Publication No. 2012-039256 and Japanese Patent Application Publication No. 2004-064676. Japanese Patent Application Publication No. 2012-039256 discloses a configuration where, when an image is cropped at a set aspect ratio during image recording, an evaluation value for white balance control is calculated by changing the usage ratio between the entire image to be cropped and a common area of ​​the image that is used regardless of the aspect ratio. In Japanese Patent Application Publication No. 2012-039256, when the main subject is included in a non-common area that is used or not used depending on the aspect ratio setting, the usage ratio of the common area is increased to calculate the color level for white balance control.

[0004] Japanese Patent Application Publication No. 2004-064676 discloses a configuration where, when a specific area is cut out and stored as an electronic zoom area during image recording, colors for white balance are also extracted from outside the area to be cut out. In Japanese Patent Application Publication No. 2004-064676, when a light source different from the light source illuminating the subject exists outside the electronic zoom area, white balance control is performed by reducing the weight of colors extracted from outside the area.

[0005] Japanese Patent Application Publication No. 2012-039256 suppresses changes in brightness and color balance of the main subject, such as a person's face, caused by variations in aspect ratio settings, and controls white balance differences between images with different aspect ratios already set. Japanese Patent Application Publication No. 2004-064676 assumes a specific viewpoint is cut out and stored, and its aim is to achieve appropriate color balance and brightness in the cut-out image by effectively utilizing information from the external area to be cut out. Therefore, for example, when a light source different from the light source illuminating the subject exists outside the electronic zoom area, the weight of colors extracted from the outside is reduced; however, if the light source exists inside the electronic zoom area, this weight is not changed.

[0006] Even when this known technology is applied to extended image data generated by AI, the effects described in the various patent documents only apply to the extended image data. Therefore, it is difficult to improve image quality, such as unnaturalness in the post-processed image, caused by differences in characteristics between the original and extended regions. Summary of the Invention

[0007] The technology disclosed herein enables image processing that reduces the impact of the extended region on the original region while also taking into account the extended region.

[0008] According to one aspect of this disclosure, an image processing apparatus is provided, comprising: a determining component for determining parameters for correction processing of the expanded image data generated by the image expansion function based on feature information of each region in the region of the original image region before expansion and the expanded region included in the expanded image data generated by the image expansion function, wherein image data is expanded in the expanded image data; and a correction component for correcting the expanded image data using the determined parameters.

[0009] According to another aspect of this disclosure, a computer-readable storage medium is provided storing a program that, when loaded and executed on a computer, causes the computer to perform processing, wherein the processing includes: determining parameters for correction processing of the expanded image data generated by the image expansion function based on feature information of various regions in the original image region before expansion and the region of the expanded region included in the expanded image data generated by the image expansion function, wherein image data is expanded in the expanded image data; and correcting the expanded image data using the determined parameters.

[0010] According to another aspect of this disclosure, a control method for an image processing apparatus is provided, the control method comprising: determining parameters for correction processing of the expanded image data generated by the image expansion function based on feature information of each region in the region of the original image region before expansion and the region of the expanded region included in the expanded image data generated by the image expansion function, wherein image data is expanded in the expanded image data; and correcting the expanded image data using the determined parameters.

[0011] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when loaded and executed on a computer, causes the computer to perform processing, wherein the processing includes: determining parameters for correction processing of the expanded image data generated by the image expansion function based on feature information of various regions in the original image region before expansion and the region of the expanded region included in the expanded image data generated by the image expansion function, wherein image data is expanded in the expanded image data; and correcting the expanded image data using the determined parameters.

[0012] Based on the above configuration, image processing can be performed that reduces the impact of the extended region on the original region while also taking the extended region into account.

[0013] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is given by way of example. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating the hardware configuration of a camera device 100 according to embodiments and variations.

[0015] Figure 2A and Figure 2B This is a flowchart illustrating the extended image storage processing and extended image display processing to be performed in the camera device 100 according to the embodiments and variations thereof.

[0016] Figure 3 This is a diagram used to describe the extended image storage processing to be performed in the camera device 100 according to embodiments and variations thereof.

[0017] Figure 4 This is a diagram used to describe the image parameter calculation process to be performed in the camera device 100 according to a modified example 1 of the embodiment.

[0018] Figure 5 This is a diagram used to describe the histogram calculation process to be performed in the camera device 100 according to a modified example 2 of the embodiment.

[0019] Figure 6 This is a diagram used to describe the extended region acquisition process to be performed in the camera device 100 according to a modified example 3 of the embodiment.

[0020] Figure 7 This is a diagram used to describe an example of user operations according to the second embodiment.

[0021] Figure 8 This is a flowchart illustrating the extended image display processing to be performed in the camera device 100 according to the second embodiment. Detailed Implementation

[0022] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are required, and several such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions thereof are omitted.

[0023] First Embodiment The embodiments described below are examples of camera devices as image processing apparatuses, which have the function of performing image processing that reduces the influence of the expanded region on the original region in an expanded image while also taking the expanded region into account. However, the techniques disclosed herein can be applied to any apparatus capable of generating images that have undergone image processing that reduces the influence of the expanded region on the original region in an expanded image while also taking the expanded region into account.

[0024] Hardware configuration of camera equipment In the following text, reference will be made to Figure 1 The block diagram illustrates the hardware configuration of the camera device 100 according to this embodiment. The camera device 100 may include, for example, devices designed for video recording purposes, such as digital cameras and digital video cameras, or electronic devices with video recording capabilities, such as mobile phones equipped with cameras and computers equipped with cameras.

[0025] Optical system 101 is an image-forming optical system including a lens group, a shutter, and an aperture. The lens group may include a correction lens and a focusing lens for correcting camera shake, etc. Optical system 101 forms an image of the subject's light onto the imaging surface of imaging element 102 based on control signals received from CPU 103, described later. Imaging element 102 is, for example, an imaging sensor such as a charge-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor. Imaging element 102 photoelectrically converts the optical image formed on the imaging surface by optical system 101 into an image signal. This image signal is digitized into digital image data (referred to as image data) and stored or processed through image processing.

[0026] CPU 103 is a control device that controls the operation of various blocks included in the imaging device 100. CPU 103 controls the operation of each block by reading the operation program of each block stored in the secondary storage unit 107 and expanding it to the main storage unit 104 for execution. In the operation control of each block, CPU 103 appropriately transmits control signals corresponding to the respective operation to each block. CPU 103 may be referred to as a processor.

[0027] The secondary storage unit 107 is, for example, a non-volatile storage device such as an electrically erasable programmable read-only memory (EEPROM). In addition to the operating programs for each block included in the imaging device 100 and the firmware of the imaging device 100, the secondary storage unit 107 also stores various types of setting information required for the operation of each block. The primary storage unit 104 is, for example, a volatile storage device such as random access memory (RAM). The primary storage unit 104 serves not only as an expansion area for the operating programs of each block but also as a working memory for storing temporary data output from the operation of each block. The information stored in the primary storage unit 104 can be used by the image processing unit 105 or recorded in the recording medium 106.

[0028] Recording medium 106 is a recording device, such as a semiconductor memory card, detachably configured with the camera device 100. When a photograph is taken in the camera device 100, recording medium 106 is used to record image data stored in the main storage unit 104 by the photograph. The data recorded on recording medium 106 can be used on an external device, such as a personal computer (PC), by attaching recording medium 106 to that external device.

[0029] Display unit 108 is, for example, a display device such as an LCD. Display unit 108 is used to display viewfinder images during shooting, display captured images, and display graphical user interface (GUI) images for interactive operation. During shooting by camera device 100, live view display can be performed on display unit 108. Image data selected from captured image data (image files) stored in recording medium 106 can also be displayed on display unit 108.

[0030] The operation unit 109 is, for example, a user interface such as a button, lever, or touch panel. When operation input is made to various operation components, the operation unit 109 transmits control signals corresponding to the operation input to the CPU 103. In addition, the operation unit 109 may also include input devices using voice or gaze. The operation unit 109 also includes a touch panel integrated with the display unit 108. Users can perform operations using the touch panel on the display of the display unit 108.

[0031] Image processing unit 105 is a device for performing various types of image processing on images. In this embodiment, image processing unit 105 is configured to apply multiple types of image processing to images. For example, the image processing unit 105 may determine which image processing method (pattern) to perform for each shooting mode. In this regard, the image processing method to be applied by image processing unit 105 to the captured image is controlled based on information about the shooting mode set by the user. Image processing to be performed on the captured image includes processing such as tone adjustment and so-called image processing. In this embodiment, image processing unit 105 performs AI extended processing on image data. Image processing unit 105 may include a graphics processing unit (GPU) as a processor to perform the processing.

[0032] Note that in Figure 1 In the example, the description is given under the assumption that the camera device 100 includes an image processing unit 105 as hardware and that the image processing unit 105 performs image processing; however, the implementation of this disclosure is not limited thereto. A portion of the image processing can be implemented by executing a corresponding processing program via the CPU 103. Alternatively, the image processing can be performed by an information processing device, such as a server connected via a communication unit (not shown). In this case, image data can be transmitted to the information processing device via the communication unit, and the processed image data can be transmitted to the camera device 100.

[0033] Overview of Imaging Parameter Calculation and Processing The following will refer to Figure 2A and Figure 2B The flowchart describes the specific processing of the extended image display processing performed by the camera device 100 in this embodiment. The processing corresponding to the flowchart is divided into extended image storage processing for storing AI extended images. Figure 2A (S201 to S203) and extended image display processing for displaying the stored image ( Figure 2B The extended image storage processing (S204 to S205) can be described as follows. The former, extended image storage processing, can be implemented, for example, by the CPU 103 reading the corresponding processing program stored in the secondary storage unit 107 and expanding it to the main storage unit 104 for execution. This extended image storage processing will be described as starting, for example, when an operation input related to AI extended processing is performed. The latter, extended image display processing, can be executed by the CPU 103 in a manner similar to extended image storage processing, or, for example, in an external application (not shown). Both processes are implemented by the CPU 103 executing programs loaded in memory; however, this embodiment includes processing controlled by the CPU 103 to be performed by the image processing unit 105.

[0034] In the extended image storage processing of this embodiment, since the image processing unit 105 performs AI extension of the captured image data, the learned model, which is stored in the secondary storage unit 107 and learned through machine learning, is loaded into the main storage unit 104 for use. This learned model takes the captured image data (RAW image data in this example) and parameters for extension as input, and outputs extended image data. The parameters for extension may include, for example, the subject of the image added through extension and the location where the extended region is superimposed or added (e.g., a location inside or outside the image). Alternatively, it may include subjects to be removed from the original image. The learned model can be a model learned by an information processing device other than the camera device 100. Known methods can be used for learning, and learning can be performed using, for example, the original image data and the parameters for extension as input data and the extended image data to be output or its feature data as correct answer data. Of course, the administrator can view the image to be output and provide feedback to the machine learning program. Note that learning can be performed by the camera device 100, but in this case, the learning processing can be performed by the image processing unit 105.

[0035] Figure 2A The processing can be performed in response to the user selecting image data and inputting instructions including parameters for image expansion in the operation unit 109 of the camera device 100.

[0036] In S201, the image processing unit 105 performs AI extension under the control of the CPU 103. The extension processing can be performed using various types of generative AI extensions. AI extension is performed with object image data and parameters (extension parameters) used for extension as input, and extended image data is output. The object image data mentioned here can be image data selected from captured image data stored in the recording medium 106. The extension parameters can be gestures or strings input by the user from the operation unit 109, etc. An example of a gesture will also be referred to in the second embodiment. Figure 7 The description can include operations such as reducing size by pinching or setting a region by assigning a box. Alternatively, it can be parameters selected by the user from set parameters. For example, expansion parameters can be collected into a set of parameters as an expansion profile, and in this case, the user can select the desired expansion profile to indicate expansion. Note that this embodiment assumes that both the object image data and the expanded image data are RAW data.

[0037] In S201, for example, the CPU 103 loads the image data of the object selected by the user from the recording medium 106 into the main storage unit 104, and gives an instruction to the image processing unit 105 to extend the AI ​​parameter. The image processing unit 105 can perform AI extension processing in response to the instruction. The extended image generated by the image processing unit 105 is stored in the main storage unit 104.

[0038] In the expansion, for example, regions not present in the original image can be generated in the vertical and horizontal directions, creating an image with a wider field of view, and subjects not present in the original image can be generated in specific regions of the image. How the object image data is expanded is assigned by the parameters used for expansion. However, the specific expansion method corresponding to the parameters depends on the learned model. This can be, for example, from... Figure 3 The captured image 301 yields an expanded image 302 with the field of view extending upwards and to the left.

[0039] In S202, the image processing unit 105 acquires the extended region under the control of the CPU 103. Acquiring the extended region can involve acquiring information used to specify the extended region (referred to as extended region specification information). Note that the extended region specification information is also called region specification information because the extended region is distinguished from the original image region before expansion. Here, for example, such as... Figure 3 Similar to mask data 303, mask data corresponding to an extended region where the original region is 1 and the extended region is 0 can be generated. That is, here, mask data corresponding to the extended region is generated as information specifying the extended region. For example, the correlation between the image data of the object image 301 and the image data of the extended image 302 can be obtained to determine the region corresponding to the object image in the extended image, and mask data where the corresponding region is 1 and the non-corresponding region is 0 can be generated as extended region specifying information. Alternatively, the extended region specifying information can be maintained as coordinate information indicating the extended region. For example, the extended region can be specified by positional information such as the coordinates of the boundary (or contour) between the original image region and the extended region, and the positional information or the vector connecting the positional information can be used as extended region specifying information. Extended region specifying information, such as the already generated mask data, is stored in the main storage unit 104. Similar to S201, in S202, the CPU 103 can give instructions to the image processing unit 105 for generating extended region specification information, and the image processing unit 105 can respond to this by generating extended region specification information and storing it in the main storage unit 104.

[0040] In S203, CPU 103 stores the image file. Here, the extended image 302 and the extended region specification information (such as the mask data 303 obtained in S202) are stored together (i.e., in association with each other) as the image file 304.

[0041] Note that in extended image storage processing, CPU 103 can perform the entire process without using image processing unit 105.

[0042] from Figure 2B Starting with S204, extended image display processing is performed. This extended image display processing will be described as starting, for example, when inputting an AI extended image display processing operation. In extended image display processing, RAW data stored as extended image data is processed. In the RAW data display processing, post-processing such as white balance, brightness adjustment, hue adjustment, and noise reduction are performed to generate displayed image data (displayed image data), and this displayed image data is stored in a different format than the RAW data, for example, as JPEG data. Note that in this embodiment, the determination of those post-processing parameters is also performed. In this embodiment, extended image display processing will also be described as being implemented by the camera device 100 (specifically its CPU 103) executing a program loaded into a memory such as the main storage unit 104. Note that this program includes processing controlled by the CPU 103 to be executed by the image processing unit 105. Figure 2B The processing can begin when the user selects the image data of the object to be displayed in the camera device 100 and assigns it to be displayed.

[0043] In S204, CPU 103 reads the extended image and information related to the extended region. Here, as... Figure 4 As in the example, extended image data 402 and extended region specification information 403 stored together with the extended image are read from image file 401, and the extended image data 402 and extended region specification information 403 are loaded into main storage unit 104, wherein image file 401 is selected from image files stored in recording medium 106.

[0044] In S205, the image processing unit 105 performs image processing parameter calculations under the control of the CPU 103. The image processing parameters can be parameters used for correction processing of the image data. Here, for example, based on mask data as extended region specification information 403, a region 404 corresponding to a region with a value of 1 in the mask data is extracted from the extended image data 402. That is, a region corresponding to the original image is extracted from the extended image data. Then, a feature value is extracted only from this region (i.e., from the original image region excluding the extended region), and the image processing parameters are calculated based on this feature value. The image processing parameters are parameters for post-processing performed in S206, and are obtained for various types of post-processing. The calculation of the image processing parameters can be performed, for example, by a predetermined process, or using a process performed by a known digital camera. However, the target region is not the entire captured image, but is limited to the original image region specified by extended region specification information such as mask data.

[0045] For example, when performing white balance adjustment as part of post-processing, the color distribution on the image is obtained as a feature quantity, using the region (here, the original image region corresponding to the mask data) as the object for determining parameters. Then, the color temperature of the light source is estimated based on the feature quantity, and the coefficients of each color channel used to correct that color temperature are determined as parameters. When performing exposure correction, a histogram of brightness is created using the original image region corresponding to the mask data as the object, and the parameters used to correct the brightness are determined based on this histogram. Of course, these are examples, and parameters for other post-processing can be determined, or they can be determined using other methods.

[0046] In S206, the image processing unit 105, under the control of the CPU 103, performs post-processing by applying display parameters. Here, the display parameters calculated in S205 are applied to the entire extended image data 402, and post-processing is performed to generate the final image. The generated image data can be stored as is, or it can be compressed and stored as a JPEG file, etc.

[0047] Note that in extended image display processing, CPU 103 can perform the operation without using image processing unit 105. Figure 2B The entire process.

[0048] As described above, the image processing apparatus according to this embodiment can reduce the impact of the extended region on the original region. For example, when post-processing is performed on the entire image region including the region extended by the extended processing (extended region), over-adjustment may occur in the region corresponding to the original image (original image region), but this embodiment can prevent over-adjustment of the original image region. Conversely, when post-processing is performed on the entire image region including the extended region, adjustment in the original image region may become insufficient, but this embodiment can prevent insufficient adjustment relative to the original image region. This also allows for post-processing suitable for the original image region in the post-processing of the extended image, and can improve the quality of the developed image.

[0049] Variation Example 1 In the above embodiments, aspects of calculating imaging parameters based solely on the region represented by the mask data have been described. However, implementations of this disclosure are not limited to this. For example, the color temperature of the white balance (e.g., calculated from the original image region 404 represented by the mask data) can also be used. Figure 4 The imaging parameters are calculated using the color temperature (x) of color temperature curve 405 in the image and the white balance color temperature (y) calculated from the entire extended image data (color temperature curve 406). This calculation can be performed, for example, by the following expression. Color temperature = x × 0.8 + y × 0.2

[0050] That is, although the weight (or contribution rate) of the color temperature of the original image region is set to 1 in the above embodiment, in this variation, the color temperature is determined by synthesizing the value by setting the weight of the color temperature of the original image region to 0.8 and the weight of the color temperature of the entire extended image to 0.2. In other words, the feature information of the entire extended image data is determined by synthesizing the feature information generated from each region of the extended image data with the weight of each region.

[0051] In this way, by applying the imaging parameters calculated considering the characteristics of the original image region to the characteristics of the entire extended image, it is possible to obtain an imaging result that reduces the impact of the extended region on the original region while also taking into account the extended region itself to be stored.

[0052] Variation Example 2 In the above embodiments, aspects of synthesizing and applying imaging parameters calculated from various regions of the original image region and the entire region including the extended region have been described. However, implementations of this disclosure are not limited thereto. For example, as Figure 5As in the example, the original image region 504 and the extended region 505 are obtained based on the extended image data 502 and mask data 503 read from image file 501. In the extended image data 502, dark faces are generated in the extended region due to underexposure. Therefore, when a histogram indicating the brightness distribution in the extended image data 502 is obtained as is, a histogram is also obtained in the dark areas (left side of the curve), as shown in histogram 506. When dark area correction is usually performed as is, strong dark area correction can be applied to the entire extended image to brighten the dark areas, and overcorrection may be performed, including the original image region. Therefore, when the histogram 507 obtained from the original image region 504 is h1(L) and the histogram 508 obtained from the extended region 505 is h2(L), the histogram 509 can be calculated as follows. Note that histogram 509 is represented by the name h3(L).

[0053] The correction amount for shadow correction is calculated based on the histogram obtained in this way. That is, also in this variation, the feature information of the entire extended image data is determined by synthesizing the feature information generated from each region of the extended image data with the weights of each region. This approach yields a rendering result that considers the extended region itself while reducing its influence on the original image region. In other words, by using rendering parameters calculated by considering the characteristics of the extended region in addition to the characteristics of the original image region for post-processing, a rendering result that considers the extended region while reducing its influence on the original image region can be obtained.

[0054] Variation Example 3 In the above embodiments, the processing of the original image region and the extended region using mask data and coordinate information indicating the extended region has been described. However, the implementation of this disclosure is not limited thereto. For example, the mask data may have multiple mask values.

[0055] In generative AI, in addition to processing the viewpoint of image data as described previously in the first embodiment, a subject can be added to a specific region of the image. The subject can be removed by synthesizing a background image. For example, not only from... Figure 6The captured image 601 is expanded to the same viewpoint as in expanded image 602, and trees can be added to region 603. In this case, as mentioned above, the generated region does not necessarily have the same characteristics as the original region, but the display parameters of the region are likely to be desired to some extent, since the subject is intentionally added by the user. Note that the generation of an image, or the removal of a subject, by adding objects such as other subjects to the original image, is referred to as image expansion, and the region including the added objects is called the expanded region even when the viewpoint of the original image has not changed. Therefore, image expansion can be referred to as image alteration or compositing, and the expanded region can be referred to as the altered region or the composite region.

[0056] Therefore, under the above circumstances, based on the expansion method of each region, the following mask data is generated, in which the coefficient of the unexpanded original image region 604 is 1, the coefficient of the automatically generated expanded region is 0.3, and the coefficient of the region generated by the user-assigned subject is 0.7. In this case, for example, when calculating the histogram, the histogram h(L) can be calculated based on the mask value m(x,y) and the brightness L(x,y) at coordinates (x,y) as follows.

[0057] Here, δ is the Dirac delta function, returning 1 when the independent variable is zero and 0 otherwise. The histogram h(L) indicates the number of pixels of brightness L weighted by the mask value m(x,y) for the entire extended image. Here, weighting is achieved by multiplying the position of the pixels of brightness L by the corresponding mask value. That is, also in this variation, the feature information of the entire extended image data is determined by synthesizing the feature information generated from each region of the extended image data with the weights of each region. However, in this variation, the weights are not fixed values, but are given as mask data. It is possible to predetermine which weight to give to which region indicated by the mask data, as in the example above.

[0058] Note that instead of keeping the weights as mask values ​​as described above, they can also be used as label information or as coefficients corresponding to the coordinates of each region. In this case, the weights (or coefficients) depending on the label or region can be set again when calculating the imaging parameters. More flexible weighting can be achieved by configuring the settings to be user-assignable.

[0059] In this way, by using the imaging parameters calculated by considering the characteristics of the extended region while taking into account the characteristics of the original image region for post-processing, it is possible to obtain an imaging result that reduces the impact of the extended region on the original image region while also considering the extended region. Furthermore, in this variant example, the weights of each region can be set by setting the coefficients as mask values.

[0060] Second Embodiment In the above embodiments and variations, the aspect of storing expanded images, such as those applied to captured images using AI, in a single image file has been described. In this embodiment, another aspect of performing expansion processing simultaneously during image capture will be described.

[0061] Reference Figure 7 Examples describing scenarios where extended processing is performed simultaneously during filming. Figure 7 Examples of images displayed on a display unit 108 including a touch panel and their operations are shown. Screen 701 shows an example of a live view display during shooting standby. On this screen, for example, if a tree is found to be obstructing the right side of the image and it is desired to remove it, as shown in screen 702, the user slides a guide at the edge of the screen and restricts the shooting area, thereby enabling the generation of an image 703 with the tree removed by synthesizing an expanded image on the outside of that area. If it is desired to shoot at a wider angle than the current live view display, the user can generate an image 705 by pinching the screen, reducing the shooting area, and expanding the insufficient area, as shown in screen 704. The CPU 103 acquires screen operations as trajectory data of operations on that screen, and generates expansion parameters corresponding to the operations (such as the removal of the subject in the assigned area or the expansion of the area that becomes empty due to reduction), and stores them, for example, in the main storage unit 104. In this way, in this embodiment, expansion parameters for expanding the object image data are set before shooting the object image data.

[0062] The following will refer to Figure 8 The flowchart describes the specific processing of the extended image display processing performed by the camera device 100 in this embodiment. The processing corresponding to the flowchart can be implemented by the CPU 103, for example, reading the corresponding processing program stored in the secondary storage unit 107, expanding it to the main storage unit 104 for execution. Figure 8 Extended image display processing will be described, for example, when the user makes an operation input (e.g., Figure 7When performing extended processing shooting (as shown on screen 702 or screen 704), it begins immediately after shooting in response to a shooting command using the shutter button, etc. At the beginning, the extended parameters input in the screen operation before shooting are stored, for example, in the main storage unit 104, and the image data captured in response to the subsequent shooting operation is also stored in the main storage unit 104. The image data stored in the main storage unit 104 is the object image data of the extended processing, and the object image data is also referred to as the captured image in the following description. Note that the execution of S801 and S803 can begin in parallel or asynchronously. However, if the end time point of S802 and the end time point of S805 are used as synchronization points, S806 and subsequent steps are executed after waiting for any delayed processing to complete.

[0063] In S801, the image processing unit 105 acquires the image characteristics of the captured image under the control of the CPU 103. In this process, similar to normal shooting without generation extension, regardless of the user's generation extension assignment, it can always obtain the image characteristics from the captured image. Figure 7 The image data corresponding to frame 701 (i.e., the unprocessed RAW data that has been captured) is used to obtain characteristic information for calculating display parameters. For example, the characteristic information to be obtained includes white balance and a histogram indicating brightness distribution, as shown in the first embodiment.

[0064] In S802, the image processing unit 105, under the control of the CPU 103, determines the display parameters by calculating the display parameters. Similarly, in this process, similar to S801, the display parameters can be calculated in the same way as in normal shooting. For example, the calculation can be performed in the same way as in S205, where the display parameters are calculated for the original image area corresponding to the mask data.

[0065] In S803, the image processing unit 105 performs AI extension processing under the control of the CPU 103. This can obtain an extended image such as image 703 or image 705. Here, the AI ​​extension processing can be compared with... Figure 2A The same method is used in S201, but the object image data is the captured image, and the extended parameters stored in the main storage unit 104 can be used as extended parameters.

[0066] In S804, the image processing unit 105, under the control of the CPU 103, acquires the extended region. Here, mask data for distinguishing the extended region can be generated as described above, and coordinate information indicating the extended region can be acquired. This process can be the same as the process in S202.

[0067] In S805, the image processing unit 105, under the control of the CPU 103, acquires the characteristics of the extended region (extended region characteristics). The extended region characteristics can be characteristics targeting only the region extended by the extended processing, or they can be characteristics targeting the entire extended image data, including both the original image region and the extended region. As described in Variations 1 to 3 of the first embodiment, this can be determined based on the weights given to the extended region's characteristic information (i.e., the coefficients to be multiplied). For example, as referenced... Figure 4 As stated above, if the feature information of the entire extended image data is weighted, the feature information of the entire extended image data can be obtained. Alternatively, as referred to... Figure 5 and Figure 6 If the original image region and the extended region are each weighted, then characteristic information focusing only on the extended region can be obtained.

[0068] Next, in S806, the image processing unit 105, under the control of the CPU 103, recalculates and determines the outcome. At this time, it synchronizes with the completion of S802, which has already been executed in parallel. That is, if S802 has finished, S806 is executed; otherwise, S806 is executed after waiting for completion.

[0069] In the recalculation judgment, for example, the histogram distributions of the original image region and the expanded region are compared, and if the similarity is high, it can be determined that recalculation is not necessary. Similarity can be judged based on various criteria, such as when the mean, variance, or peak distribution is within a predetermined range. Alternatively, if there is no subject in the expanded region or the expanded region was originally narrow, it can be determined that recalculation is not necessary.

[0070] If it is determined in S806 that a recalculation is needed, then the image processing unit 105, under the control of the CPU 103, updates the display parameters in S807. In this process, [further details can be added]. Figure 2B The processing in S205 is similar. On the other hand, if it is determined in S806 that recalculation is unnecessary, the processing proceeds to S808. This is because at the starting point of S806, the display parameters of the original image region have already been calculated in S802, and therefore the display parameters can be used for post-processing. Thus, the processing time can be shortened by skipping the recalculation of the display parameters.

[0071] In S808, the image processing unit 105, under the control of the CPU 103, performs post-processing by applying display parameters. Here, the calculated display parameters are applied to the entire extended image 703 and 705, and final image data is generated. The generated image data corresponds to the displayed image and is stored in the recording medium 106. When stored, the image data can be stored as compressed image data such as JPEG.

[0072] According to this embodiment, AI expansion processing, acquisition of characteristic information of the original image region, and determination of display parameters can be performed in parallel through the above configuration and processing procedures. Therefore, when it is not necessary to re-determine the display parameters corresponding to the expanded region, processing time can be shortened, and post-processing (i.e., correction processing or RAW display processing) can be performed quickly. If post-processing corresponding to the expanded region is required, its execution can improve the quality of the AI-expanded image after display.

[0073] Other variations Note that it is also possible to... Figure 8 The imaging parameters are always updated based on the extended image during the process. In this case, S801, S802, and S806 are not required, and S803 to S805, S807, and S808 can be executed.

[0074] Note that while AI-based image expansion has been described in all the above embodiments and variations, the techniques disclosed herein can also be applied to images expanded without using AI. For example, all embodiments and variations can be applied to expanded image data created by compositing a captured image with a pre-prepared background, and to expanded image data created by compositing multiple image data by segmentation. All the above embodiments and variations can also be applied to expanded data created by filling a manually assigned area with color.

[0075] In the above embodiments and variations, parameters are generated by weighting the feature information of each region using feature information (also called feature quantities) of the region specified by the region specification information. On the other hand, when maintaining the original image data before expansion and the expanded image data after expansion, as in the second embodiment, the original image region and the expanded region can be specified from this image data, thus eliminating the need to generate region specification information. If each region is specified without generating region specification information, parameters can be determined as in all the above embodiments and variations, and imaging processing can be performed. This eliminates the need for processing to generate region specification information, eliminates the need to store the generated region specification information, and reduces the processing load and the necessary amount of resources used for storage.

[0076] In the above embodiments and variations, examples of image expansion and display processing of expanded image data performed by a camera device such as a digital camera have been described. On the other hand, image data captured by a camera device such as a digital camera can be stored in an information processing device such as a personal computer, and this image data can be expanded and displayed by the information processing device. In this case, generative AI-based services provided via the Internet can be used for expansion processing.

[0077] Other embodiments Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0078] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: A determining component is configured to determine parameters for correction processing of the expanded image data generated by the image expansion function, based on feature information of each region in the regions of the original image region before expansion and the expanded region included in the expanded image data generated by the image expansion function, wherein the image data in the expanded image data is expanded; and A correction component for correcting the extended image data using the determined parameters.

2. The image processing apparatus according to claim 1, further comprising: A generation component is used to generate region specification information, which specifies the original image region and the extended region included in the extended image data generated by the image extension function. The determining component determines the parameters for the correction processing of the extended image data based on the feature information of the original image region and the extended region specified by the region designation information.

3. The image processing apparatus according to claim 2, further comprising: A storage component for storing the extended image data and the region designation information.

4. The image processing apparatus according to claim 2, in, The region designation information is mask data corresponding to each region in the original image region and the extended region.

5. The image processing apparatus according to claim 2, in, The region specification information is the location information of the boundary between the original image region and the extended region.

6. The image processing apparatus according to claim 1, in, The determining component determines the parameters based on feature quantities generated from the original image region excluding the extended region.

7. The image processing apparatus according to claim 1, in, The determining component determines the parameters based on feature information, which includes feature information generated from the extended region and the original image region, as well as feature information generated from the original image region excluding the extended region.

8. The image processing apparatus according to claim 2, in, The determining component synthesizes the feature information generated from each region specified by the region specification information by applying the weights of each region.

9. The image processing apparatus according to claim 8, in, The weights of each region are determined based on the expansion method applied to each region by the image expansion function.

10. The image processing apparatus according to claim 9, in, The expansion methods for each region include automatic expansion without user assignment to the subject, expansion with user assignment to the subject, and no expansion.

11. The image processing apparatus according to claim 1, further comprising: Provide components for providing the image expansion functionality.

12. The image processing apparatus according to claim 11, further comprising: A camera, used to take pictures.

13. A computer-readable storage medium storing a program that, when loaded and executed on a computer, causes the computer to perform processing, wherein the processing includes: Based on feature information of each region within the original image region before expansion and the expanded region included in the expanded image data generated by the image expansion function, parameters for the correction processing of the expanded image data generated by the image expansion function are determined, wherein the image data in the expanded image data is expanded; and The extended image data is corrected using the determined parameters.

14. A computer program product comprising a computer program that, when loaded and executed on a computer, causes the computer to perform processing, wherein the processing includes: Based on feature information of each region within the original image region before expansion and the expanded region included in the expanded image data generated by the image expansion function, parameters for the correction processing of the expanded image data generated by the image expansion function are determined, wherein the image data in the expanded image data is expanded; and The extended image data is corrected using the determined parameters.

15. A control method for an image processing device, the control method comprising: Based on feature information of each region within the original image region before expansion and the expanded region included in the expanded image data generated by the image expansion function, parameters for the correction processing of the expanded image data generated by the image expansion function are determined, wherein the image data in the expanded image data is expanded; and The extended image data is corrected using the determined parameters.

Images