Image processing device, image processing method

Abstract

To provide technology that enables the execution of image enhancement processing in consideration of the effects of image enhancement processing. [Solution] The degree of effectiveness of the image enhancement process on the input image is estimated. Then, a decision is made whether or not to perform the image enhancement process on the input image based on the degree of effectiveness, and if it is decided to perform the image enhancement process on the input image, the image enhancement process is performed on the input image.

Description

Technical Field

[0001] The present invention relates to high-image-quality technology.

Background Art

[0002] As a process for reducing or removing image degradation, there is a high-image-quality process. Specific process contents include a wide variety such as noise removal, color interpolation, aberration correction, super-resolution, and haze removal. In the technology disclosed in Non-Patent Document 1, by using a machine learning model such as a deep neural network for high-image-quality processing, it is possible to improve the high-image-quality performance of an image compared to the conventional rule-based method.

[0003] In recent years, high-performance high-image-quality processing using a machine learning model has been put into practical use as a product or service. Although the above high-image-quality processing is high-performance, it is characterized by a large amount of resources consumed in the execution of the processing. This is because the large number of learning parameters of the machine learning model causes an enormous amount of calculation in the processing. As a result, in products and services, there are cases where a large amount of power consumption, a large amount of time consumption, and associated usage fees for the processing occur as resources consumed.

[0004] In the technology disclosed in Patent Document 1, before executing high-image-quality processing that is high-performance and consumes a large amount of resources, the priority order of performing high-image-quality processing on an image is determined based on subject information, and high-image-quality processing is executed based on that order. Thereby, the resources consumed in high-image-quality processing can be consumed in the order of higher priority of high-image quality.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] [Patent Document 1] Patent No. 6282136 [Overview of the project] [Problems that the invention aims to solve]

[0007] Patent Document 1 describes an approach that performs image enhancement on all images, regardless of the magnitude of the enhancement effect. This approach, however, performs enhancement even on images where the enhancement effect is minimal, resulting in inefficient use of resources such as power, processing time, and costs. The present invention provides a technology that enables image enhancement in a manner that takes into account the effectiveness of the enhancement process. [Means for solving the problem]

[0008] One aspect of the present invention is characterized by comprising: estimation means for estimating the degree of effect of image quality enhancement processing on an input image; and image quality enhancement means for determining whether or not to perform image quality enhancement processing on the input image according to the degree of effect, and, in accordance with the determination to perform image quality enhancement processing on the input image, performing image quality enhancement processing on the input image. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technology that enables the execution of image quality enhancement processing in view of the effects of image quality enhancement processing. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing an example of the hardware configuration of the image processing device 100. [Figure 2] A block diagram showing an example of the functional configuration of the image processing device 100. [Figure 3] A flowchart illustrating the operation of the image processing device 100. [Figure 4] (a) is a flowchart showing the details of the process in step S302, and (b) is a diagram showing an example of the table configuration. [Figure 5] A block diagram showing an example of the functional configuration of the image processing device 500. [Figure 6] Flowchart of the operation of the image processing device 500. [Figure 7] A diagram showing an example of the configuration of List 701. [Figure 8] A block diagram showing an example of the hardware configuration of the image processing device 800. [Figure 9] A block diagram showing an example of the functional configuration of the image processing device 800. [Figure 10] Flowchart of the operation of the image processing device 800. [Figure 11] A diagram showing an example of the GUI display. [Figure 12] A block diagram showing an example of the system's hardware configuration. [Figure 13] A block diagram showing an example of the system's functional configuration. [Figure 14] A flowchart of the system's operation. [Figure 15] A diagram showing an example of the GUI display. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] [First Embodiment] In this embodiment, the degree of the effect of the first image quality improvement process for noise removal is estimated, and based on the result of the estimation, it is determined whether to actually execute the first image quality improvement process. If it is determined to execute, a case where the first image quality improvement process is actually performed on the input image will be described.

[0013] First, an example of the hardware configuration of the image processing apparatus 100 according to this embodiment will be described using the block diagram of FIG. 1. Note that the hardware configuration shown in FIG. 1 is merely an example of the hardware configuration for realizing the operations of the image processing apparatus described below, and can be appropriately modified / changed.

[0014] The control unit 101 is a processor such as a Central Processing Unit. The control unit 101 executes various processes using the computer programs and data stored in the RAM 103. Thereby, the control unit 101 controls the overall operation of the image processing apparatus 100 and executes or controls various processes described as the processes performed by the image processing apparatus 100.

[0015] The ROM (Read-Only Memory) 102 stores, for example, setting data of the image processing apparatus 100, computer programs and data related to the startup of the image processing apparatus 100, and computer programs and data related to the basic operations of the image processing apparatus 100. The ROM 102 also stores computer programs and data for causing the control unit 101 to execute or control various processes described as the processes performed by the image processing apparatus 100.

[0016] The RAM (Random Access Memory) 103 has an area for storing computer programs and data loaded from the ROM 102 and storage unit 104, and an area for storing captured images output from the imaging unit 105. Furthermore, the RAM 103 has a work area used by the control unit 101 when executing various processes. In this way, the RAM 103 can provide various areas as appropriate.

[0017] The storage unit 104 is a removable disk such as an SD card. However, the storage unit 104 may be any other type of storage device as long as it functions as non-volatile memory. For example, the storage unit 104 may be a hard disk drive, a solid-state drive, or other storage device. The storage unit 104 can also be implemented using a media (recording medium) and an external storage drive to access the media. Examples of such media include flexible disks (FD), CD-ROMs, DVDs, USB memory, MOs, and flash memory. Furthermore, the storage unit 104 is not limited to being included in the image processing device 100; for example, it may be a server device that can communicate with the image processing device 100 via a network such as a LAN or the Internet.

[0018] The imaging unit 105 includes an imaging lens, aperture, aperture motor, aperture control device, focus motor, focus control device, image sensor, A / D converter, image processing circuit, etc. The imaging lens includes a group of lenses such as a fixed lens, zoom lens, and focus lens. The aperture motor controls the drive of the aperture. The aperture control device controls the aperture of the imaging lens by changing the aperture opening diameter through drive control of the aperture motor.

[0019] The focus motor controls the drive of the focus lens. The focus control device controls the focus state of the imaging lens by driving the focus lens based on the phase difference of a pair of focus detection signals obtained from the image sensor.

[0020] The image sensor is a sensor such as a CCD or CMOS. The image of the subject is formed on the image sensor by the imaging lens, and the image sensor converts the formed image into an analog image signal. The A / D converter performs A / D conversion on the analog image signal converted by the image sensor to convert it into a digital image signal.

[0021] The image processing circuit generates an captured image by performing various image processing operations such as development, gain correction, gamma correction, and color correction on the digital image signal converted by the A / D converter. The control unit 101 stores the captured image in the RAM 103 or storage unit 104. The captured image may be a still image or an image of each frame in a moving image.

[0022] In this embodiment, as shown in Figure 1, the case in which the image processing device 100 includes an imaging unit 105 is described. However, it is not limited to this, and for example, the imaging unit 105 may be an external device to the image processing device 100. For example, the imaging unit 105 may be connected to the image processing device 100 by wire or wireless connection. In such a case, a computer device such as a personal computer can be used as the image processing device 100.

[0023] The resource supply unit 106 is a battery that supplies power to the image processing device 100, and the image processing device 100 operates using the power supplied by the resource supply unit 106. Note that the method of supplying power to the image processing device 100 is not limited to the method using the resource supply unit 106. The control unit 101, ROM 102, RAM 103, storage unit 104, imaging unit 105, and resource supply unit 106 are all connected to the system bus 107.

[0024] An example of the functional configuration of the image processing apparatus 100 according to this embodiment is shown in the block diagram of Figure 2. In this embodiment, the case in which each functional unit shown in Figure 2 is implemented by software (computer program) will be described. In the following, the functional units shown in Figure 2 may be described as the main processing units, but in reality, the functions corresponding to the functional units are realized by the control unit 101 executing the computer program corresponding to the functional unit. Note that one or more of the functional units shown in Figure 2 may be implemented by hardware. The operation of the image processing apparatus 100 will be described according to the flowchart in Figure 3.

[0025] In step S301, the image acquisition unit 201 acquires an image that has not undergone the first image enhancement process (unprocessed image) as the input image. The method by which the image acquisition unit 201 acquires the input image is not limited to a specific method. For example, the image acquisition unit 201 may acquire an unprocessed image stored in the storage unit 104 as the input image, or it may acquire an captured image (unprocessed image) output from the imaging unit 105 as the input image.

[0026] In step S302, a process is performed to estimate the degree of the effect of the first image enhancement process on the input image (degree of effect) based on the input image and a second image enhancement image obtained by performing a second image enhancement process on the input image that "reduces resource consumption, such as the amount of power required to execute the process, due to having less computation and fewer parameters required for the calculation than the first image enhancement process."

[0027] Resource consumption may include processing time or processing fees. An example of image enhancement processing with different computational loads and the number of parameters required for processing is a case where image enhancement processing using machine learning models such as neural networks trained using deep learning is designated as the first image enhancement process, and image enhancement processing using rule-based interpolation or filtering is designated as the second image enhancement process.

[0028] The details of the processing in step S302 will be explained according to the flowchart in Figure 4(a). In step S401, the image enhancement unit 202 performs a second image enhancement process on the input image acquired in step S301 to generate a second image enhancement image.

[0029] In step S402, the estimation unit 203 calculates a comparison score (image quality evaluation score) of image quality between the input image acquired in step S301 and the second high-resolution image generated in step S401. In this embodiment, by comparing the input image and the second high-resolution image, if it can be confirmed that there is a large change in the image due to the image enhancement process, it can be assumed that the degree of effectiveness of the first image enhancement process on the input image is large. In this embodiment, the case in which PSNR is calculated as the image quality comparison score is described, but the image quality comparison score is not limited to PSNR, and other image quality evaluation indices such as SSIM and MSE may also be calculated as the image quality comparison score.

[0030] In this case, when PSNR and SSIM are used as image quality comparison scores, a smaller image quality comparison score indicates a greater change in the image due to the second image quality enhancement process, meaning that the effect of the first image quality enhancement process is greater.

[0031] Furthermore, when MSE is used as the image quality comparison score, a higher image quality comparison score indicates a greater change in the image due to the second image quality enhancement process, meaning that the first image quality enhancement process is more effective.

[0032] In step S403, the estimation unit 203 determines the "degree of effectiveness of the first image enhancement process on the input image" based on the image quality comparison score calculated in step S402. The "degree of effectiveness of the first image enhancement process on the input image" is a measure that quantitatively expresses the image enhancement effect obtained by the first image enhancement process. In this embodiment, the "degree of effectiveness of the first image enhancement process on the input image" is a measure that expresses how much noise generated in the input image can be reduced by the first image enhancement process.

[0033] For example, the estimation unit 203 refers to the table (effectiveness rating table) illustrated in Figure 4(b) and identifies the degree of effectiveness corresponding to the image quality evaluation score calculated in step S402. In the case of the table in Figure 4(b), the estimation unit 203 identifies the corresponding degree of effectiveness as "high" if the image quality evaluation score calculated in step S402 is less than 25. The estimation unit 203 also identifies the corresponding degree of effectiveness as "medium" if the image quality evaluation score calculated in step S402 is 25 or more and less than 35. The estimation unit 203 also identifies the corresponding degree of effectiveness as "low" if the image quality evaluation score calculated in step S402 is 35 or more.

[0034] In this embodiment, we have described a case where the degree of effectiveness has three levels, but the degree of effectiveness is not limited to this; it may also have two levels or N levels (where N is an integer of 4 or more). Furthermore, an image quality evaluation score may be used as the degree of effectiveness.

[0035] Returning to Figure 3, in step S303, the determination unit 204 determines whether the first image enhancement process is highly effective on the input image, based on the "degree of effectiveness" estimated in step S302.

[0036] For example, if the degree of effectiveness is determined using the table in Figure 4(b), the determination unit 204 determines that the first image enhancement process is highly effective on the input image if the degree of effectiveness is "high," and that the first image enhancement process is less effective on the input image if the degree of effectiveness is "medium" or "low." Also, for example, if the image quality evaluation score is used as the degree of effectiveness, the determination unit 204 determines that the first image enhancement process is highly effective on the input image if the degree of effectiveness is above a threshold, and that the first image enhancement process is less effective on the input image if the degree of effectiveness is below a threshold.

[0037] In this embodiment, an image quality evaluation score is calculated based on the input image and the second high-resolution image, the degree of effectiveness is identified based on the image quality evaluation score, and it is determined whether or not the first high-resolution processing is highly effective on the input image based on the degree of effectiveness. Such a determination process is merely one example of a process that determines whether or not the first high-resolution processing is highly effective on the input image based on the difference in image quality between the input image and the second high-resolution image.

[0038] As a result of this assessment, if it is determined that the first image enhancement process is highly effective on the input image, the process proceeds to step S304. If it is determined that the first image enhancement process is not highly effective on the input image, the process is terminated according to the flowchart in Figure 3.

[0039] In step S304, the image enhancement unit 205 generates a first high-resolution image by performing a first high-resolution processing on the input image acquired in step S301. The first high-resolution processing is, for example, an image enhancement process using a machine learning model such as a neural network trained using deep learning, which consumes more resources than the second high-resolution processing.

[0040] The image enhancement unit 205 then stores the generated first high-resolution image in the RAM 103 or storage unit 104. The output destination of the first high-resolution image by the image enhancement unit 205 is not limited to a specific destination. Furthermore, the image enhancement unit 205 may also compress and encode the first high-resolution image before outputting it.

[0041] Thus, according to this embodiment, before executing the first image enhancement process, it is possible to determine whether or not the effect of the first image enhancement process is high, and the first image enhancement process can be executed if a significant effect of the first image enhancement process can be expected. As a result, the first image enhancement process can be performed only on images for which a significant effect of the first image enhancement process can be expected, thereby enabling more efficient use of battery power, reducing processing time, and reducing the amount of resources consumed.

[0042] The image enhancement unit 202 may perform a second image enhancement process on all input images acquired by the image acquisition unit 201 to generate a second image enhancement image, and store the generated second image enhancement image in the storage unit 104. This ensures that all input images, including those that did not undergo the first image enhancement process, are stored in the storage unit 104 as image enhancement images that have undergone a certain degree of enhancement processing.

[0043] <Example 1 of the degree of effectiveness> The estimation unit 203 may estimate the degree of effectiveness based on the shooting parameters at the time of shooting the input image acquired by the image acquisition unit 201. For example, in the case of noise reduction, the estimation unit 203 may acquire the ISO sensitivity at the time of shooting the input image acquired by the image acquisition unit 201 as the degree of effectiveness, with a higher ISO sensitivity indicating a higher degree of effectiveness. Images taken with a high ISO sensitivity have amplified noise compared to images taken with a low ISO sensitivity, resulting in a higher amount of noise. Images with a high amount of noise are expected to have a higher degree of effectiveness because more noise can be reduced in the first image quality enhancement process.

[0044] The determination unit 204 then determines that the effectiveness of the first image enhancement process on the input image is high if the degree of effectiveness (high ISO sensitivity) is above a threshold. On the other hand, the determination unit 204 determines that the effectiveness of the first image enhancement process on the input image is low if the degree of effectiveness (high ISO sensitivity) is below a threshold. In such cases, the operation of the image enhancement unit 202 is unnecessary, so the image processing device 100 does not need to have an image enhancement unit 202.

[0045] <Modification of the degree of effectiveness 2> The estimation unit 203 may estimate the degree of effect based on the pixel values ​​of pixels in the input image acquired by the image acquisition unit 201. For example, the estimation unit 203 may use a value calculated by referring to the pixel values ​​of the input image, such as the brightness value of a pixel in the input image, the statistical value of the brightness value, or the histogram of the brightness value, as the degree of effect.

[0046] For example, in noise reduction, the estimation unit 203 determines the degree of effectiveness such that the lower the average value of the brightness in the input image acquired by the image acquisition unit 201, the higher the degree of effectiveness, and the higher the average value, the lower the degree of effectiveness.

[0047] This is because, due to the human perceptual characteristic that noise occurring in low-luminance areas of an image is more easily perceived than noise in high-luminance areas, it is expected that this method will be highly effective in reducing more of the easily perceived noise.

[0048] The determination unit 204 then determines that the effectiveness of the first image enhancement process on the input image is high if the degree of effectiveness (a value calculated by referring to the pixel values ​​of the input image) is equal to or greater than a threshold. On the other hand, the determination unit 204 determines that the effectiveness of the first image enhancement process on the input image is low if the degree of effectiveness (a value calculated by referring to the pixel values ​​of the input image) is less than a threshold. In such cases, the operation of the image enhancement unit 202 is unnecessary, so the image processing device 100 does not need to have an image enhancement unit 202.

[0049] In the first embodiment, the case in which the image quality enhancement process is noise reduction was described, but in addition to or instead of noise reduction, processes such as debayering, aberration correction, and super-resolution may also be performed as image quality enhancement processes.

[0050] [Second Embodiment] In the following embodiments and modifications, including this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, they will be the same as the first embodiment. In this embodiment, the first image enhancement process is performed on multiple input images within a specified resource consumption range.

[0051] Figure 5 shows an example of the functional configuration of the image processing device 500 according to this embodiment. In this embodiment, as in the first embodiment, the case in which each functional unit shown in Figure 5 is implemented by software (computer program) will be described, but one or more of the functional units shown in Figure 5 may be implemented by hardware. The hardware configuration example of the image processing device 500 is assumed to be the same as that of the image processing device 100. The operation of the image processing device 500 will be described according to the flowchart in Figure 6.

[0052] In step S601, the image acquisition unit 201 acquires multiple input images. In this embodiment, the processing in step S302 is performed on each input image acquired in step S601, and as a result, the degree of effectiveness is obtained for that input image. In this embodiment, as an example, a case in which the PSNR value, which is an image quality comparison score, is used as the degree of effectiveness will be described.

[0053] In step S603, the determination unit 505 sorts the multiple input images acquired in step S601 in descending order of effectiveness. For example, as shown in Figure 7, the determination unit 204 generates a list 701 in which the set of input images and the effectiveness level of those input images is sorted in descending order of effectiveness (in descending order of PSNR).

[0054] In step S604, the acquisition unit 504 acquires resource information necessary for the decision-making process in the decision-making unit 505. In this embodiment, the acquisition unit 504 acquires the amount of power consumed when the image enhancement unit 205 performs the first image enhancement process, and the total amount of power that can be supplied from the resource supply unit 106. Here, the type of resource information to be acquired is not limited to the amount of power, but may also be, for example, time.

[0055] The determination unit 505 then selects, based on the sorting result in step S603 and the resource information acquired in step S604, the input images acquired by the image acquisition unit 501 that are to be subjected to the first image enhancement process as selected images.

[0056] More specifically, the determination unit 505 selects an input image from the input images acquired by the image acquisition unit 501 as the input image to be subjected to the first image enhancement process, such that the resource information falls within a range of pre-set criteria.

[0057] For example, suppose the pre-set threshold for resource consumption is "power consumption that falls within 3% of the total battery energy." If the power consumption of the first image enhancement process for one input image is 1% of the total battery energy, then for the first image enhancement process on three input images, the resource consumption will fall within the pre-set threshold. In such a case, the determination unit 204 selects the top three input images with the highest degree of effectiveness as the input images to be processed for the first image enhancement process.

[0058] In step S605, the image enhancement unit 205 generates a first image enhancement image by performing a first image enhancement process on the input image (selected image) selected in step S604.

[0059] Thus, according to this embodiment, before executing the first image enhancement process, the first image enhancement process can be executed on input images that are expected to show a significant effect from the first image enhancement process, within a specified power consumption range. This makes it possible to significantly increase the effect of the image enhancement process on the resources consumed by the first image enhancement process.

[0060] [Third Embodiment] In this embodiment, the degree of effectiveness and resource consumption are presented to the user, and the user is asked to select an input image to be subjected to the first image enhancement process. The first image enhancement process is then performed on the input image selected by the user. An example of the hardware configuration of the image processing device 800 according to this embodiment will be explained using the block diagram in Figure 8.

[0061] The display unit 806 has a liquid crystal screen or a touch panel screen and displays the processing results from the control unit 101 as images, text, etc. The display unit 806 may also be an external device connected to the image processing device 800 by wire or wireless connection.

[0062] The operation unit 807 is a user interface such as a keyboard, mouse, or touch panel, and allows the user to input various instructions and information to the image processing device 800 through its operation. If the operation unit 807 is a device that can output images and text, such as a touch panel, it can be configured as the same hardware as the display unit 806. The control unit 101, ROM 102, RAM 103, storage unit 104, imaging unit 105, display unit 806, and operation unit 807 are all connected to the system bus 808.

[0063] An example of the functional configuration of the image processing device 800 is shown in the block diagram of Figure 9. In this embodiment, as in the first embodiment, the case in which each functional unit shown in Figure 9 is implemented by software (computer program) will be described, but one or more of the functional units shown in Figure 9 may be implemented by hardware. The operation of the image processing device 800 will be described according to the flowchart in Figure 10.

[0064] In step S1005, the reception unit 905 displays a GUI on the display unit 806 that includes each input image acquired in step S601, the degree of effect estimated for each input image in step S302, and the resource information acquired by the acquisition unit 504. The reception unit 905 controls the display of the GUI. An example of the GUI display is shown in Figure 11. Here, we will describe the case where the display unit 806 is a touch panel.

[0065] GUI1100 displays an input image 1104, an estimated degree of effectiveness 1101 for the input image 1104, and a checkbox 1103 for the input image 1104. GUI1100 also displays resource information 1102, including power consumption and processing time. Resource information 1102 may also represent processing charges.

[0066] The user can select the input image to be processed for the first image enhancement by touching the corresponding checkbox 1103 with their finger, referring to the input image, effect level, resource information, etc., displayed on the GUI 1100. Figure 11 shows a check mark on the checkbox 1103 of the leftmost input image 1104, indicating that the user has selected this leftmost input image 1104 as the input image to be processed for the first image enhancement.

[0067] In step S1006, the reception unit 905 receives user input to select an input image to be processed for the first image enhancement. When the GUI 1100 shown in Figure 11 is displayed, the reception unit 905 receives touch input to the checkbox 1103. In step S1007, the image enhancement unit 205 generates a first enhanced image by performing the first image enhancement process on the input image selected according to the user input.

[0068] Note that the GUI 1100 in Figure 11 is merely one example of a GUI that presents the user with input images, effect level, and resource information, and allows the user to select the input image to be processed for the first image enhancement. In other words, any GUI that can achieve a similar purpose is acceptable. For example, the information displayed in the GUI is not limited to all of the input image, effect level, and resource information, and other information may be displayed in addition to or in addition to this information. Also, other selection methods may be used instead of the input image selection method using checkbox 1103.

[0069] Thus, according to this embodiment, the first image enhancement process can be performed on an input image selected by the user based on the amount of resources consumed and the degree of effectiveness for image enhancement.

[0070] [Fourth Embodiment] In this embodiment, we will describe a case in which the third embodiment is implemented using a client-side device such as a camera and a server-side device such as a cloud.

[0071] An example of the hardware configuration of a system that functions as an image processing device according to this embodiment will be described using the block diagram in Figure 12. The system according to this embodiment has a client 1200 and a cloud 1210, and the client 1200 and the cloud 1210 are configured to be able to communicate data with each other via a wired and / or wireless network 1280.

[0072] The client 1200 is a client-side device such as a camera, and in addition to the configuration of the image processing device 800, it has a communication unit 1208. The RAM 103 has an area for storing data received from the cloud 1210 by the communication unit 1208. The communication unit 1208 performs data communication with the cloud 1210 via the network 1280.

[0073] The CPU 101, ROM 102, RAM 103, storage unit 104, imaging unit 105, display unit 806, operation unit 807, and communication unit 1208 are all connected to the system bus 1209.

[0074] Next, we will explain Cloud 1210. Cloud 1210 is a server-side device, such as in a cloud environment. Figure 12 shows a case where Cloud 1210 is implemented with a single device, but it can also be configured with multiple devices.

[0075] The control unit 1211 executes various processes using computer programs and data stored in the RAM 1213. In doing so, the control unit 1211 controls the overall operation of the cloud 1210 and executes or controls the various processes described as those performed by the cloud 1210.

[0076] ROM1212 stores configuration data for Cloud 1210, computer programs and data related to starting Cloud 1210, computer programs and data related to the basic operation of Cloud 1210, and so on.

[0077] RAM 1213 has an area for storing computer programs and data loaded from ROM 1212 and HDD 1214, and an area for storing data received from client 1200 by communication unit 1215. Furthermore, RAM 1213 has a work area used by control unit 1211 when executing various processes. In this way, RAM 1213 can provide various areas as appropriate.

[0078] HDD1214 stores computer programs and data that cause the control unit 1211 to execute or control various processes described as being performed by the OS and cloud 1210.

[0079] The communication unit 1215 performs data communication with the client 1200 via the network 1280. The control unit 1211, ROM 1212, RAM 1213, HDD 1214, and communication unit 1215 are all connected to the system bus 1216.

[0080] An example of the functional configuration of the system according to this embodiment is shown in the block diagram in Figure 13. In this embodiment, as in the first embodiment, the case in which each functional part shown in Figure 13 is implemented by software (computer program) will be described, but one or more of the functional parts shown in Figure 13 may be implemented by hardware. The operation of the system according to this embodiment will be described according to the flowchart in Figure 14.

[0081] In step S1407, the reception unit 905 transmits the input image selected in response to the user operation to the cloud 1210 via the communication unit 1208. In step S1408, the image enhancement unit 205 receives the input image transmitted from the client 1200 via the communication unit 1213. The image enhancement unit 205 then generates a first high-resolution image by performing a first high-resolution processing on the received input image. In step S1409, the image enhancement unit 205 transmits the first high-resolution image generated in step S1408 to the client 1200 via the communication unit 1213.

[0082] Subsequently, the client 1200 receives the first high-resolution image transmitted from the cloud 1210 via the communication unit 1208. The output destination of the received first high-resolution image is not limited to a specific destination, as in the first embodiment.

[0083] In such a system, for example, charges could be incurred based on the number of first high-resolution images generated on the server side. In that case, the charges can be suppressed by keeping the number of images to the minimum necessary. The pay-per-use system may be implemented on client 1200 or on other systems.

[0084] Note that the system's functional configuration shown in Figure 13 is just one example and can be modified as appropriate. For example, one or more of the image enhancement unit 202, estimation unit 203, and acquisition unit 504 may be included in the cloud 1210. In other words, the way in which the operation of the image processing device 800 according to the third embodiment is divided between the client 1200 and the cloud 1210 is not limited to a specific configuration. Furthermore, other devices may be added to the system in addition to the client 1200 and the cloud 1210. In this case as well, the way in which the operation of the image processing device 800 according to the third embodiment is divided between the client 1200, the cloud 1210, and other devices is not limited to a specific configuration. Thus, according to this embodiment, if the server-side image enhancement processing is billed on a pay-per-use basis based on the number of images processed, the cost can be reduced by applying image enhancement processing only to images for which enhancement is effective. Furthermore, by performing resource-intensive image enhancement processing on the server side, power consumption on the client side, such as cameras, can be reduced.

[0085] Figure 15 shows an example of the GUI display in step S1005. Here, we describe the case where the display unit 806 is a touch panel. The reception unit 905 sorts the input images in descending order of effectiveness and displays the sorted input images on the GUI 1500, with the input images with higher effectiveness positioned higher. The GUI 1500 also includes an estimated effectiveness level 1501 for the input image 1504 and a checkbox 1503 for the input image 1504. Resource information 1502, including processing time and incurred charges, is also displayed on the GUI 1500. The incurred charges are calculated according to the number of input images with a checkmark in the checkbox 1503.

[0086] The user can select the input images to be processed for the first image enhancement by touching the corresponding checkbox 1503 with their finger, referring to the input images, effect level, resource information, etc., displayed on the GUI 1500. In Figure 15, the checkboxes 1503 for the topmost input image and the second-to-last input image are checked, indicating that these two input images have been selected by the user as the input images to be processed for the first image enhancement. The charges in resource information 1502 are calculated based on these two input images selected by the user.

[0087] By presenting the GUI1500 to the user, they can quickly identify images that will benefit greatly from image enhancement. In addition, users can select images to enhance after considering factors such as the cost involved and the balance between the enhancement effect and the cost.

[0088] The numerical values, processing timing, processing order, processing entity, data (information) structure / acquisition method / destination / source / storage location, etc., used in the above embodiment are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0089] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0090] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0091] The inventions described herein include the following image processing apparatus, image processing method, and computer program. (Item 1) An estimation means for estimating the degree of the effect of image quality enhancement processing on an input image, A means for performing image quality enhancement on the input image is provided to determine whether or not to perform image quality enhancement processing on the input image according to the degree of the effect, and to perform image quality enhancement processing on the input image according to the determination that the image quality enhancement processing on the input image is to be performed. An image processing apparatus characterized by comprising: (Item 2) The image processing apparatus according to item 1, characterized in that the estimation means estimates the degree of the effect based on an input image and an enhanced image obtained by performing an enhanced image processing on the input image that consumes fewer resources than the enhanced image processing. (Item 3) The image processing apparatus according to item 2, characterized in that the estimation means estimates the degree of the effect based on the difference in image quality between the input image and the high-resolution image. (Item 4) The image processing apparatus according to item 1, characterized in that the estimation means estimates the degree of the effect based on the shooting parameters of the input image. (Item 5) The image processing apparatus according to item 1, characterized in that the estimation means estimates the degree of the effect based on the pixel values ​​of the input image. (Item 6) An estimation means for estimating the degree of effectiveness of image enhancement processing for multiple input images, A high-quality enhancement means that performs high-quality enhancement on an input image selected from the plurality of input images based on the degree of the effect and the amount of resources consumed for the high-quality enhancement processing on the input image. An image processing apparatus characterized by comprising: (Item 7) The image processing apparatus according to item 6, characterized in that the image quality enhancement means performs image quality enhancement processing on input images selected according to the consumption amount, starting from input images with a greater degree of the effect. (Item 8) moreover, The system includes a means for presenting the degree of the effect and the amount of consumption to the user. The image enhancement means performs image enhancement processing on an input image selected from the plurality of input images in response to user operation. The image processing apparatus according to item 6 or 7, characterized by the features described therein. (Item 9) The image processing apparatus according to item 8, characterized in that the image processing apparatus comprises a first apparatus having the estimation means and the presentation means, and a second apparatus having the image quality enhancement means. (Item 10) The presentation means transmits the input image selected in response to user operation to the second device. The image enhancement means performs image enhancement processing on the input image transmitted by the presentation means. The image processing apparatus according to item 9, characterized in that (Item 11) The image processing apparatus further comprises an imaging means, The image processing apparatus according to any one of items 1 to 10, characterized in that the input image is an image captured by the imaging means. (Item 12) An image processing method performed by an image processing device, The estimation means of the image processing device includes an estimation step of estimating the degree of the effect of the image quality enhancement process on the input image, The image processing apparatus's image enhancement means determines whether or not to perform image enhancement processing on the input image according to the degree of the effect, and, in response to the determination to perform image enhancement processing on the input image, performs an image enhancement step of performing image enhancement processing on the input image. An image processing method characterized by comprising: (Item 13) A computer program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 through 11.

[0092] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0093] 100: Image processing device 201: Image acquisition unit 202: Image quality enhancement unit 203: Estimation unit 204: Judgment unit 205: Image quality enhancement unit

Claims

1. An estimation means for estimating the degree of the effect of image quality enhancement processing on an input image, A means for performing image quality enhancement on the input image is provided to determine whether or not to perform image quality enhancement processing on the input image according to the degree of the effect, and to perform image quality enhancement processing on the input image according to the determination that the image quality enhancement processing on the input image is to be performed. An image processing apparatus characterized by comprising:

2. The image processing apparatus according to claim 1, characterized in that the estimation means estimates the degree of the effect based on an input image and an enhanced image obtained by performing an enhanced image processing on the input image that consumes fewer resources than the enhanced image processing.

3. The image processing apparatus according to claim 2, characterized in that the estimation means estimates the degree of the effect based on the difference in image quality between the input image and the high-resolution image.

4. The image processing apparatus according to claim 1, characterized in that the estimation means estimates the degree of the effect based on the shooting parameters of the input image.

5. The image processing apparatus according to claim 1, characterized in that the estimation means estimates the degree of the effect based on the pixel values ​​of the input image.

6. An estimation means for estimating the degree of effectiveness of image enhancement processing for multiple input images, A high-quality enhancement means that performs high-quality enhancement on an input image selected from the plurality of input images based on the degree of the effect and the amount of resources consumed for the high-quality enhancement processing on the input image. An image processing apparatus characterized by comprising:

7. The image processing apparatus according to claim 6, characterized in that the image quality enhancement means performs image quality enhancement processing on input images selected according to the consumption amount, starting from input images with a greater degree of the effect.

8. moreover, The system includes a means for presenting the degree of the effect and the amount of consumption to the user. The image enhancement means performs image enhancement processing on an input image selected from the plurality of input images in response to user operation. The image processing apparatus according to claim 6.

9. The image processing apparatus according to claim 8, wherein the image processing apparatus comprises a first apparatus having the estimation means and the presentation means, and a second apparatus having the image quality enhancement means.

10. The presentation means transmits the input image selected in response to user operation to the second device. The image enhancement means performs image enhancement processing on the input image transmitted by the presentation means. The image processing apparatus according to feature 9.

11. The image processing apparatus further comprises an imaging means, The image processing apparatus according to claim 1, characterized in that the input image is an image captured by the imaging means.

12. An image processing method performed by an image processing device, The estimation means of the image processing device includes an estimation step of estimating the degree of the effect of the image quality enhancement process on the input image, The image processing apparatus's image enhancement means determines whether or not to perform image enhancement processing on the input image according to the degree of the effect, and, in response to the determination to perform image enhancement processing on the input image, performs an image enhancement step of performing image enhancement processing on the input image. An image processing method characterized by comprising:

13. A computer program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 11.

Images