Image processing device and control method thereof, imaging device, and program

Abstract

To provide an image processing device in which image processing effects of a plurality of images which are displayed in a list can be easily ascertained.SOLUTION: An imaging device 100 includes a display unit 111 configured to display captured images in a list, and an image processing unit 104 configured to edit an image selected out of a plurality of images displayed in the list through a predetermined retouch process. A photographing image is acquired (S402) and an automatically retouched image is acquired (S403). The imaging device 100 calculates a retouch score as an evaluation value (S404) and performs retouch simulation on an image of which a retouch score is equal to or greater than a threshold value (S405). The image processing unit 104 generates thumbnail images based on images before and after the editing of the image, and the display unit 111 displays the list using the generated thumbnail images.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a list display process of images in an imaging device or the like. [Background technology]

[0002] In recent years, users can enjoy various photographic expressions by retouching photographs (photo retouching). In doing so, it is troublesome to search for an appropriate retouching method for each photographed image through trial and error. Patent Document 1 discloses an image processing device and an image processing method that enable a user to easily notice the effect and select an easy-to-use special image processing. By displaying images with retouching effects side by side at the time of shooting, the user can shoot while checking the retouched image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-121914 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the method described in Patent Document 1, if the user tries to retouch all the images to be taken in order to check the effect of retouching when taking the image, the shooting operation takes a long time. The user has to concentrate on shooting, and it may become unclear intuitively which image will change and how with what retouching. An object of the present invention is to provide an image processing device that allows easy confirmation of the image processing effects of a plurality of images displayed in a list. [Means for solving the problem]

[0005] The image processing apparatus according to the embodiment of the present invention includes a selection unit for selecting an image from a plurality of images displayed on a display unit in response to a user's operation on an operation unit, and a process for selecting a first image from the first image selected by the selection unit. image processing selected based on the result of image analysis of a live view image acquired when the first image was captured, or image processing selected in response to a user's operation on the operation means The image processing device includes a generating means for generating a second image by performing image processing, and a control means for displaying a third image, which is an image that changes over time from one of the first image and the second image to the other, as at least one of the images displayed in a list on the display means. Effect of the Invention

[0006] According to the present invention, it is possible to provide an image processing device that allows easy confirmation of the image processing effects of a plurality of images displayed in a list. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an example of the configuration of an imaging apparatus according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram illustrating an image list display. [Diagram 3] 13 is a flowchart illustrating an operation of a user. [Figure 4] 13 is a flowchart of an automatic retouching simulation process. [Diagram 5] FIG. 13 is a diagram showing changes in coefficients over time in an automatic retouching simulation. [Figure 6] 10 is a flowchart showing processing during shooting for performing an automatic retouching simulation. [Figure 7] 13 is a flowchart of a process for automatically selecting a retouching method. [Figure 8] 11A and 11B are schematic diagrams illustrating calculation of a reflection component in a relighting process. [Figure 9] FIG. 4 is a diagram showing an example of a screen of a display unit during shooting. [Figure 10] 13 is a flowchart showing another example of a process for automatically selecting a retouching method. [Figure 11]13 is a flowchart of an arbitrary retouching simulation process designated by a user. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In this embodiment, an image processing device according to the present invention is applied to an image capture device such as a smartphone. For example, the image capture device is provided with a list display application and is capable of displaying a list of images.

[0009] 1 is a block diagram showing the configuration of an image capturing apparatus 100 according to this embodiment. The optical system 101 is an image capturing optical system having optical members such as a focus lens, an aperture, a shutter, etc. In the optical system 101, focusing on a subject is performed by driving the focus lens during shooting, and the amount of exposure is adjusted by controlling the aperture and the shutter.

[0010] The image sensor 102 performs photoelectric conversion on the subject image formed by the optical system 101 and outputs an electrical signal. The image sensor 102 includes a photoelectric conversion element using a CCD (charge-coupled device), a CMOS (complementary metal-oxide semiconductor), or the like. The A / D conversion unit 103 performs digitalization processing on the analog electrical signal input from the image sensor 102. The digitized image signal is output to an image processing unit 104.

[0011] The image processing unit 104 performs synchronization processing, white balance correction processing, gamma processing, retouching processing, etc. on the image signal obtained from the A / D conversion unit 103, and outputs the image signal after image processing to the recording unit 107.

[0012] The distance measuring unit 105 acquires distance information (subject distance information) to the subject at the time of shooting and generates distance map data. The distance map data is a two-dimensional array of data that indicates subject distance information in pixel units in the captured image. The distance map data is output to the image processing unit 104.

[0013] The face detection unit 106 detects the face area of ​​the subject person in the captured image. Face detection is performed to determine the subject area in the retouching process performed by the image processing unit 104. For example, the coordinate values ​​of the subject's face organs such as the right eye, left eye, and mouth are detected, and the distance between each face organ (such as the distance between the right eye and the left eye) is calculated. Based on the distance between each face organ, an elliptical area is set with the center of gravity coordinates of each organ coordinate as the center so that the background area is not included, and the elliptical area is detected as the face area. Information on the face detection result is output to the image processing unit 104.

[0014] The recording unit 107 acquires the image signal output from the image processing unit 104, converts it into data in an image format such as JPEG (Joint Photographic Experts Group), and records it on a recording medium. For example, the recording medium is a memory that is detachable from the imaging device 100.

[0015] The control unit 108 includes a CPU (Central Processing Unit) and controls the operation of the entire imaging device. For example, the control unit 108 calculates a target exposure amount in the optical system 101 based on brightness information of a subject detected immediately before shooting. The control unit 108 also calculates a predetermined evaluation value based on a shot image and determines parameters for image processing performed by the image processing unit 104. The memory 109 stores information used by the image processing unit 104, information on settings and lighting conditions during shooting, and outputs stored data to the image processing unit 104 as necessary.

[0016] The operation unit 110 includes an operation member or an operation device for a user to input operation instructions to the imaging device. The display unit 111 includes, for example, a liquid crystal display equipped with a touch panel. The display unit 111 displays a screen for assisting in operations during shooting, and displays images such as image data stored in the recording unit 107.

[0017] Fig. 2 is a schematic diagram showing an example of the external appearance of the imaging device 100. Fig. 2(A) is a diagram showing a surface of the imaging device 100 having a display unit 111. Fig. 2(B) is a diagram showing the reverse surface of Fig. 2(A). Fig. 2(C) is a diagram showing an example of display by a list display application. Fig. 2(D) is a diagram showing a state in which a retouch simulation menu has been launched.

[0018] The display unit 111 in Fig. 2(A) has a liquid crystal display equipped with a touch panel, and is capable of display and touch detection. Fig. 2(B) shows an optical system 101 including a lens, and a distance measurement unit 105. The distance measurement unit 105 measures distances at an angle of view similar to that of the optical system 101, for example, by a Time of Flight (ToF) method.

[0019] The display screen shown in FIG. 2(C) is a display screen of a list display application. In the lower area of ​​the display screen, first to fourth buttons are displayed in order from the left. The first button 201 is a back button for a return operation, the second button 202 is a home button, the third button 203 is a retouch simulation button, and the fourth button 204 is a menu button. These operation buttons are displayed on a touch panel provided in the display unit 111, and functions associated with each button are realized in response to a tap operation by the user. The back button 201 is a button used to return the screen transition state to the previous state. The home button 202 is a button used to pause the list display application and transition to the home screen. The behavior when the user taps the retouch simulation button 203 will be described later. The menu button 204 is a button for starting a menu used to perform detailed list display settings.

[0020] As shown in FIG. 2(D), when the user continues to tap the retouch simulation button 203 for a predetermined period of time or more, the retouch simulation menu 205 is activated and an operation area is displayed.

[0021] Next, a method for checking the retouching effect (image processing effect) of a plurality of images on the image list display application will be described. Fig. 3 is a flowchart for explaining the operation of a user. In S301, the user performs an operation to start up the imaging device 100, and processing of the list display application is started.

[0022] In S302, when the user taps the retouch simulation button 203, the process proceeds to a determination process in S303. In S303, the control unit 108 performs a determination process based on the tap operation time in S302. The time that the user's finger or the like touches the display unit 111 is compared with a predetermined time (threshold time), and if a short-second tap within the predetermined time is detected, the process proceeds to a process in S304. If a long-second tap, in which the touch time on the display unit 111 is equal to or longer than the predetermined time, is detected, the process proceeds to a process in S305.

[0023] In S304, control unit 108 executes a process of presenting the results of the automatic retouching simulation to the user. The user can check the results on the display screen. The detailed process of the automatic retouching simulation will be described later.

[0024] In S305, display unit 111 displays the screen of retouching simulation menu 205. From retouching simulation menu 205, the user can select any retouching simulation.

[0025] In S306 following S305, control unit 108 executes a process of presenting the execution results of the selected retouching simulation to the user. The user can check the execution results of the desired retouching simulation on the display screen. The detailed processing content of the arbitrary retouching simulation will be described later.

[0026] After the processing of S304 or S306, the process proceeds to S307. In S307, the control unit 108 receives an image selection instruction from the operation unit 110. When the user selects an image to be checked in detail, the process transitions from the list display application to the image viewer application. The selected image can be enlarged, and the user can check the image quality resulting from the image processing effects.

[0027] The contents of the automatic retouching simulation process (FIG. 3: S304) will be described with reference to Fig. 4. Fig. 4 is a flowchart illustrating the internal process of the imaging device 100 when confirming the image processing effect that has been automatically determined by the imaging device 100.

[0028] First, in S401, control unit 108 determines the target images for simulation processing according to the screen scroll status. That is, processing is performed to select a group of images to be the targets of retouching simulation processing. With the image list display function, it is possible to display as many images as can be stored in recording unit 107. When performing retouching simulation processing, it is necessary to limit the target images, because otherwise the processing time will be enormous. In S401, the following processing is performed to avoid a decrease in usability due to an increase in processing time.

[0029] The number of images to be processed is represented as n, and a setting process is performed. The number of images to be processed n is assumed to be greater than the number of images that can be displayed at one time on the display unit 111 by the list display application. When the list display application is started in S301 of FIG. 3, the image to be displayed first is the i-th image. Images from the "in / 2"-th image to the "i+n / 2"-th image are selected and set as a group of target images for simulation processing. In association with the user's scroll operation, a process is performed to update the group of target images for simulation processing before all of the group of target images for simulation processing move outside the display screen. In the updating method, the target images for simulation processing (n / 2 images) that are far from (distant from) the target images for simulation processing displayed on the screen are excluded from the target images. Then, images (n / 2 images) in the direction of the progress of the user's scroll operation are added as new target images for simulation processing.

[0030] Although the update unit has been described as half the number of images to be processed n, there is an embodiment in which the user can arbitrarily set the number of images to be processed and the update unit. The update unit can also be set taking into account the grouping of shooting dates, etc. An embodiment in which the update process is controlled according to the scroll speed may also be used. If the scroll speed is extremely high, performing the update operation successively may interfere with the user's operation. Therefore, if the user's scroll speed is higher than a predetermined threshold, the update process is interrupted. As soon as it is detected that the user's scroll operation has stopped, the first image displayed in the list is set as the i-th image, and a process is performed to determine the target image group for the simulation process.

[0031] Next, in S402, an image at the time of shooting is acquired. A process is performed to acquire an image at the time of shooting (referred to as Im_org) as a target image for each simulation process. In S403, the control unit 108 refers to metadata of the target image for each simulation process and performs a process to acquire an automatically retouched image (referred to as Im_re) recorded at the time of shooting. The contents of the process of recording an automatically retouched image at the time of shooting will be described later.

[0032] After S403, in S404, the control unit 108 calculates a retouching score as an evaluation value for judgment. The retouching score indicates the amount of change caused by retouching. When automatic retouching processing is performed, a low retouching score is set for an image that has little change from the image at the time of shooting, and a high retouching score is set for an image that has more change from the image at the time of shooting.

[0033] In the method of calculating the retouching score, a process is executed to quantify the difference between the automatically retouched image Im_re and the image Im_org at the time of shooting. For example, the control unit 108 calculates the amount of difference using the sum of absolute differences. The sum of absolute differences in pixel values ​​of each pixel and each color channel between the automatically retouched image Im_re and the image Im_org at the time of shooting is calculated as the retouching score.

[0034] Although the method of using the sum of absolute differences has been explained, there is also a method of using a weighted sum of absolute differences in which the weight of each color channel is controlled. In addition to the method of quantifying the difference between the automatically retouched image Im_re and the image at the time of shooting Im_org, there is also a method of judging aesthetics using a database that records a huge number of retouching scores. A database is prepared in which scores are linked to various retouching results in advance, and the automatically retouched image at the time of shooting is compared with the database. This makes it possible to select similar images and obtain the score linked to the selected image as the retouching score.

[0035] In S405 following S404, the control unit 108 executes a retouching simulation only for images whose retouching scores are equal to or greater than the threshold. For images whose retouching changes are minimal, there is no need to execute a retouching simulation. For example, a gray-out process is performed on an image whose retouching score calculated in S404 is less than the threshold.

[0036] For images whose retouching score calculated in S404 is equal to or greater than the threshold, a retouching simulation is performed in the following manner. The retouching simulation is realized by displaying a time-varying image that is generated using the image Im_org at the time of shooting acquired in S402 and the automatically retouched image Im_re acquired in S403. This image is called a retouching simulation video, and is denoted by M_sim. The retouching simulation video M_sim is generated by the following (Equation 1). M_sim=αt * Im_org + (1-αt) * Im_re ...(Formula 1) The coefficient αt in (Equation 1) represents a weighting coefficient for weighted synthesis. The value of αt is a real number ranging from 0 to 1, and is a value that changes depending on the playback time. In other words, the image Im_org at the time of shooting and the automatically retouched image Im_re are weighted and synthesized using the coefficient αt. Note that if the image size of the image Im_org at the time of shooting and the image size of the automatically retouched image Im_re differ, a resizing process is performed in advance to make the image sizes uniform.

[0037] The change in the value of the coefficient αt over time will be described with reference to Fig. 5. Fig. 5(A) shows an example in which the value of the coefficient αt changes linearly over time, and Fig. 5(B) shows an example in which the value of the coefficient αt changes discontinuously. The horizontal axis is the time axis, and T represents the length of one period of the temporal change. The vertical axis shows the value of the coefficient αt from zero to time T in the range of 0 to 1.

[0038] In the graph of Fig. 5(A), the value of the coefficient αt changes over time, increasing linearly from zero to 0.5T (sec). At 0.5T (sec), the value of the coefficient αt reaches a maximum value of 1, after which it decreases linearly over time, returning to zero after T (sec). By using the characteristics shown in Fig. 5(A), it is possible to express the way in which an image at the time of shooting is continuously changed into a retouched image as a retouching simulation video M_sim.

[0039] In the graph of Fig. 5(B), the value of the coefficient αt changes over time to a minimum value of 0 or a maximum value of 1. The value of the coefficient αt is the maximum value of 1 in the period from 0 (sec) to 0.5T (sec), and the value of the coefficient αt is the minimum value of 0 in the period from 0.5T (sec) to T (sec). By using the characteristics shown in Fig. 5(B), it is possible to express the way in which images before and after retouching are switched sequentially over time as the retouching simulation video M_sim.

[0040] The retouching simulation video M_sim is repeatedly played back over time, with T relating to the change in the value of the coefficient αt being one period. The length T of one period can be arbitrarily changed by the user using the menu button 204.

[0041] The generated retouching simulation video M_sim is displayed as a list of thumbnail images, allowing the user to check the image processing effects of multiple images. An example in which a retouching score is calculated when checking a retouching simulation in a list display application has been described. However, the present invention is not limited to this, and a retouching score may be calculated when automatic retouching is performed during shooting.

[0042] The process of generating an automatically retouched image during shooting will be described with reference to the flowchart in Fig. 6. In S601, the control unit 108 analyzes the current shooting conditions and the image being displayed in live view, and determines the retouching method. The live view display is a sequential display of captured images acquired before shooting for recording. The process of determining the retouching method will be described later with reference to Fig. 7.

[0043] In S602, the control unit 108 performs processing to notify the user of what automatic retouching processing has been selected. An example of the notification processing will be described later with reference to FIG. 9. Next, in S603, the control unit 108 executes a capture process of the recorded image in conjunction with a tap of the release button (FIG. 9: 902). In S604, the image processing unit 104 and the control unit 108 perform retouching processing using the retouching method selected in S601. At this time, the target image to be retouched may be a reduced version of the recorded image, from the viewpoints of increasing the processing speed and reducing the recording capacity.

[0044] In S605, the recording unit 107 and the control unit 108 record the data of the retouched image processed in S604 as meta information of the captured image in S603, that is, the acquired image (recorded image).

[0045] The retouching method determination process will be described with reference to Fig. 7. Fig. 7 is a flowchart illustrating the retouching method determination process. In Fig. 7, a determination process is executed to select one of the following three types of image processing methods as a candidate retouching method. (1) Relighting processing, in which light from a virtual light source is shone on the subject. (2) Background blurring, which separates the foreground from the background and adds blur to the background area to recreate an image with a shallow depth of field. (3) Art filter processing that changes the overall atmosphere of the image.

[0046] First, in S701, the control unit 108 executes image capture processing by live view shooting by controlling the image sensor 102, etc. Next, in S702, the distance measurement unit 105 is started, and the control unit 108 determines whether or not the shooting mode is set in which distance information from the image capture device 100 to the subject can be acquired. If it is determined that the distance information of the subject can be acquired, the process proceeds to S703, and if it is determined that the distance information of the subject cannot be acquired, the process transitions to S708.

[0047] In S703, the control unit 108 judges whether the captured image acquired in S701 is separable into the foreground and background. For example, the control unit 108 performs a process of assigning the distance information acquired in S703 to each pixel of the captured image in S701. The control unit 108 performs a process of generating a histogram of the distance information of each pixel, and checks whether the generated histogram is bimodal. The bimodal nature is determined based on the frequency peak of the histogram. If the bimodal nature is confirmed, the control unit 108 judges that the background clusters and the foreground clusters can be separated. As a method of checking whether the histogram is bimodal, the bimodal nature of the histogram can be checked by using a method known as Silverman's test. If the bimodal nature of the histogram is confirmed, that is, if it is determined that the foreground and background can be separated, the process proceeds to S704, and if it is determined that the foreground and background cannot be separated, the process transitions to S708.

[0048] In S704, a process is performed to separate the foreground and background regions of the image captured in S701. One method for separating the regions is, for example, discriminant analysis (a method known as Otsu's binarization). With this method, binarization can be performed automatically by determining the threshold value that maximizes the degree of separation. After binarization processing of the histogram of distance information calculated in S703, processing is performed in which small clusters of distance information are treated as the foreground and large clusters of distance information are treated as the background.

[0049] In S705, the control unit 108 judges whether or not the three-dimensional shape of the foreground subject separated in S704 can be acquired. The process of acquiring the three-dimensional shape can be realized by referencing the three-dimensional shapes of various subjects recorded in the recording unit 107 (referring to the shape data) and performing fitting. Here, a case where the foreground subject is a person will be described as an example. The face detection unit 106 detects a face area from the foreground subject. Fitting is performed so that the error between the distance information corresponding to the detected face area and the three-dimensional information of the face referred to from the recording unit 107 is minimized. At this time, the error value is compared with a threshold value. If the error value is less than the threshold value, that is, if it is determined that the three-dimensional shape of the foreground subject can be acquired, the process proceeds to S706. If the error value is equal to or greater than the threshold value, the process transitions to S707.

[0050] In S706, the control unit 108 selects relighting processing. The relighting processing will be described later with reference to FIG. 8. In S707, the control unit 108 selects background blurring processing. Low-pass filter processing is performed on the background region separated in S704, and the processed image is combined with the foreground image to perform background blurring processing. For example, an average filter is used for low-pass filter processing, but a filter that reproduces the blur characteristics of a lens may also be used.

[0051] In S708, the control unit 108 selects an art filter process. The art filter process is a process for changing the impression of an image, for example, by making the image sepia-toned or by enhancing the saturation. The art filter process is well known, so a description thereof will be omitted. In S708, one of the art filter processes based on the information recorded in the recording unit 107 is performed. For example, a process for selecting the art filter process that is most frequently used by the user is performed.

[0052] Fig. 8 is a schematic explanatory diagram relating to the calculation of reflection components in relighting processing. The diagram shows the positional relationship between image capture device 100, subject 801, and virtual light source 802 during shooting, and the reflection characteristics of the virtual light source. The left-right direction in Fig. 8 is defined as the horizontal coordinate axis, and the up-down axis perpendicular to this is defined as the depth axis (axis of depth information representing distance D). Note that the vertical pixel position on the axis perpendicular to the paper surface of Fig. 8 is omitted for simplicity of explanation.

[0053] The subject 801 shown in FIG. 8 is a foreground subject separated in S704 in FIG. 7. The relighting process is realized by adding the diffuse reflection components Rd, Gd, and Bd from the virtual light source 802 set at a predetermined position to the color signals R, G, and B of the input image. In the calculation process of the diffuse reflection components Rd, Gd, and Bd, the horizontal pixel position of the image captured by the image capture device 100 is denoted as H1. The normal vector at the horizontal pixel position H1 of the subject 801 is denoted as N1, and the directional vector of the light ray from the virtual light source 802 to the subject position (the intersection position of the normal vector N1 and the subject 801) is denoted as L1. The distance between the virtual light source 802 and the subject position is denoted as K1. The directional vector of the reflected light at the subject position is denoted as S1, and the directional vector from the subject position to the image capture device 100 is denoted as V1.

[0054] The diffuse reflection component at horizontal pixel position H1 is proportional to the dot product of normal vector N1 at H1 and direction vector L1, and is inversely proportional to the square of distance K1 between virtual light source 802 and the subject position. The intensity of the diffuse reflection component due to virtual light source 802 is denoted as Pd. Pd can be expressed by the following (Equation 2), and the diffuse reflection components Rd, Gd, and Bd for each color can be expressed by the following (Equation 3).

number

number

[0055] The diffuse reflection components Rd, Gd, and Bd calculated in this manner are added to the color signals R, G, and B of the input image, respectively. As shown in the following (Equation 4), color signals R_out, G_out, and B_out after the relighting process are generated.

number

[0056] 9 is a diagram showing an example of the screen of the display unit 111 during shooting in this embodiment. An example is shown in which an icon 901 and a release button 902 are displayed on the screen. The icon 901 indicates the retouching method (e.g., a relighting processing method) selected in S601 of FIG. 6. By displaying the retouching method as the icon 901, the user can easily recognize the retouching method. Although an example of notification using an icon has been shown, notification using text display, etc. may also be adopted.

[0057] Although three types of retouching methods have been described in Fig. 7, methods of adding other image processing effects may also be included. Fig. 10 is a flowchart showing an example in which a retouching method (image processing method) using tone mapping processing is further added. The difference from Fig. 7 is that S709 and S710 have been added in Fig. 10, and only these processes will be described.

[0058] If the judgment condition of S702 is not satisfied or if the judgment condition of S703 is not satisfied, the process proceeds to S709. S709 is a judgment process of whether or not it is better to execute a tone mapping process. As a judgment method, for example, there is a method of calculating a luminance histogram of a photographed image and judging the bias of the histogram. When judging the bias of the histogram, the control unit 108 compares the variance value of the histogram with a predetermined threshold. If it is judged that the variance value of the histogram is equal to or less than the threshold, the process proceeds to S708. If it is judged that the variance value of the histogram is greater than the threshold, that is, that the histogram is biased, the process proceeds to S710. In S710, a tone mapping process is performed so that the variance value of the histogram becomes a predetermined value. The tone mapping process is well known, so a description thereof will be omitted.

[0059] As described above, in the embodiment in which another retouching process is added, the control unit 108 judges whether it is appropriate to perform the added retouching process. In other words, by adding a process for selecting and determining a retouching process to the flowchart, it is possible to select one of various types of retouching processes.

[0060] An arbitrary retouching simulation designated by the user in S306 in Fig. 3 will be described in detail with reference to Fig. 11. Fig. 11 is a flowchart for explaining internal processing of the imaging device 100 when confirming an arbitrary image processing effect designated by the user. Since the processes of S1001, S1002, S1004, and S1005 in Fig. 11 correspond to S401, S402, S404, and S405 in Fig. 4, respectively, their explanations will be omitted and only the process of S1003 will be described.

[0061] In S1003, a process is executed to generate a retouched image using the retouching method selected by the user in S305 of Fig. 3. For example, a process example in which the user selects a relighting processing method will be described.

[0062] The relighting process requires that all of the judgment conditions of S702, S703, and S705 described in Fig. 7 are satisfied. That is, the conditions for execution are that the image is shot in a shooting mode that can acquire distance information, that the foreground and background are separable, and that the three-dimensional shape of the foreground subject can be acquired. Therefore, the control unit 108 selects an image that satisfies the conditions for execution from among the images to be processed determined in S1001. The relighting process is executed only for the selected images. Images that were not selected among the images to be processed are grayed out when displayed in a list so that the user can recognize that the process could not be executed.

[0063] Although the example in which the user selects the relighting processing method has been described, the user can select another image processing method. In that case, if it is determined that the conditions for making the image retouching impossible are met, a process is performed to notify the user that the retouching process could not be performed by using an icon or a message when the image list is displayed.

[0064] Next, the process when the user checks the retouched image in detail using the image viewer application in S307 in Fig. 3 will be described. The retouched images generated in S604 in Fig. 6 or S1003 in Fig. 11 are generated in a relatively small image size for list display. Therefore, they are not suitable for checking in an image viewer. Therefore, when the image viewer is started, the retouching process is performed again in an image size equivalent to that of the image recorded at the time of shooting.

[0065] When retouching is performed again, the retouching strength is not limited to the strength of the retouching process equivalent to the automatic retouching at the time of shooting. The retouching simulation moving image changes according to the time axis shown on the horizontal axis in Fig. 5, and a process is executed to change the strength of the re-retouching (image processing) at the time the image viewer is started according to the time value at the moment when the user taps. For example, when a user taps an image retouched by relighting processing, it is assumed that the user taps at the time (T / 2) of a half period of the retouching simulation that is repeated periodically. In this case, a relighted image is generated with the strength parameter (coefficient α) shown in (Equation 2) set to α / 2.

[0066] In this embodiment, thumbnails of the retouching simulation moving images are displayed on an image list display application, so that the user can easily check the image processing effects of multiple image processing methods. According to this embodiment, it is possible to provide an imaging device that reduces the operational burden on the user when shooting and allows the user to easily check the image processing effects of multiple images when checking images on a list display application.

[0067] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0068] 101 Optical system 102 Image sensor 104 Image processing section 105 Ranging section 108 Control Unit 110 Operation section 111 Display section

Claims

1. a selection means for selecting an image from a plurality of images displayed on the display means in response to a user's operation on the operation means; a generating means for performing image processing on a first image selected by the selecting means, the image processing being selected based on an image analysis result of a live view image acquired when the first image was captured, or the image processing being selected in response to a user's operation on the operating means, to generate a second image; a control means for displaying a third image, which is an image that changes over time from one of the first image and the second image to the other, as at least one of the images displayed in a list on the display means; Equipped 13. An image processing device comprising:

2. The control means instructs the generating means to select a method from a plurality of image processing methods depending on the photographing conditions of the photographed image or the state of the subject.

2. The image processing device according to claim 1,

3. An acquisition means for acquiring distance information of a subject, The control means performs a process of determining an image processing method for the first image by determining distance information of the subject, information of foreground and background subjects based on the distance information, or a three-dimensional shape of the foreground subject.

3. The image processing device according to claim 2.

4. The control means performs a process of automatically selecting the image processing method, or selecting the image processing method corresponding to a user's operation on the operation means.

4. The image processing device according to claim 2, wherein the first and second inputs are input to the image processing apparatus.

5. The control means performs a process of notifying the determined image processing method.

5. The image processing device according to claim 4.

6. The control means causes the third image generated by the determined image processing method to be displayed as at least one of the images displayed in a list on the display means.

6. The image processing device according to claim 2, wherein the image processing device is a computer.

7. The control means controls the display means to display the third image at a preset cycle.

7. The image processing device according to claim 6,

8. The control means performs a process of changing the intensity of the image processing by the generation means in accordance with the time value at which the image is selected.

7. The image processing device according to claim 5, wherein the first and second inputs are input to the image processing apparatus.

9. The control means calculates an evaluation value for determination from the first image and the second image, and when the evaluation value is equal to or greater than a threshold value, controls the generation means to generate the third image.

9. The image processing device according to claim 1,

10. An image processing device according to any one of claims 1 to 9; An imaging means for acquiring an image.

1. An imaging device comprising:

11. The control means performs a process of determining an image processing method corresponding to a user's operation on the operation means at the time of shooting or the time of the user's operation on the operation means, and causes the third image generated by the determined image processing method to be displayed as at least one of the images displayed in a list on the display means.

11. The imaging device according to claim 10.

12. a recording means for recording the first image and the second image on a recording medium; The control means controls the recording means to record the information of the second image as meta information for the information of the first image.

12. The imaging device according to claim 10 or 11.

13. An imaging means for acquiring an image; An operating means; A display means; a selection means for selecting an image from a plurality of images displayed on the display means in response to a user's operation on the operation means; a generating means for generating a second image by performing image processing on the first image selected by the selecting means using an image processing method determined based on at least one of the photographing conditions when an image is captured by the imaging means, the state of a subject whose image is captured by the imaging means, and a user's operation on the operating means; and a control means for displaying a third image, which is an image that changes over time from one of the first image and the second image to the other, as at least one of the images displayed in a list on the display means; Equipped 1. An imaging device comprising:

14. A control method executed in an image processing device, comprising: a selection step of selecting an image from a plurality of images displayed on the display means in response to a user's operation on the operation means; a generating step of generating a second image by performing image processing on the first image selected by the selecting step, the image processing being selected based on an image analysis result of a live view image acquired when the first image was captured, or the image processing being selected in response to a user's operation on the operating means; and a control step of displaying a third image, which is an image that changes over time from one of the first image and the second image to the other, as at least one of the images displayed in a list on the display means. A control method comprising:

15. A control method executed by an imaging device having an imaging means for acquiring an image, an operation means, and a display means, comprising: a selection step of selecting an image from a plurality of images displayed on the display means in response to a user's operation on the operation means; a generating step of generating a second image by performing image processing on the first image selected by the selecting step using an image processing method determined based on at least one of the photographing conditions when an image is captured by the imaging means, the state of a subject whose image is captured by the imaging means, and a user's operation on the operation means; and a control step of displaying a third image, which is an image that changes over time from one of the first image and the second image to the other, as at least one of the images displayed in a list on the display means. A control method comprising:

16. The steps according to claim 14 or 15 are executed by a computer of an image processing device. A program characterized by:

Images