Electronic device, control method, and computer-readable medium

Abstract

The present invention provides an electronic device, a control method, and a computer-readable medium. The electronic device according to the present invention includes: an acquisition unit configured to acquire position information indicating the position of a subject in a captured image; a display control unit configured to control the display unit to display an item having a length in a first direction corresponding to a range in a depth direction of the image, and to display a graphic indicating the presence of the subject in association with a position in the item corresponding to the position information; a receiving unit configured to be able to receive an operation for specifying a setting range that is at least a part of the item; and a processing unit configured to perform predetermined processing based on the setting range.

Description

Technical Field

[0001] The present invention relates to an electronic device, and in particular, to processing (control) using at least a part of a range in a depth direction of a captured image. Background Art

[0002] In recent years, smartphones have been performing image processing such as background blurring on images that have already been captured (recorded). In background blurring, by blurring subjects other than the main subject (primary subject), that is, blurring the subject serving as the background, an image similar to an image with a shallow depth of field (blurred background) captured by a digital single-lens (reflex) camera can be generated.

[0003] Background blurring blurs objects other than the main subject, necessitating main subject detection to detect the main subject. This process distinguishes the main subject from other objects based on image features. However, depending on the image features, there are cases where subjects are incorrectly identified or their edges are incorrectly detected. Furthermore, if the main subject is not accurately detected, it may even be blurred, resulting in an image in which background blurring fails.

[0004] For example, in an image of a person who is located far away from a smartphone (the smartphone photographs the person), the size of the person in the image is small, so there may be a situation where the person is mistakenly identified (recognized) as part of the background rather than the main subject, and an image in which the person is blurred is generated.

[0005] Japanese Patent Laid-Open No. 2009-278623 discloses a technique in which a user specifies a specific element (subject) on a displayed image, a region of the specific element is automatically determined by segmentation, and image editing processing is performed on the determined region.

[0006] According to the technology disclosed in Japanese Patent Application Laid-Open No. 2009-278623, the region of the subject specified by the user is automatically determined by segmentation based on the feature quantity of the image. However, in some cases, such as when the feature quantity of the subject specified by the user is similar to that of its background, the region intended by the user may not necessarily be determined. Summary of the Invention

[0007] The present invention provides a technology that allows a user to easily designate a desired subject area so that desired processing (control) is performed.

[0008] According to an electronic device of the present invention, the device includes: an acquisition unit configured to acquire position information indicating the position of a subject in a captured image; a display control unit configured to control so that an item having a length in a first direction is displayed in the display unit, and a graphic indicating the presence of the subject is displayed in association with a position in the item corresponding to the position information, wherein the length in the first direction corresponds to a range in the depth direction in the image; a receiving unit configured to be capable of receiving an operation for specifying a setting range that is at least a part of the item; and a processing unit configured to perform predetermined processing based on the setting range.

[0009] According to the present invention, a control method of an electronic device includes: acquiring position information for indicating the position of a subject in a captured image; controlling so that an item having a length in a first direction is displayed in a display unit, and a graphic for indicating the presence of the subject is displayed in association with a position in the item corresponding to the position information, wherein the length in the first direction corresponds to a range in the depth direction in the image; receiving an operation for specifying a setting range that is at least a part of the item; and performing predetermined processing based on the setting range.

[0010] According to the present invention, a computer-readable medium stores a program for causing a computer to function as each unit of the above-mentioned electronic device.

[0011] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A and Figure 1B is an external view of a smart phone according to Example 1;

[0013] Figure 2 is a block diagram showing an example of the configuration of a smartphone according to Embodiment 1;

[0014] Figure 3 is a diagram showing an example of a menu screen according to Embodiment 1;

[0015] Figure 4 is a flowchart of a shooting process according to Example 1;

[0016] Figure 5 is a diagram showing an example of display according to Embodiment 1;

[0017] Figure 6 is a diagram showing an example of a positional relationship between a smartphone and various subjects according to Embodiment 1;

[0018] Figure 7A and Figure 7B is a diagram showing an example of a method for generating a blur intensity map according to Embodiment 1;

[0019] Figure 8A and Figure 8B is a diagram showing an example of a method for generating a blur intensity map according to Embodiment 1;

[0020] Figure 9A and Figure 9B Each is a diagram showing an example of a smoothing filter according to Embodiment 1;

[0021] Figure 10A and Figure 10B Each is a diagram showing an example of a result of blurring processing according to Embodiment 1;

[0022] Figures 11A to 11C Each is a diagram showing a method for specifying a setting range as a modification of Embodiment 1;

[0023] Figure 12 is a flowchart of a shooting process according to Example 2;

[0024] Figure 13 is a flowchart of the fuzzy adjustment process according to Example 2;

[0025] Figure 14A and Figure 14B Each is a diagram showing an example of display according to Embodiment 2; and

[0026] 15A to 15D 3 is a diagram showing an example of region extraction processing as a modification of Embodiment 2. DETAILED DESCRIPTION

[0027] Example 1

[0028] Hereinafter, Example 1 of the present invention will be described. Figure 1A and Figure 1B The appearance of a smartphone (smartphone with a camera; cellular phone unit) 100 serving as an example of an electronic device to which the present invention is applied is shown. Figure 1A is a front perspective view of the smartphone 100, and Figure 1B It is a rear perspective view of the smartphone 100 .

[0029] The smartphone 100 has a vertically long shape. A touchscreen display 102, a button 103D, a front camera 104, an illuminance sensor 105, and a speaker 106 are arranged on the front face (front surface) 101A of the smartphone 100. A rear camera 108 and a light 109 are arranged on the back face (rear surface) 101B of the smartphone 100. Buttons 103A to 103C and a connector 107 are arranged on the side face (side surface) 101C of the smartphone 100.

[0030] Button 103A is a power button, and by pressing button 103A, smartphone 100's sleep mode can be set or canceled. Furthermore, smartphone 100 can be turned on or off by pressing and holding button 103A. Button 103B is a shutter button, and is located in a position where it is easily pressed when smartphone 100 is held horizontally. A shutter button (an item having the function of a shutter button) may be displayed on touchscreen display 102. Button 103C is a mute button, and is used to switch the output of sounds such as shutter sounds and warning sounds from speaker 106 on (ringing) or off (silent). Button 103D is a home button, and is used when the home screen is displayed on touchscreen display 102. Sleep mode of smartphone 100 can also be canceled by using button 103D instead of button 103A. The user can assign any function to button 103D. In the following description, there are cases where a button is not recognized as one of the buttons 103A to 103D, but the buttons 103A to 103D are collectively referred to as the button 103 capable of receiving an operation by a user.

[0031] Touchscreen display 102 includes a liquid crystal display 102A and a touchscreen 102B. Touchscreen 102B detects contact with touchscreen 102B by an operating object such as a finger or a stylus. Furthermore, touchscreen 102B can detect contact with multiple portions of touchscreen 102B, for example, contact with multiple fingers. Touchscreen 102B uses a capacitance method or a resistive film method, for example, as a detection method.

[0032] Smartphone 100 determines the type of gesture operation based on the contact, contact position, and contact duration detected by touchscreen 102B. A gesture operation is an operation performed on touchscreen 102B by an operator. Examples of gesture operations determined by smartphone 100 (gesture operations that can be received by smartphone 100) include touch, release, tap, drag, swipe, flick, pinch-in, and pinch-out. Smartphone 100 performs processing (control) corresponding to the gesture operation performed on touchscreen 102B.

[0033] Touch is a gesture operation in which an operating body comes into contact with the touch screen 102B. Release is a gesture operation in which an operating body moves away from the touch screen 102B. Tap is a gesture operation in which a touch is made on the touch screen 102B and the operating body is quickly released from the touch screen 102B. Drag is a gesture operation in which a long press is made on an object displayed on the liquid crystal display 102A and the object is dragged while the operating body is in contact with the touch screen 102B. Sweep is a gesture operation in which an operating body is moved on the touch screen 102B while the operating body is in contact with the touch screen 102B so as to slide on the touch screen 102B. A linear sweep can be regarded as a slide. Flick is a gesture operation in which a touch is made on the touch screen 102B and the operating body is released from the touch screen 102B so as to sweep across the touch screen 102B. Pinch is a gesture operation in which multiple operating bodies (multiple fingers) are moved on the touch screen 102B so as to pinch (reduce the space between the fingers) the touch screen 102B. Spread is a gesture operation in which multiple operating bodies are moved on the touch screen 102B so as to spread out the multiple operating bodies.

[0034] Figure 2 is a block diagram showing an example of the configuration of the smartphone 100 .

[0035] As described above, the smartphone 100 includes the touch screen display 102, and the touch screen display 102 includes the liquid crystal display 102A and the touch screen 102B. The liquid crystal display 102A displays letters, images, and icons according to display control of the controller 14. The touch screen 102B detects gesture operations.

[0036] Front camera 104 includes a lens and an imaging device such as a CCD or CMOS that converts optical images into electrical signals. Front camera 104 is a small camera module that includes autofocus (AF), aperture, and shutter speed adjustment functions. Front camera 104 captures an image of a subject facing front 101A.

[0037] The illuminance sensor 105 detects the brightness around the smartphone 100, specifically, the illuminance of light (ambient light; natural light) applied to the smartphone 100. The controller 14 sets imaging conditions (imaging parameters) for the front camera 104 and the rear camera 108, and adjusts the brightness of the liquid crystal display 102A based on the detected illuminance.

[0038] When the output of sound is set to "ON" (ringing), the speaker 106 outputs sounds such as a shutter sound and a warning sound. For example, when shooting is performed, the speaker 106 outputs a shutter sound.

[0039] Connector 107 is used to connect smartphone 100 to external devices. For example, an AC adapter for charging the battery in power module 11 is connected to connector 107. Connector 107 is also used when inputting and outputting image and audio data using storage device 15. Connector 107 may be a terminal specifically designed for smartphone 100, such as a dock connector, or a general-purpose terminal such as a universal serial bus (USB).

[0040] The rear camera 108 is a small camera module similar to the front camera 104. The rear camera 108 captures an image of a subject facing the rear surface 101B. The light 109 is a light emitting module and functions as a flash when the rear camera 108 captures an image.

[0041] The communication module 10 communicates according to a predetermined wireless communication standard. For example, the communication module 10 supports at least one of the IEEE 802.11 standard (so-called Wi-Fi), Bluetooth (registered trademark), and NFC. The communication module 10 is used when inputting and outputting image data obtained by imaging and downloading a function-adding program module to the smartphone 100.

[0042] The power module 11 includes a rechargeable battery and supplies power (electricity) to the entire smartphone 100. As the battery provided in the power module 11, for example, a lithium-ion battery or a nickel metal hydride battery is used. The battery is charged by an AC adapter or another external device connected by USB via the connector 107.

[0043] The acceleration sensor 12 detects the direction and magnitude of acceleration acting on the smartphone 100. The acceleration sensor 12 can detect the magnitude of acceleration in each of three axial directions: X, Y, and Z. The gyro sensor 13 detects the angle (posture) and angular velocity of the smartphone 100. The depth sensor 17 measures the distance from a camera (front camera 104 or rear camera 108) to a subject to be photographed. Methods for measuring distance include emitting infrared light, light, or ultrasonic waves, reflecting them from a subject, and measuring the time required for the emitted infrared light, light, or ultrasonic waves to reflect back to the point from which they were emitted. Alternatively, an image sensor used in a camera or the like can be used as the depth sensor 17. In this case, one method involves providing multiple cameras arranged in parallel and acquiring (calculating) the distance to the subject based on multiple captured images (parallax images). Alternatively, the distance can be acquired by providing microlenses in the pixel portion of the image sensor and acquiring multiple parallax images. The image sensor at this time may be independently provided as the depth sensor 17, or the image sensor of the rear camera 108 may be used as the depth sensor 17. In particular, some smartphones produced in recent years have multiple cameras, and in this case, the distance can be acquired by using images captured by the multiple cameras.

[0044] The controller 14 is a processor such as a general-purpose CPU or a single-chip system (system on chip (SoC)) designed for the smartphone 100. The controller 14 controls the entire smartphone 100. The controller 14 implements processing related to the flowchart described later by loading a control program 15A stored in the storage device 15 into the system memory 16 and executing the control program 15A. The system memory 16 also serves as a work area for the controller 14. The system memory 16 is, for example, a RAM.

[0045] The storage device 15 is a semiconductor memory and is, for example, an electrically erasable / recordable nonvolatile memory (flash ROM, etc.). In the storage device 15, a control program 15A and various data are stored (recorded). Figure 2 In the embodiment of the present invention, a control program 15A, an image editing module 15B, setting data 15C, image data 15D, and an additional function module 15E are stored in the storage device 15. The control program 15A and various data stored in the storage device 15 can be input through wireless communication of the communication module 10 or can also be input via the connector 107. The storage device 15 may or may not be detachable from the smartphone 100.

[0046] The control program 15A provides a function for controlling the entire smartphone 100. That is, the controller 14 controls the Figure 2. For example, controller 14 provides a camera function by controlling touchscreen display 102, button 103, front camera 104, illuminance sensor 105, rear camera 108, and light 109 according to control program 15A. Furthermore, controller 14 executes processing based on gesture operations detected by touchscreen 102B according to control program 15A and displays information corresponding to the results of the processing on liquid crystal display 102A. Control program 15A can be used in conjunction with other programs (modules) stored in storage device 15.

[0047] The image editing module 15B is a module that provides functions for editing and managing image data obtained by imaging. The setting data 15C is data related to various operations (various settings) in the smartphone 100, and is data related to, for example, the settings of various modules. The image data 15D includes image data captured by the front camera 104 and image data captured by the rear camera 108. The additional function module 15E is a module that provides functions added to the smartphone 100. For example, the additional function module 15E provides a browser function for displaying web pages.

[0048] Figure 3 2 is a diagram showing an example of a menu screen displayed on the liquid crystal display 102A. Figure 3 The menu screen in has a menu area 301 and a status display area 302.

[0049] The menu area 301 is an area for the user to select a function to be executed from a plurality of functions of the smartphone 100. In the menu area 301, a plurality of icons 306 are displayed, and the plurality of icons 306 are associated with a plurality of functions of the smartphone 100. When the controller 14 detects a tap on an icon 306, the controller 14 executes the function associated with the icon 306. Figure 3 Eight icons 306 associated with eight functions, namely, calendar, clock, browse, camera mode, settings, video capture, picture, and camera, are displayed.

[0050] The status display area 302 is an area for displaying various statuses of the smartphone 100. Figure 3 , a status display area 302 is located above the menu area 301, and a horizontal icon 303 indicating Wi-Fi strength, time 304, and a battery icon 305 indicating the remaining battery level are displayed in the status display area 302. The status display area 302 also serves as an area for displaying various notification messages.

[0051] Figure 41 is a flowchart showing details of a photographing process (photographing process including image editing process) performed in the smartphone 100. The controller 14 loads the control program 15A stored in the storage device 15 into the system memory 16 and executes the control program 15A, and thereby realizes the above-mentioned process. Figure 3 When the icon 306 associated with the camera function (camera function) among the multiple icons 306 in the image is tapped, the camera function is started. Figure 4 in the processing.

[0052] In S401, the controller 14 activates the camera function and starts capturing images. Although the front camera 104 can be used to capture images, the rear camera 108 can also be used to capture images. In addition, the controller 14 drives the depth sensor 17 to start acquiring depth information indicating the depth range (range in the depth direction) of the captured image and the distance from the smartphone 100 to the subject (position information indicating the position of the subject in the captured image).

[0053] In S402, the controller 14 displays on the liquid crystal display 102A an image captured by the rear camera 108. In S403, the controller 14 displays a blur adjustment UI on the liquid crystal display 102A.

[0054] Figure 5 1 is a diagram showing an example of displaying a captured image and a blur adjustment UI. In area 501, an image captured by the rear camera 108 is displayed. The blur adjustment UI includes a depth map (depth bar) 502, boundary adjustment sliders 503 and 505, and subject icons 521 to 523.

[0055] Depth map 502 is an item (graphic) displayed based on depth information, and its horizontal length corresponds to the depth range of the captured image (the image displayed in area 501). Therefore, depth map 502 indicates the depth range in the captured image (the image displayed in area 501). In depth map 502, the distance (depth) from smartphone 100 is expressed using brightness. Specifically, in depth map 502, brightness varies depending on the horizontal position, so that depth map 502 becomes darker as depth decreases and becomes brighter as depth increases. Note that in depth map 502, it is only necessary to visually indicate depth, and therefore, at least one of brightness or color may vary depending on, for example, the horizontal position. Specifically, in depth map 502, low depth can be expressed using blue, medium depth can be expressed using purple, and high depth can be expressed using red.

[0056] Each of the subject icons 521 to 523 is a graphic (icon; item) displayed based on the position information of the subject. Each of the subject icons 521 to 523 indicates the presence of the subject in the captured image (the image displayed in the area 501). Specifically, the subject icon 521 indicates the presence of the subject 511, the subject icon 522 indicates the presence of the subject 512, and the subject icon 523 indicates the presence of the subject 513. Figure 5 In the present embodiment, based on the position information (position in the depth direction) of the corresponding subject, each of the subject icons 521 to 523 is displayed at a position (corresponding position) corresponding to the position information in the depth map 502. Therefore, the blur adjustment UI can be said to be information in which the depth information of the captured image and the position information of the subject are associated with each other.

[0057] By viewing the depth map 502 and the subject icons 521 to 523 , the user can easily understand that, among the subjects 511 to 513 , the subject 513 is closest to the smartphone 100 , and the subject 512 is farthest from the smartphone 100 .

[0058] In Embodiment 1, it is assumed that the controller 14 performs main subject detection processing by using the image editing module 15B, and displays the face icon of the detected main subject (main subject) as the subject icon. Figure 5 , the background subject 514 is not detected as the main subject, and therefore the subject icon of the subject 514 is not displayed, but a subject icon corresponding to the background subject may also be displayed.

[0059] Note that the subject icon does not necessarily have to be a face icon (face image), as long as the corresponding subject can be determined, and can include a character string and an arrow. In addition, the main subject does not have to be a person. The subject icon of each subject can be displayed without performing the main subject detection process. The subject icon only needs to be displayed in association with the corresponding position in the depth map 502 (in the item), and the subject icon does not need to be displayed in the depth map 502. For example, the subject icon can be displayed outside the depth map 502 so as to indicate the corresponding position.

[0060] Each of the boundary adjustment sliders 503 and 505 is an item (graphic) for the user to specify a setting range (which is at least a portion of the depth map 502). In Example 1, the range in the horizontal direction is specified as the setting range by using the boundary adjustment sliders 503 and 505. Therefore, it can be said that the setting range is at least a portion of the depth range of the captured image (the image displayed in the area 501). Specifically, the positions (positions in the horizontal direction) of the boundary adjustment sliders 503 and 505 can be changed by operations such as dragging, sweeping, and sliding. In the area from the boundary adjustment slider 503 to the boundary adjustment slider 505, a setting range bar (graphic; item) 504 occupying the area is displayed. The range in the depth map 502 that matches the range of the setting range bar 504 (the range in the horizontal direction) is used as the setting range.

[0061] In Example 1, the depth range serving as the range of the main subject is specified by using boundary adjustment sliders 503 and 505, and the area in the captured image (the image displayed in area 501) that does not correspond to the specified depth range (set range) is blurred by blur processing.

[0062] Figure 6 is a diagram showing an example of the positional relationship between the smartphone 100 and various subjects. Figure 5 The depth range indicated by the depth map 502 in FIG. 5 is a range from the position 611 of the smartphone 100 to the deepest position 615. The subject 601 and the subject 603 included in the depth range from the position 611 to the position 613 of the smartphone 100 are respectively Figure 5 The subject 511 and the subject 513 in the image correspond to each other. The subject 602 included in the depth range from the position 613 to the position 614 corresponds to the subject 511 and the subject 513 in the image. Figure 5 The subject 512 in the image corresponds to the image 604, and the subject 604 included in the depth range from the position 614 to the position 615 corresponds to the image 604 in the image 604. Figure 5 Corresponding to the subject 514 in .

[0063] Depend on Figure 5 The depth range (setting range) indicated by the boundary adjustment sliders 503 and 505 (setting range bar 504) in FIG is the depth range from the position 611 to the position 613. Therefore, the subjects 601 and 603 ( Figure 5 In other words, the depth range for blurring is set so that blurring is not performed on the regions of the subjects 601 and 603 in the captured image.

[0064] Return to Figure 4In S404, the controller 14 determines whether the non-blur range is changed by operating the blur adjustment UI. The non-blur range is a depth range in which blur processing is not applied and is determined by Figure 5 The depth range (setting range) indicated by the boundary adjustment sliders 503 and 505 (setting range bar 504) in FIG.

[0065] In S405, the controller 14 generates a blur intensity map based on the unchanged unblurred range. If the unblurred range has not been changed even once and the process in S405 is performed, the blur intensity map is generated based on the default unblurred range. If the unblurred range has been changed and the process in S405 is then performed without additionally changing the unblurred range, the blur intensity map is generated based on the unblurred range immediately after the change. The blur intensity map is a map indicating the blur intensity (intensity of blur processing; blur degree) of each area in the captured image. In Example 1, the blur intensity map is generated so that the blur intensity is low in areas corresponding to the unblurred range, and the blur intensity is high in areas not corresponding to the unblurred range.

[0066] In S406 , the controller 14 generates a blur intensity map based on the changed non-blur range.

[0067] Figure 7A and Figure 7B 4 is a diagram showing an example of the processing in S405. Specifically, Figure 7A and Figure 7B A method for generating a blur intensity map based on a default unblurred range is shown. Figure 7A The correspondence relationship between the depth (distance from the smartphone 100 ) and the blur strength is shown by using a graph. Figure 7A The horizontal axis of the graph in indicates depth, which increases to the right in the graph, and Figure 7A The vertical axis of the graph in indicates the blur intensity, which increases in the upper side of the graph. Figure 7A A depth map, a subject icon, and a setting range bar are shown in association with this figure. Figure 7B A blur intensity map is shown. Figure 7B The blur intensity is shown so that the blur intensity graph becomes darker as the blur intensity becomes higher. Here, it is assumed that the captured image is Figure 5 The same image is displayed in the area 501 in FIG. Figure 7A Assume the default settings of the range bar and Figure 5In the depth range (non-blurred range) indicated by the setting range bar 504, subjects 511 and 513 corresponding to subject icons 521 and 523 exist, but subject 512 corresponding to subject icon 522 does not exist.

[0068] like Figure 7A As shown in FIG, in the case where the range bar 504 is set, the blur intensity is set to low (non-blurred) at the depths of the subjects 511 and 513 corresponding to the subject icons 521 and 523. The blur intensity is set to medium at the depth of the subject 512 corresponding to the subject icon 522. In addition, the blur intensity is set to high at a larger depth (the depth of the subject 514, etc.). As a result, Figure 7B Blur intensity map in . Figure 7B Regions 701 and 703 in FIG. 7 are regions of the subjects 511 and 513, and the blur intensity is set to low in regions 701 and 703. Region 702 is a region of the subject 512, and the blur intensity is set to medium in region 702. Furthermore, in the remaining regions, the blur intensity is set to high.

[0069] Figure 8A and Figure 8B is a diagram showing an example of the processing in S406. Figure 7A similar, Figure 8A Shows a graph (correspondence between depth and blur strength), a depth map, a subject icon, and a setting range bar. Figure 7B similar, Figure 8B Here, it is assumed that the captured image is Figure 5 The same image is displayed in the area 501 in FIG. Figure 8A As shown in , it is assumed that the range bar (non-ambiguous range) is set from Figure 5 504 is expanded to a setting range bar 804. In the depth range (non-blurred range) indicated by the setting range bar 804, not only the subjects 511 and 513 corresponding to the subject icons 521 and 523 but also the subject 512 corresponding to the subject icon 522 exists.

[0070] like Figure 8A As shown in FIG, in the case where the range bar 804 is set, the blur intensity is set to low (non-blurred) at the depths of the subjects 511 and 513 corresponding to the subject icons 521 and 523. The blur intensity is also set to low at the depth of the subject 512 corresponding to the subject icon 522. In addition, the blur intensity is set to high at a larger depth (the depth of the subject 514, etc.). As a result, Figure 8B The blur intensity map in . Figure 7ASimilarly, the blur intensity is set to low in the areas 701 and 703 of the subjects 511 and 513. Figure 7A Different from the above, the blur intensity is also set to low in the area 802 of the subject 512. Figure 7A Similarly, in the remaining areas, set the blur strength to high.

[0071] Notice, Figure 7A and Figure 8A Each shows an example in which the blur intensity increases discontinuously (step by step) in response to a continuous increase in depth, but the blur intensity may increase continuously in response to a continuous increase in depth.

[0072] Return to Figure 4 , in S407, the controller 14 determines whether the shutter button of the smartphone 100 has been pressed. In a case where the shutter button has been pressed, the process proceeds to S408, and in a case where the shutter button has not been pressed, the process proceeds to S402.

[0073] In S408, the controller 14 starts the main exposure process for capturing an image to be recorded in the storage device 15. In S409, the controller 14 ends the main exposure process and records the captured image in the storage device 15. In S410, the controller 14 uses the blur intensity map generated in S405 or S406 to blur the image recorded in S409. Note that S409 and S410 can be interchanged, and the captured image can be blurred and the blurred image can be recorded.

[0074] In the blurring process, for example, a filter process having a blurring strength shown in the blurring strength map is performed. The blurring strength of the filter process is changed by changing, for example, the filter size of the smoothing filter used in the filter process. Figure 9A A smoothing filter having coefficients (filter coefficients) of three rows and three columns is shown as an example of a smoothing filter of a medium blur intensity. Figure 9B A smoothing filter having five rows and five columns of coefficients (filter coefficients) is shown as an example of a smoothing filter with high blur strength. For each area in the captured image, a blur strength indicated by the blur strength map is selected. Figure 9A Smoothing filter in Figure 9B, and filtering processing using the selected smoothing filter is performed. Here, it is assumed that the filtering processing is not performed on the area with low blur intensity. Thus, the blur corresponding to the blur intensity indicated by the blur intensity map can be added to the captured image. In the filtering processing, for example, according to the following formula 1, the pixel value I (x, y) of the position ((position in the horizontal direction, position in the vertical direction) = (x, y)) in the captured image is converted into the pixel value I' (x, y). In formula 1, the variable m is the filter size in the vertical direction (the number of rows of filter coefficients), the variable n is the filter size in the horizontal direction (the number of columns of filter coefficients), and the variable h (i, j) is the filter coefficient of the i-th row and the j-th column.

[0075] [Mathematical formula 1]

[0076]

[0077] Figure 10A Shown by taking an image ( Figure 5 The image displayed in area 501 in the Figure 7B Example of the result obtained by blurring the blur intensity map in . Figure 10B Shown by taking an image ( Figure 5 The image displayed in area 501 in the Figure 8B Example of the result obtained by blurring the blur intensity map in . Figure 10A The subject 1001 and Figure 10B The subjects 1002 in Figure 5 , but the blur intensity of the subject 1001 is different from that of the subject 1002. Figure 7B The blur intensity in the region 702 (region of the subject 512) is set to medium, and thus Figure 10A The subject 1001 in is blurred with blur intensity = "medium". Figure 8B The blur intensity in the region 802 (region of the subject 512) is set to low, and thus Figure 10B The subject 1002 in FIG. 1 is not blurred. Therefore, by adjusting the blur intensity map, the blur intensity of the subject can be adjusted individually.

[0078] Return to Figure 4As described above, in S411, the controller 14 determines whether a shooting end instruction (shooting end operation) has been issued. If the shooting end instruction has been issued, the shooting process ends. If the shooting end instruction has not been issued, the shooting process proceeds to S402 for the next shooting. For example, if the home button or the power button is pressed, the controller 14 determines that the shooting end instruction has been issued and ends the shooting process.

[0079] As described so far, according to Embodiment 1, the user can easily designate the region of a desired subject by using the blur adjustment UI so that desired blur processing (desired processing; desired control) is performed.

[0080] Note that when the unblurring range (setting range) is specified, the user can change imaging parameters (camera settings) such as exposure (aperture value and shutter speed). Before shooting, not only the unblurring range (the depth range used as the range of the main subject) but also the exposure can be adjusted, thereby obtaining a more appropriate image.

[0081] Furthermore, while the unblurred range can be set (set range) by moving the boundary adjustment slider by an amount equal to the amount of operation, it is also possible to perform processing (control) to assist in specifying the unblurred range by using the subject's positional information. This facilitates specifying the unblurred range (improving operability).

[0082] For example, even when the operation amount of the operation for expanding the setting range in the direction in which the subject exists is smaller than the operation amount of the operation for expanding the setting range to include the subject, the operation for expanding the setting range in the direction in which the subject exists may be received as the operation for expanding the setting range to include the subject. That is, when the controller 14 detects movement of the boundary adjustment slider in the direction in which the setting range is expanded (expansion direction), the controller 14 may automatically expand the setting range so that the subject that is located on the expansion direction side relative to the setting range before expansion and is closest to the setting range before expansion is included in the setting range.

[0083] Figure 11A Shows an example of expanding the setting range by moving the boundary adjustment slider. Figure 11A The left side of the figure shows the state before expansion, and Figure 11A The right side of the figure shows the expanded state. The setting range before expansion includes the subject icons 521 and 523, but does not include the subject icon 522. Figure 11AAs shown on the left side of FIG, it is assumed that the user has moved the boundary adjustment slider 505 to the right to expand the setting range. When the controller 14 detects this operation (movement of the boundary adjustment slider 505), the controller 14 detects (searches for) a subject that exists on the expansion direction side relative to the setting range before expansion and is closest to the setting range before expansion by using the position information of the subject. Here, the expansion direction is a direction corresponding to the movement direction of the boundary adjustment slider 505, and is a direction from the near side toward the far side. Then, the subject 512 ( Figure 5 ). Thereafter, the controller 14 automatically moves (jumps) the boundary adjustment slider 505 to expand the setting range so that the detected subject 512 (subject icon 522) is included in the setting range ( Figure 11A on the right side of the ).

[0084] Similarly, when the controller 14 detects movement of the boundary adjustment slider in the direction of narrowing the setting range, the controller 14 can automatically narrow the setting range so that the subject included in the setting range and closest to the narrowed side of the setting range is excluded.

[0085] An operation for specifying a subject icon can be received as an operation for expanding the setting range to a range including the subject icon (the subject corresponding to the subject icon), so that the setting range can be specified more intuitively.

[0086] Figure 11B An example is shown in which the setting range is expanded by designating a subject icon. Figure 11B The left side of the figure shows the state before expansion, and Figure 11B The right side of the figure shows the expanded state. The setting range before expansion includes the subject icons 521 and 523, but does not include the subject icon 522. Figure 11B As shown on the left side of FIG, it is assumed that the user has touched the subject icon 522 so that the subject icon 522 is included in the setting range. When the controller 14 detects this operation (the touch on the subject icon 522), the controller 14 automatically moves (jumps) the boundary adjustment slider 505 to expand the setting range so that the subject icon 522 is included in the setting range ( Figure 11B on the right side of the ).

[0087] Similarly, an operation for specifying a subject icon included in the setting range may be received as an operation for narrowing the setting range to a range excluding the subject icon.

[0088] While the example of automatically displaying the subject icon from the start when the blur adjustment UI is displayed has been described, the subject icon does not need to be displayed from the start. For example, a subject icon may be displayed in response to an operation for specifying a subject on the captured and displayed image. In this case, the setting range may be automatically expanded to include the specified subject (displayed subject icon).

[0089] Figure 11C An example is shown in which the setting range is expanded by specifying a subject on a captured image. Figure 11C The left side of the figure shows the state before expansion, and Figure 11C The right side of the figure shows the expanded state. Before the expansion, the subject icons 521 and 523 corresponding to the subjects 511 and 513 are displayed, but the subject icon 522 corresponding to the subject 512 is not displayed. In addition, the range including the subject icons 521 and 523 is set as the setting range. Here, as shown in FIG. Figure 11C As shown on the left side of FIG, it is assumed that the user has touched the subject 512 on the captured image so that the subject 512 (subject icon 522) is included in the set range. When the controller 14 detects this operation (the touch on the subject 512), the controller 14 performs subject detection processing in the area including the touched position to detect the subject 512. Subsequently, the controller 14 displays the subject icon 522 ( Figure 11C In addition, the controller 14 automatically moves (jumps) the boundary adjustment slider 505 to expand the setting range so that the subject icon 522 is included in the setting range ( Figure 11C on the right side of the ).

[0090] While the example of specifying a non-blurred range as the set range has been described, that is, an example in which the area corresponding to the set range is not blurred, this specification is not limited to this. For example, the user can specify a blur range (a depth range to which blurring is applied) as the set range. That is, the area corresponding to the set range can be blurred, while areas not corresponding to the set range can be not blurred.

[0091] In addition, the operation for specifying the setting range does not necessarily have to be a touch operation on the touch screen. For example, the setting range can be specified by operating a physical button provided in a smart phone (electronic device) or using an external device such as a mouse. It is only required that the electronic device to which the present invention is applied has a function of controlling the display of the display and a function of performing predetermined processing based on the setting range, and the electronic device does not need to include a display or a camera. At least one of the display or the camera may be an external device of the electronic device to which the present invention is applied.

[0092] Example 2

[0093] Hereinafter, a second embodiment of the present invention will be described. In the second embodiment, an example of adjusting the non-blur range after shooting and performing blur processing will be described. Note that the appearance and configuration of the smartphone according to the second embodiment are different from those of the first embodiment ( Figure 1A , Figure 1B and Figure 2 ) are the same in appearance and configuration, and therefore their description will be omitted.

[0094] Figure 12 1 is a flowchart showing details of a photographing process (photographing process excluding image editing process) performed in the smartphone 100 according to Embodiment 2. The controller 14 loads the control program 15A stored in the storage device 15 into the system memory 16 and executes the control program 15A, and thereby realizes the above-mentioned process. For example, in Figure 3 When the icon 306 associated with the camera function (camera function) among the multiple icons 306 in the image is tapped, the camera function is started. Figure 12 in the processing.

[0095] In S1201, with Figure 4 Similar to S401 in FIG. 1 , the controller 14 activates the camera function and starts shooting with the rear camera 108. Figure 4 Similar to S401 in the previous example, the controller 14 begins acquiring depth information of the captured image and position information of the subject. In the second embodiment, the controller 14 acquires information indicating not only the position in the depth direction but also the position in a specific direction perpendicular to the depth direction as the position information of the subject. The specific direction is not particularly limited, and in the second embodiment, it is assumed that the specific direction is the horizontal direction (left-to-right direction) of the captured image.

[0096] In S1202 , similar to S402 , the controller 14 displays the image captured by the rear camera 108 on the liquid crystal display 102A.

[0097] In S1203, similarly to S407, the controller 14 determines whether the shutter button of the smartphone 100 has been pressed. In a case where the shutter button has been pressed, the process proceeds to S1204, and in a case where the shutter button has not been pressed, the process proceeds to S1202.

[0098] In S1204, similar to S408, the controller 14 starts the main exposure process for capturing an image to be recorded in the storage device 15. In S1205, similar to S409, the controller 14 ends the main exposure process and records the captured image in the storage device 15. At this time, the depth information and position information of the subject are also recorded in the storage device 15 in association with the captured image.

[0099] Figure 13 1 is a flowchart showing details of the blur adjustment process (image editing process) performed in the smartphone 100. The controller 14 loads the control program 15A stored in the storage device 15 into the system memory 16 and executes the control program 15A, and thereby realizes the above-mentioned process. Figure 3 When an icon 306 corresponding to the picture function (a browse / edit function for browsing and editing captured (recorded) images) is tapped among the multiple icons 306, a screen for that function (browse / edit screen) is displayed on the liquid crystal display 102A. Then, when the user selects an image to be processed (blur adjustment processing target) from the multiple captured (recorded) images, the selected image is displayed on the browse / edit screen. Thereafter, when the user issues an instruction (operation) for executing blur adjustment processing, the process starts. Figure 13 in the processing.

[0100] In S1301, the controller 14 reads the position information of the processing target image from the storage device 15 (via Figure 12 Then, the position information of the subject is obtained by the shooting process in Figure 4 Similar to S405 in FIG. 1 , the controller 14 generates a blur intensity map based on the read position information and the unchanged non-blur range. The unchanged non-blur range is, for example, a default blur range.

[0101] In S1302, similar to S410, the controller 14 performs blurring processing on the processing target image using the blur intensity map generated in S1301 (or S1305 described later). Thus, the image (processing target image) displayed on the browsing / editing screen is replaced with the image that has been blurred.

[0102] In S1303, similar to S403, the controller 14 displays the blur adjustment UI on the liquid crystal display 102A and prompts the user to adjust the non-blur range. Note that in Embodiment 2, the blur adjustment UI is different from the blur adjustment UI in Embodiment 1 (details will be described later).

[0103] In S1304, similar to S404, the controller 14 determines whether the non-blur range is changed by operating the blur adjustment UI. If the non-blur range is not changed, the blur adjustment process is ended, and if the non-blur range is changed, the blur adjustment process proceeds to S1305.

[0104] In S1305 , similarly to S406 , the controller 14 generates a blur intensity map based on the changed non-blur range.

[0105] Figure 14A 14 is a diagram showing an example of the display of a processing target image 1400 and a blur adjustment UI (depth map) 1420. The blur adjustment UI 1420 is an item (graphic) displayed based on depth information and image position information. The vertical length of the blur adjustment UI 1420 corresponds to the depth range in the processing target image, and the horizontal length of the blur adjustment UI 1420 corresponds to the horizontal range (the range in the horizontal direction; the range in a specific direction perpendicular to the depth direction) in the captured image. The blur adjustment UI 1420 can be said to be a diagram having a vertical axis indicating the position in the depth direction and a horizontal axis indicating the position in the horizontal direction.

[0106] When the blur adjustment UI 1420 is displayed, similar to the first embodiment, based on the position information of the subject, the subject icon is displayed at a position corresponding to the position information in the blur adjustment UI 1420. Figure 14A , subject icons 1411 to 1413 corresponding to subjects 1401 to 1403 are displayed.

[0107] By designating at least a portion of the blur adjustment UI 1420 as a non-blur range (setting range), the user can designate a range including not only a range in the depth direction but also a range in the horizontal direction as a non-blur range. Figure 14A , a non-blur range 1461 (default) is set. The subject icon 1413 is not included in the non-blur range 1461, and thus the subject 1403 corresponding to the subject icon 1413 is blurred in the image 1400 by blurring processing.

[0108] Figure 14B An example of the display after the non-ambiguous range update is shown. Specifically, Figure 14B An example of display after the user has performed an operation such as dragging, sweeping, or sliding in the direction of arrow 1441 (so that the subject 1403 corresponding to the subject icon 1413 is not blurred) is shown. Figure 14B In the example, the non-ambiguous range is changed from Figure 14AThe non-blur range 1461 in the image is expanded to the non-blur range 1431. When the controller 14 detects this operation, the controller 14 updates the blur intensity map based on the non-blur range 1431 and performs blur processing again. As a result, the processing object image is changed from Figure 14A The image 1400 in FIG. 14 is updated to the image 1410. The subject icon 1413 is included in the non-blurred range 1431, and thus the blur of the subject 1403 corresponding to the subject icon 1413 in the image 1410 is removed.

[0109] As described so far, according to Example 2, by using a two-dimensional depth map (blur adjustment UI), the user can specify the non-blur range more easily and intuitively than in Example 1 (improvement in operability). In addition, it is possible to specify a range in a specific direction perpendicular to the depth direction, and thus it is possible to allow multiple areas having the same position in the depth direction to have different blur intensities, and it is possible to obtain a more appropriate image than that of Example 1 (improvement in the accuracy of processing (control) based on the set range).

[0110] Although the example in which the photographing process and the blur adjustment process are performed separately has been described, the present invention is not limited thereto. Figure 12 After the processing in S1205 in the above, a preview screen of the captured image can be automatically displayed. Figure 13 The blur adjustment processing in makes the blur adjustment UI be displayed on the preview screen.

[0111] In addition, although an example has been described in which the blur adjustment processing is performed after shooting, the blur adjustment processing may be performed before shooting as in Example 1. The blur adjustment processing may also be performed before and after shooting. Note that before shooting, it is preferable to be able to shoot quickly, and therefore, by using the blur adjustment UI in Example 1 before shooting, it is preferable to reduce the amount of arithmetic calculations (processing load) required for the controller 14 to a level lower than that in the case of using the blur adjustment UI in Example 2. Then, after shooting, in order to allow more appropriate instructions or processing, it is preferable to use the blur adjustment UI in Example 2. The same blur adjustment UI can also be used before and after shooting.

[0112] Furthermore, the set range is not limited to the non-blurred range, and the predetermined processing based on the set range is not limited to the blurring processing. For example, the predetermined processing may include an area extraction processing for extracting an area corresponding to the set range from the image.

[0113] 15A to 15D is a diagram illustrating an example of region extraction processing. Figure 15AA captured image 1510 is shown, and the image 1510 includes subjects 1511 to 1513 . Figure 15B A range specification UI 1520 for specifying a setting range is shown. The range specification UI 1520 is an item (graphic) similar to the blur adjustment UI in the second embodiment. Figure 15B Subject icons 1521 to 1523 corresponding to the subjects 1511 to 1513 are also shown.

[0114] exist Figure 15B , a setting range 1525 including the subject icons 1521 to 1523 is designated by dragging from the start point to the end point of the arrow 1524. Based on the setting range 1525, the controller 14 generates region information indicating a region corresponding to the setting range 1525 in the image 1510. Figure 15C The generated region information 1530 is shown. Regions 1531 to 1533 indicated by the region information 1530 correspond to the subjects 1511 to 1513. Thereafter, the controller 14 extracts (crops) images of the regions 1531 to 1533 indicated by the region information 1530 from the image 1510 (region extraction information). Figure 15D An extracted image 1540 is shown. The image 1540 includes only subjects 1511 to 1513 .

[0115] The above-mentioned extracted results can be used in various image editing processes (such as lighting adjustment for adjusting the brightness of the image). For example, the controller 14 performs lighting adjustment on the image 1540 so that the brightness of the subjects 1511 to 1513 becomes uniform. The lighting adjustment includes, for example, a process for multiplying the pixel value by the gain value. Subsequently, the controller 14 replaces the subjects 1511 to 1513 in the image 1510 with the subjects 1511 to 1513 that have undergone lighting adjustment. Thus, the effect of the lighting adjustment for making the brightness of the subjects 1511 to 1513 uniform can be added to the image 1510. Therefore, by using the extracted results, lighting adjustment for adjusting the brightness of the area corresponding to the set range can be performed. That is, the predetermined processing based on the set range may include lighting adjustment for adjusting the brightness of the area corresponding to the set range.

[0116] In addition, the predetermined processing based on the set range is not limited to image processing. For example, the predetermined processing may include processing for determining the imaging parameters of a digital single-lens (reflex) camera or the like based on the set range. For example, the controller 14 may set (change) the aperture value of the lens so that the depth range corresponding to the set range is included in the depth of field. The controller 14 may set the area (subject) corresponding to the specified range as an area for automatic exposure (AE) processing and automatic white balance (AWB) processing. Thus, it is possible to capture (capture) a suitable image such as an image of a subject on which the user's attention is focused with high image quality.

[0117] Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to specific embodiments, and various forms within the scope of the present invention are also included in the present invention. In addition, the above-mentioned embodiments only illustrate exemplary embodiments of the present invention, and the embodiments may also be appropriately combined with each other.

[0118] Note that the above-described various control operations described as control operations performed by the controller 14 may be performed by one piece of hardware, or control of the entire apparatus may be performed by having a plurality of hardware (eg, a plurality of processors or circuits) share the processing.

[0119] According to the present disclosure, the user can easily designate the region of a desired subject so that desired processing (control) is performed.

[0120] Other embodiments

[0121] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0122] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An electronic device comprising: an acquisition unit configured to acquire position information indicating a position of a subject in a captured image; a display control unit configured to control so that an item having a length in a first direction corresponding to a range in a depth direction in an image is displayed on the display unit, and a graphic indicating the presence of the subject is displayed in association with a position in the item corresponding to the position information; a receiving unit configured to receive an operation for specifying a setting range as at least a part of the item; as well as a processing unit configured to perform predetermined processing based on the setting range, wherein The receiving unit is capable of receiving an operation for designating a graphic for indicating the presence of the subject as an operation for expanding the setting range to a range including the subject.

2. The electronic device according to claim 1, wherein The item indicates a position in the depth direction by at least one of brightness and color.

3. The electronic device according to claim 1 or 2, wherein: The display control unit controls so that the graphic is displayed at a position corresponding to the position information in the item.

4. The electronic device according to claim 1 or 2, wherein: The subject is a person, and The graphic is a facial image.

5. The electronic device according to claim 1 or 2, wherein: The position in the second direction perpendicular to the first direction in the item indicates the position in the second direction in the image.

6. The electronic device according to claim 1 or 2, wherein: Even in a case where the operation amount of the operation for expanding the setting range in the direction of the existence of the subject is smaller than the operation amount of the operation for expanding the setting range to a range including the subject, the receiving unit is capable of receiving the operation for expanding the setting range in the direction of the existence of the subject as the operation for expanding the setting range to a range including the subject.

7. The electronic device according to claim 1 or 2, wherein: The operation for specifying the graphic is an operation of touching the graphic.

8. The electronic device according to claim 1 or 2, wherein: The display control unit further controls so that the image is displayed on the display unit, and The receiving unit is capable of receiving an operation for touching the subject on the image as an operation for expanding the setting range to a range including the subject.

9. The electronic device according to claim 1 or 2, wherein: The predetermined processing includes processing for blurring a region in the image that does not correspond to the set range.

10. The electronic device according to claim 1 or 2, wherein: The predetermined processing includes processing for extracting a region corresponding to the set range from the image.

11. The electronic device according to claim 1 or 2, wherein: The predetermined processing includes processing for adjusting brightness of a region corresponding to the setting range in the image.

12. The electronic device according to claim 1 or 2, wherein: The predetermined processing includes processing for determining an imaging parameter based on the setting range.

13. A method for controlling an electronic device, comprising: acquiring position information indicating a position of a subject in a captured image; performing control so that an item having a length in a first direction corresponding to a range in a depth direction of an image is displayed on a display unit, and a graphic indicating the presence of the subject is displayed in association with a position in the item corresponding to the position information; receiving an operation for specifying a setting range that is at least a portion of the item; as well as Based on the setting range, predetermined processing is performed, wherein An operation for designating a graphic indicating the presence of the subject can be received as an operation for expanding the setting range to a range including the subject. 14 . A computer-readable medium storing a program, wherein when the program is executed by a processor, the program implements the steps of the electronic device control method according to claim 13 . 15 . A computer program product comprising a program, wherein when the program is executed by a processor, the program implements the steps of the method for controlling an electronic device according to claim 13 .

Images