Imaging device, control method thereof, and computer-readable medium

Abstract

The present invention relates to a photographing apparatus, a control method of the photographing apparatus, and a computer-readable medium. A photographing apparatus includes: a display unit configured to display a live image of a subject and a previously captured image on a display; a photographing unit configured to capture an image of the subject; and a recording unit configured to record the captured image of the subject captured by the photographing unit and the previously captured image in association with layout information in a recording medium, the layout information including a display position of the captured image of the subject and a display position of the previously captured image.

Description

Technical Field

[0001] The present invention relates to a photographing apparatus, a control method for the photographing apparatus, and a computer-readable medium. Background Art

[0002] In recent years, it has been possible to view a plurality of images posted on a social networking service (SNS) or the like in a list display. When the subject, composition, brightness, hue, etc. are similar among the plurality of images displayed in the list, a sense of unity and a good impression can be visually provided.

[0003] As a technique for photographing a plurality of images, Japanese Patent Application Laid-Open No. 2017-116872 discloses a technique in which a re-shot object image and a live image as a current motion image being shot are displayed side by side, allowing a user to make a pose or facial expression during photographing while checking the re-shot object image.

[0004] However, even when a live view image as a current motion image being shot is displayed side by side with a previously shot image, in order to shoot an image consistent with the previously shot image, it may take time and effort, such as re-shooting the image by changing the pose and position of the subject. Summary of the Invention

[0005] The present invention provides a photographing apparatus capable of easily generating an image group including a plurality of images having a sense of unity.

[0006] The photographing apparatus according to the present invention includes: a display unit configured to display a live image of a subject and a previously shot image on a display; a photographing unit configured to photograph an image of the subject; and a recording unit configured to record the photographed image of the subject photographed by the photographing unit and the previously shot image in a recording medium in association with layout information including a display position of the photographed image of the subject and a display position of the previously shot image.

[0007] Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0008] Figure 1 is a block diagram depicting a structural example of a photographing apparatus;

[0009] Figure 2A and 2B is a diagram illustrating an appearance of the photographing apparatus;

[0010] Figure 3 is a flowchart illustrating a photographing process according to Embodiment 1;

[0011] Figures 4A to 4CIt is a diagram for describing the arrangement of a live image and a previously captured image;

[0012] Figures 5A to 5C It is a diagram for describing the adjustment of a live image;

[0013] Figure 6 It is a flowchart illustrating the imaging process according to Embodiment 2;

[0014] Figures 7A to 7C It is a diagram for describing a screen example in the imaging process according to Embodiment 2;

[0015] Figure 8 It is a diagram illustrating the metadata indicating the image features according to Embodiment 3;

[0016] Figure 9 It is a flowchart illustrating the imaging process according to Embodiment 3;

[0017] Figure 10 It is a flowchart illustrating the process of arranging previously captured images according to Embodiment 3;

[0018] Figure 11A and 11B It is a diagram illustrating the UI according to Embodiment 3;

[0019] Figure 12 It is a flowchart illustrating the process of arranging previously captured images according to Embodiment 4;

[0020] Figures 13A to 13D It is a diagram illustrating the UI according to Embodiment 4;

[0021] Figure 14 It is a flowchart illustrating the process of arranging previously captured images according to Embodiment 5;

[0022] Figure 15 It is a flowchart illustrating the process of arranging previously captured images according to Embodiment 6;

[0023] Figure 16A and 16B It is a diagram illustrating the template for the display position of previously captured images according to Embodiment 7;

[0024] Figures 17A to 17C It is a diagram illustrating the live view display screen according to Embodiment 7;

[0025] Figure 18 It is a flowchart illustrating the imaging process according to Embodiment 8; and

[0026] Figures 19A to 19F It is a diagram illustrating the live view display screen according to Embodiment 8. Detailed Description

[0027] Embodiment 1

[0028] Embodiments of the present invention will be described with reference to the accompanying drawings. In Embodiment 1, a previously captured image is displayed together with a live image (a moving image of a subject to be photographed) on a live view display screen that displays the live image. On the screen of the imaging device, the display position of the live image and the display position of the previously captured image can be determined based on a user's operation. The user can capture an image that is consistent with the previously captured image by adjusting the composition, brightness, hue, etc. of the live image according to the previously captured image. The elements by which the user senses consistency are different. One example is the commonality of composition, such as the commonality of the position and size of the subject between images, or the commonality of the horizontal position in the landscape between images, etc. Another example is the commonality of the subject category in terms of food, people, landscape, and animals. In addition, the user can sense consistency through the commonality of the hue and gray scale of the subject between images or the commonality of the filters (e.g., saturation emphasis) applied to the images.

[0029] The imaging device according to Embodiment 1 is described as a smart phone having a camera function, but is not limited thereto. The imaging device can be a different device having an imaging function, such as a digital camera, a digital video camera, a portable phone, and other portable electronic devices.

[0030] Device Structure: Figure 1 is a block diagram depicting an example of the structure of the imaging device. With reference to Figure 1 An example of the structure of the imaging device 100 according to Embodiment 1 will be described.

[0031] For example, the control unit 101 is a processor such as a CPU or a DSP. The control unit 101 reads a control program for each block included in the imaging device 100 from the ROM 102, expands the control program in the RAM 103, and executes the program. Thereby, the control unit 101 can control the operations of each block included in the imaging device 100.

[0032] The ROM 102 is an electrically erasable and recordable non-volatile memory. The ROM 102 stores not only the operation programs for each block included in the imaging device 100, but also parameters for the operations of each block, etc.

[0033] The RAM 103 is a rewritable non-volatile memory. The RAM 103 is used to expand the programs executed by the control unit 101, etc., and temporarily stores the data generated during the operations of each block included in the imaging device 100.

[0034] The optical system 104 is composed of a lens group including a zoom lens and a focusing lens, and forms an image of a subject on the imaging surface of the imaging unit 105 (imaging element). For example, the imaging unit 105 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 105 performs photoelectric conversion on the optical image formed by the optical system 104 on the imaging surface of the imaging unit 105, and outputs the acquired analog image signal to the A / D conversion unit 106.

[0035] The A / D conversion unit 106 converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily stored in the RAM 103.

[0036] The image processing unit 107 expands the digital image data based on the image data stored in the RAM 103, and applies various types of image processing to display or record the image data. Examples of the image processing applied to the image data include: defect correction processing for pixels in the case of defects caused by the optical system 104 or the imaging element, demosaicing processing, white balance correction processing, color interpolation processing, and gamma processing.

[0037] The recording unit 108 records the data including the image data to an internal recording medium. The display unit 109 includes a display device such as a liquid crystal display (LCD), and displays the image stored in the RAM 103 and the image recorded in the recording unit 108 on the display device. The display unit 109 also displays a user interface for operation, which is used to receive instructions from the user.

[0038] The input unit 110 is an input interface including various physical operation members such as a touch panel and a shutter button, and receives input instructions from the user. The communication unit 111 (transmission unit) is wirelessly connected to an external server to transmit / receive image data and data including the arrangement information mentioned later. The external server is, for example, a social network service (hereinafter referred to as SNS), which provides the following services: receiving images taken by the user and posting these pictures to the viewers of the SNS.

[0039] Based on the control of the control unit 101, the imaging device 100 performs live view display of sequentially displaying the analog image signal (sequentially output from the imaging unit 105) via the A / D conversion unit 106, the RAM 103, the image processing unit 107, and the display unit 109. During the live view display, the user can adjust the composition for actual imaging to record the image data to the recording medium. The user can also prepare for actual imaging during the live view display by changing the image processing conditions to change the exposure conditions (e.g., aperture, shutter speed) and the image quality (e.g., brightness, gray scale, hue).

[0040] Figure 2A And 2B are diagrams illustrating the appearance of the imaging device. Reference will be made to Figure 2A and 2B to describe the appearance of the smartphone used as the imaging device 100. Figure 2A is a front view of the imaging device 100 (smartphone), Figure 2B is a rear view of the imaging device 100.

[0041] The imaging device 100 includes a display unit 109 and a camera unit 201. The camera unit 201 includes Figure 1 the optical system 104, the imaging unit 105, and the A / D conversion unit 106 described in

[0042] Imaging processing according to Embodiment 1: Figure 3 is a flowchart illustrating the imaging processing according to Embodiment 1. Reference will be made to Figure 3 to describe in detail the imaging processing of the imaging device 100 according to Embodiment 1. Each step of the imaging processing is executed by the control unit 101 or by each block of the imaging device 100 that receives an instruction from the control unit 101.

[0043] Reference will be made to Figures 4A to 4C and Figures 5A to 5C to describe Figure 3 the live view display screen and the image group display screen shown in Figures 4A to 4C and Figures 5A to 5B that are displayed on the display unit 109 during the imaging processing. The live view display screens in Figure 5C are display screens before imaging, and are used to display a live image, arrange previously captured images, and adjust the live image. The image group display screen in Figure 5C is a display screen after imaging, and is used to display a group of images composed of the captured images of the adjusted live image and the arranged previously captured images. Figures 4A to 4C is a diagram for describing the arrangement of the live image and the previously captured images. Figures 5A to 5C is a diagram for describing the adjustment of the live image.

[0044] When the user sets the imaging device 100 to an imaging mode in which an image can be captured by the camera, the imaging processing in Figure 3 is started. For example, the user can select the imaging mode by starting a camera function application on the smartphone.

[0045] In step S301, the control unit 101 starts the live view display by controlling the optical system 104, the imaging unit 105, the A / D conversion unit 106, the RAM 103, the image processing unit 107, and the display unit 109.

[0046] ​During the live view display period, the imaging device 100 sequentially captures and acquires live images. The acquired live images are displayed on the display device of the display unit 109. While checking the sequentially displayed live images, the user can determine the composition or change the exposure conditions, image processing conditions, etc. The processing steps in S302, S303, S304, S305, and S306 (described later) are processing steps executed during the live view display period.

[0047] In step S302, the control unit 101 displays a template indicating the display position of the image arranged on the display unit 109. The template has a plurality of pre-divided areas. The control unit 101 displays a predetermined template or a template selected by the user.

[0048] The reference Figure 4A Describes the template for the display position of the image. The template for the display position includes a plurality of frames (areas) for arranging the image. For example, the user can select the display position of the image by dragging the image selected from the non-arranged images displayed on the display unit 109 and placing the image in the designated area of the template.

[0049] Figure 4A A live view display screen indicating the template 401 for the display position displayed on the display unit 109. The template 401 includes a total of nine rectangular areas (three rows × three rows). Each rectangular area indicates the position for arranging the image. One image is arranged in one rectangular area. In other words, nine images can be arranged in the template 401.

[0050] The template for the display position is not limited to Figure 4A The example in. For example, the template for the display position can be divided into four areas where four images can be arranged. Each of the plurality of divided areas is not limited to a rectangle, but can be various shapes including a circle and a polygon. The sizes of the respective areas of the template for the display position can be different from each other. In addition, the user can freely arrange each image within the predetermined area without using the template to determine the display position of the image.

[0051] In step S303, the control unit 101 (determination unit) determines the display position of the live image of the subject to be photographed displayed in the template 401 for the display position on the live view display screen. The control unit 101 can determine the area selected by the user from the areas included in the template 401 as the display position of the live image.

[0052] The reference Figure 4B To describe the determination of the display position of the live image. In Figure 4AIn the state where the template 401 that is currently displaying the display position, the user selects a rectangular area through a touch operation to display a live image. The control unit 101 sets the rectangular area selected by the user as the display position of the live image, and displays the live image in this rectangular area. Figure 4B The live view display screen in shows an example in the case where the user selects the rectangular area 402 located at the center of the nine rectangular areas as the display position of the live image. The control unit 101 determines the rectangular area 402 as the display position of the live image, and displays a live image of a plate with a fried egg and slices of bread placed in the rectangular area 402. The subject is not limited to food, but can be a person, an animal, a building, a landscape, etc.

[0053] The determination of the display position of the live image is not limited to the user's selection. For example, the display position of the live image can be set in advance for each template.

[0054] In step S304, based on the user operation, the control unit 101 arranges the previously captured image in the template 401 at the display position. For example, the user operation is an operation of dragging an image selected from the previously captured images that are displayed on the display unit 109 and not yet arranged, and placing the image at the target position (area) to arrange the image in the template 401. The control unit 101 arranges the image selected from the previously captured images that are not yet arranged in the target area through a drag-and-drop operation.

[0055] will refer to Figure 4C Describe the selection and arrangement of the previously captured images. Figure 4C The live view display screen in shows an example in the case where five previously captured images are arranged around the rectangular area 402 where the live image is displayed as a result of the process in step S304.

[0056] In step S303, when the display position of the live image is determined, the control unit 101 displays a list 403 of previously captured images that are not arranged at a position different from the template 401. In Figure 4C the example of, five previously captured images that are not yet arranged are displayed in the list 403 of previously captured images that are not arranged. The images displayed in the list 403 of previously captured images that are not arranged are obtained from the recording unit 108. The user selects a desired previously captured image while checking the previously captured images that are not arranged and displayed on the live view display screen, and drags and drops the selected image to the arrangement target position in the template 401, whereby the previously captured images can be arranged one by one.

[0057] To provide a sense of consistency to the images within the image group in the list display, it is assumed that the user selects an image from the list 403 of previously taken images that have not been arranged, which is similar to the live image and the previously taken images that have been arranged in terms of subject, composition, brightness, and tone. In Figure 4C 's example, the images of the chicken and the flower in the list 403 of previously taken images that have not been arranged are different subjects from the slice of bread in the live image displayed in the rectangular area 402. Therefore, instead of selecting the images of the chicken and the flower, the user selects the image of the slice of bread, which is the same subject as the live image, from the list 403 of previously taken images that have not been arranged, and arranges this image. In this way, the control unit 101 arranges the previously taken image in the template 401 based on the operation of the user who arranges the desired image at the desired position.

[0058] The arrangement of the previously taken image is not limited to the arrangement determined based on the user operation. The control unit 101 can analyze each image in the list 403 of previously taken images that have not been arranged, and arrange the image that is similar to the live image in terms of subject, composition, brightness, or tone in the template 401.

[0059] In step S305, the control unit 101 adjusts the shooting conditions of the live image so that the composition, brightness, or tone of the live image becomes similar to the arranged previously taken image. The shooting conditions are, for example, the conditions related to the image processing operations that can be adjusted when shooting the live image (e.g., composition, exposure conditions, color, grayscale).

[0060] As an example of adjusting the shooting conditions, the method of adjusting the composition will be described with reference to Figure 5A and 5B . Figure 5A and 5B indicate examples of the live view display screen after arranging 8 previously taken images in step S304. The zoom bar 501 is a user interface for changing the zoom factor of the live image. The shooting button 502 is a UI for actually shooting an image by pressing (touch operation).

[0061] In Figure 5A 's live view display screen, the slice of bread, which is the subject of the live image in the rectangular area 402, is displayed in a smaller size than the slices of bread in the previously taken images arranged around the live image. If an image is shot in this state, the subject sizes are different between the live image and the previously taken images, so the display of the image group including the shot subject image and the previously taken images cannot provide a sense of consistency, which is not desirable.

[0062] Figure 5BAn example after the live view display screen indicates that the user performs an operation to increase the zoom factor of the live image, thereby providing a sense of consistency in the display of the image group. The user can increase the zoom factor by moving the tab of the zoom bar 501 in the + direction. The subject size is increased by the operation of increasing the zoom factor. In Figure 5A the subject size of the live image is smaller than the subject size of the previously captured images around the live image, but in Figure 5B as a result of increasing the zoom factor, the subject size of the live image is approximately the same as the subject size of the previously captured images around the live image. By changing the zoom factor of the live image, the user can make the subject size consistent between the live image and the previously captured images, so that the list of the live image and the previously captured images can be displayed with a sense of consistency.

[0063] Figure 5A and 5B indicate examples of adjusting the composition, but the adjustment of the shooting conditions is not limited to this. The user can adjust the shooting conditions such as the brightness or hue of the live image on the live view display screen. In other words, the live view display screen can include a UI for adjusting the brightness and hue of the live image. By adjusting one or a combination of the composition, brightness, and hue, etc., the user can capture an image with a higher sense of consistency with the previously captured images.

[0064] The change of the shooting conditions is not limited to the user's change, but can be automatically performed by the imaging device 100 or can be suggested by the imaging device 100. For example, the control unit 101 (changing unit) can automatically change the brightness and hue of the live image, etc., to provide a sense of consistency with the previously captured images. Similarly for the composition, the control unit 101 can automatically change the zoom factor of the live image so that the subject size of the live image becomes consistent with the subject size of the previously captured images.

[0065] In the automatic adjustment of the shooting conditions, the imaging device 100 adjusts the shooting conditions so that the composition, brightness, and hue, etc., become similar between the previously captured images arranged and the live image. The shooting conditions are image processing conditions for changing the composition, exposure conditions, color, and grayscale, etc.

[0066] As an example of adjusting the shooting conditions, the method of adjusting the composition will be described with reference to Figure 5A and 5B Figure 5A and 5B indicate examples of the live view display screen after eight previously captured images are selected and arranged in step S304. The shooting button 502 is a UI for receiving a shooting operation.

[0067] In Figure 5A ​In the live view display screen, the toast of the live image displayed in the rectangular area 402 is displayed in a size smaller than that of the toast of the previously captured images around the live image. If an image is captured in this state, the subject sizes between the live image in the rectangular area 402 and the previously captured images are different, and the list display of these images may not provide a sense of consistency.

[0068] To provide a sense of consistency for the subject size, the control unit 101 calculates the subject size of each previously captured image (e.g., the longitudinal and lateral lengths of a rectangle surrounding the subject). For example, the control unit 101 may set the average value of the sizes of each previously captured image as the subject size of the live image and accordingly adjust the subject size of the live image. In the case where the size of the area for displaying each previously captured image is different from the size of the area where the live image is displayed, the control unit 101 may set the subject size of the live image according to the surface area ratio for each area.

[0069] The control unit 101 adjusts the focal length of the optical system 104 so that the subject in the live image is displayed in the set size. In Figure 5A the example, the subject of the live image displayed in the rectangular area 402 is smaller than the subject of the previously captured images surrounding the live image. Therefore, the control unit 101 changes the focal length of the lens toward the telephoto side to increase the displayed subject size, as shown in the live view display screen in Figure 5B the example.

[0070] If the imaging device 100 includes a plurality of optical systems 104, the optical system 104 to be used can be switched according to the subject size. In the case of an imaging device 100 having a single focal length lens (e.g., a smart phone), by cropping or magnifying (electronic zooming) the image, the subject size of the live image can be made to match the subject size of each previously captured image. In addition, the imaging device 100 can match the subject size of the live image with the subject size of each previously captured image by changing the focal length of the optical system 104 and using electronic zooming.

[0071] By adjusting the subject size, Figure 5A the subject of the live image in the example (whose size is smaller than the subject size of the previously captured images surrounding the live image) can be magnified to Figure 5B a size similar to the subject size of the previously captured images surrounding the live image in the example. By automatically changing the zoom factor, the subject sizes of multiple images become similar, so that the list of multiple images displayed on the live image screen after imaging becomes a list with a sense of consistency.

[0072] The process of making the subject sizes of multiple images consistent is not limited to adjusting the subject size of the live image based on a previously captured image. The control unit 101 can adjust the subject size of the previously captured image based on the live image. For example, when the subject of the previously captured image is smaller than the subject of the live image, the control unit 101 can increase the subject size of the previously captured image by cropping and enlarging the previously captured image.

[0073] In addition to the subject size, the control unit 101 can also provide a sense of consistency in the degree of blur (depth of field), etc. through optical system control, image processing, or a combination thereof. For example, if the degree of blur is made consistent, the list display of the longitudinal images becomes a more desirable display. The depth of field changes according to the imaging conditions (such as focal length, subject distance, and aperture, etc.). By automatically determining the imaging conditions for the live image based on the imaging conditions of the previously captured image, the control unit 101 can make the depth of field of the live image the same as that of the previously captured image. By matching the depth of focus of the live image with the previously captured image, the control unit 101 can make the degree of blur in the list display consistent.

[0074] In step S306, the control unit 101 determines whether the adjustment of the imaging conditions in step S305 is completed. For example, the control unit 101 can determine whether the adjustment of the imaging conditions is completed based on whether the user has completed the adjustment of the imaging conditions and pressed the imaging button 502. If the adjustment of the imaging conditions is completed (step S306: Yes), the process proceeds to step S307. If the adjustment of the imaging conditions is not completed (step S306: No), the process returns to step S302, and the processing steps S302, S303, S304, and S305 are repeated until the adjustment of the imaging conditions is completed. By adjusting the layout, composition, brightness, and hue of the live image based on such user instructions, the control unit 101 can capture an image with a sense of consistency with the previously captured image.

[0075] When the adjustment of the live image is completed in step S306, the imaging button 502 is pressed, and the control unit 101 performs actual imaging in step S307. The image captured by the actual imaging step is in the state adjusted in step S305, so the composition, brightness, and hue, etc. of the captured image become similar to those of the previously captured images arranged on the live view display screen.

[0076] The control unit 101 corresponds the live image captured by the actual shooting and the previously captured images arranged on the live view display screen as an image group. In addition, the control unit 101 corresponds the image data of the captured image of the live image and the previously captured images with the arrangement information including the display position of the live image and the respective display positions of the previously captured images, and records the image data in the recording unit 108 in this state.

[0077] In step S307, actual imaging is performed by the user pressing the imaging button 502, but actual imaging can be automatically performed. When it is determined that the adjustment of the imaging conditions (e.g., subject size) of the live image based on the previously captured image in step S305 is completed, the control unit 101 can automatically perform actual imaging.

[0078] In addition, the control unit 101 can notify the user that the adjustment of the imaging conditions is completed. For example, the control unit 101 can display a message for notifying the completion of the adjustment of the imaging conditions on the display unit 109. Then, the user checks the message displayed on the display unit 109 and performs actual imaging.

[0079] In step S308, the control unit 101 acquires the image data and layout information of the image group recorded in the recording unit 108, and displays the image data of the image group on the display unit 109 based on the acquired layout information.

[0080] Reference will be Figure 5C made to the image group display screen that displays the image data in the display image group. Figure 5C The image group display screen of indicates an example in the following case: the actually captured image of the subject is displayed in the rectangular area 503, and the previously captured image corresponding to the captured image is displayed around the captured image. The control unit 101 arranges the image data in the image group on the display screen of the display unit 109 based on the layout information related to the image group. When each image is displayed on the image group display screen, the imaging process ends.

[0081] In the above-described first embodiment, the composition, brightness, hue, etc. of the live image can be adjusted while checking the previously captured image arranged on the live view display screen. Thus, the user can capture a live image that provides a sense of consistency with the previously captured image. Therefore, the user can easily generate an image group with a sense of consistency.

[0082] In the first embodiment, the adjustment of matching the composition of the live image with the previously captured image displayed on the live view display screen is described, but the adjustment of the live image is not limited thereto. For example, the adjustment of the live image includes the adjustment of exposure conditions or the adjustment of image processing conditions to change the image quality such as brightness, gray scale, and hue.

[0083] Specifically, the adjustment of exposure conditions can be a process of adjusting the brightness of the live image by adjusting the exposure correction amount. In addition, the adjustment of image processing conditions can be a process of adjusting the saturation of the live image by adjusting the chroma enhancement amount.

[0084] The control unit 101 (upload unit) of the imaging device 100 may allow a user to post an image group with a sense of consistency by uploading the image data and arrangement information of the generated image group to an external server that provides services such as SNS.

[0085] Embodiment 2

[0086] Embodiment 2 is an embodiment in which a previously captured image arranged on a live view display screen is acquired from an external server that provides services such as SNS. The structure of the imaging device according to Embodiment 2 is the same as that of the imaging device 100 according to Embodiment 1, and thus its description is omitted.

[0087] The external server receives an image group of previously captured images and corresponding arrangement information from a plurality of imaging devices 100 and records them in a recording unit of the external server. When accessing an external service via a web browser or the like, the external server can display a list of previously captured images included in the image group based on the arrangement information.

[0088] Imaging processing according to Embodiment 2: Figure 6 is a flowchart illustrating the imaging processing according to Embodiment 2. The imaging processing of the imaging device 100 according to Embodiment 2 will be described in detail with reference to Figure 6 Each step of the imaging processing is executed by the control unit 101 or by each block of the imaging device 100 that receives an instruction from the control unit 101. Steps that are the same as those of the imaging processing according to Embodiment 1 in Figure 3 are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0089] The live view display screen and the image group display screen displayed in the imaging processing shown in Figure 6 will be described with reference to FIGS. 7A to 7C. Figure 7A and 7B The live view display screens in are screens for displaying previously captured images based on the image data and the arrangement information of the image group acquired from the external server. Figure 7C The image group display screen in is a screen for displaying the captured images in the image group and has a UI for sending the captured image of the subject and the corresponding arrangement information to the external server.

[0090] In step S301, the control unit 101 starts live view display. In step S601, the control unit 101 acquires the captured images and the corresponding arrangement information in the image group recorded on the external server via the communication unit 111.

[0091] In step S602, the control unit 101 arranges the captured images (image data in the image group) acquired in step S601 on the live view display screen based on the arrangement information.

[0092] will refer to Figure 7A describe the arrangement of previously captured images. Figure 7A Indicates an example in the case where seven previously captured images to be obtained from an external server are arranged on a live view display screen based on the corresponding arrangement information. It is assumed that information related to a template for the display positions where the previously captured images are to be arranged is included in the arrangement information.

[0093] Figure 7A Is an example in the case where seven previously captured images are arranged at seven positions in a template including nine rectangular regions. Among the nine rectangular regions, no previously captured images are arranged in the lower - center and lower - right rectangular regions. In this way, the imaging device 100 can acquire the previously captured images and the arrangement information in the image group recorded on the external server, and arrange the previously captured images on the live view display screen based on the acquired arrangement information. In other words, the imaging device 100 can reproduce the image data in the image group on the live view display screen in the same arrangement as when accessing and viewing the image data on the external server.

[0094] The display positions of the previously captured images on the live view display screen can be changed by the user. The user can change the position of a target previously captured image by dragging it to a desired position. If a previously captured image has already been arranged at the target position, the image and the image whose position is changed to this position can be replaced with each other.

[0095] In step S603, the control unit 101 determines the display position of the live image. For example, as the display position of the live image, the control unit 101 can determine the area selected by the user from the rectangular regions in the template where no previously captured images are arranged.

[0096] will refer to Figure 7B to describe the determination of the display position of the live image. In Figure 7A the example, as the display position of the live image, the user selects, by a touch operation, the lower - center or lower - right rectangular region in the nine rectangular regions where the previously captured images are arranged. Figure 7B Indicates an example in the case where the user selects the rectangular region 701 at the lower - center. The control unit 101 displays the live image in the rectangular region 701 selected by the user.

[0097] In steps S305 and S306, as in Embodiment 1, the control unit 101 changes the imaging conditions of the live image based on a user instruction, and determines whether the adjustment of the imaging conditions is completed. If the adjustment of the imaging conditions is not completed (step S306: No), the process returns to step S601, and the process steps S601, S602, S603, and S305 are repeated until the adjustment of the imaging conditions is completed.

[0098] In steps S307 and S308, the control unit 101 performs actual imaging as in Embodiment 1, and corresponds the captured image of the live image with the previously captured image arranged in the template as an image group, and displays the image group on the image group display screen.

[0099] In step S604, the control unit 101 determines whether to send the image data and the corresponding arrangement information in the image group to an external server based on a user instruction.

[0100] Reference will be Figure 7C made to the reception of an instruction to send the image data and the arrangement information of the image group. Figure 7C This is an example of a screen in which a UI is added to the image group display screen for displaying the image group in step S308 in order to receive an instruction on whether to send the image data and the arrangement information of the image group to an external server. The user can check the sense of consistency in the image group in terms of composition, brightness, hue, etc., and determine whether the image data and the arrangement information of the image group will be sent to an external server. Figure 6

[0101] In the case where the image data and the arrangement information of the image group are to be sent to an external server, the user presses (touches) the "Yes" button. If the "Yes" button is pressed (touched), the control unit 101 determines that the image data and the corresponding arrangement information in the image group are to be sent to an external server (step S604: Yes), and the process proceeds to step S605.

[0102] In the case where the image data and the arrangement information of the image group are not sent to an external server, the user presses (touches) the "No" button. If the "No" button is pressed (touched), the control unit 101 determines that the image data and the corresponding arrangement information in the image group are not sent to an external server (step S604: No), and ends Figure 6 the imaging process in

[0103] In step S605, the control unit 101 sends the image data and the corresponding arrangement information in the image group to an external server via the communication unit 111, and ends Figure 6 the imaging process in

[0104] ​In the above-described Example 2, the external server can generate a screen to display the image data in the image group in the arrangement at the time of shooting based on the image data and the corresponding arrangement information in the image group received from the imaging device 100. Thus, when accessing the external server via a web browser or the like and viewing the images, the images in the image group can be viewed in the same arrangement as when the imaging device 100 captured the images.

[0105] Example 3

[0106] On the live view display screen, the user operates to select a previously captured image to be compared with the live image from the list 403 of previously captured images that are not arranged. In Example 3, in order to simplify the process for the user to select and arrange an image having the same characteristics [as the live image] from the previously captured images, related previously captured images obtained based on the characteristics of the representative image selected by the user are displayed as selection candidates in the list 403 of previously captured images that are not arranged. The representative image is an image set as the representative image of previously captured images having the same label. The previously captured images are classified by labels attached based on image characteristics.

[0107] When the user selects a representative image, the candidates for the comparison target previously captured images are narrowed down based on the characteristics of the representative image. Thus, the user can reduce the process for selecting the comparison target previously captured images. The structure of the imaging device in Example 3 is the same as Figure 1 the structure of the imaging device 100 according to Example 1 shown. Now, the processing steps different from Example 1 will be described.

[0108] Label attachment processing: The control unit 101 pre-attaches labels to the previously captured images and determines the representative image for each label. First, the label attachment processing for attaching labels to the previously captured images will be described.

[0109] In the case where a new previously captured image is added to the recording unit 108, the control unit 101 attaches a label to the added image. The control unit 101 extracts image characteristics from the captured image added to the recording unit 108. The image characteristics are, for example, the composition, color tone, and subject of the image.

[0110] In the feature extraction based on the composition, the control unit 101 extracts the outline of the subject from the extraction target previously captured image. Based on the extracted outline, the control unit 101 determines which composition in the predetermined classified compositions the image is classified into. The predetermined classified compositions include, for example, centered composition, rule of thirds, and diagonal composition.

[0111] The control unit 101 can extract the features of the composition by determining the mode of the composition based on the outline of the subject. However, the control unit 101 can extract the features of the composition by any method as long as the features of the composition can be extracted thereby. For example, the control unit 101 can extract the features of the composition based on the position or size of the subject, etc. In addition, the control unit 101 can crop the surroundings of the subject and thereby determine the composition.

[0112] In the feature extraction based on hue, the control unit 101 extracts the representative colors from the previously captured image of the extraction object. The representative colors are the pixel values that appear most frequently in the previously captured image of the object. The control unit 101 can extract multiple representative colors, including the second most frequently appearing pixel value and the third most frequently appearing pixel value. Thereby, the control unit 101 can extract the features of the hue.

[0113] The control unit 101 can extract the hue features by any method as long as the hue features can be extracted thereby. For example, the control unit 101 can extract the features of the hue by weighting the pixel values according to the appearance frequencies of the multiple representative colors.

[0114] In the feature extraction based on the subject, for the previously captured image of the extraction object, the control unit 101 determines which subject in the predetermined classified subjects the subject in the image belongs to. For example, the predetermined classified subjects include face, animal, cooking, and landscape.

[0115] The control unit 101 can extract the features of the subject by using image recognition technology. However, the control unit 101 can extract the features of the subject by any method as long as the features of the subject can be extracted thereby. For example, the control unit 101 can extract the features of the subject based on the shape or hue of the subject, etc.

[0116] The control unit 101 attaches the feature information of the image extracted from the previously captured image as a label to the previously captured image. The label is attached by adding the features of the object image to the object image data as metadata.

[0117] Figure 8 is a diagram illustrating the label (metadata) indicating the feature information of the image. Metadata is data added to the previously captured image data. In Figure 8 the example, the label of the composition of the object image is "centered composition", the label of the hue is "#E3E9DF, #DA4011", and the label of the subject is "cooking".

[0118] Representative image setting process: The control unit 101 sets a representative image for the previously captured image with a label attached. Now, the representative image setting process will be described.

[0119] For example, the control unit 101 sets a representative image for a group of images having the same tag value as the previously captured images newly added to the recording unit 108. The representative image can be set for a group of images having one same tag value, or can be set for a group of images having multiple same tag values. The tag values do not need to be exactly the same, and for example, a group of images having a hue feature with a color code indicating a similar color can be regarded as a group of images having the same tag value.

[0120] The representative image can be selected by any method as long as an image that can be a representative of the group of images having the same tag value can be selected. For example, the control unit 101 can select the representative image based on the shooting date and time of each image included in the group of images having the same tag value. Specifically, for the representative image, the control unit 101 can select the image in the group of images having the same tag value whose shooting date and time is the latest.

[0121] The control unit 101 can set the representative image based on a combination of multiple tags. For example, for the previously captured images having the tag information shown Figure 8 a representative image is set for the group of images whose composition is "centered composition" and the subject is "cooking". By combining multiple tags, the control unit 101 can set a representative image for a group of images with more restricted conditions.

[0122] The information related to the representative image is added to the image data as metadata, just like the information about the tags. The information related to the representative image includes the type of the tags shared in the group of images to which the representative image belongs and the values of the tags.

[0123] The representative image can be set at the timing when the previously captured images are newly added to the recording unit 108, or can be set at the timing when a predetermined number of images are added to the recording unit 108. In the case where a representative image has already been set for a group of images having the same tag value, programming can be performed so that the control unit 101 does not execute the representative image setting process.

[0124] Shooting process according to Embodiment 3: Figure 9 is a flowchart illustrating the shooting process according to Embodiment 3. Figure 10 is a flowchart illustrating the arrangement process of the previously captured images according to Embodiment 3. The arrangement process of the previously captured images is a sub - flow of the shooting process. Reference will be made to Figure 9 and 10 to describe the shooting process of the imaging device 100 according to Embodiment 3. Except for the process in step S901, Figure 9 the shooting process in Figure 3 is the same as the shooting process inFigure 3 The same processing steps are performed, and their descriptions are omitted. In Figure 10 the arrangement process of the previously captured images in

[0125] In step S1001, the control unit 101 receives an operation from the user to select a representative image. Reference will be made to Figure 11A and 11B to describe the UI for receiving the selection of the representative image from the user. Figure 11A It indicates a state in which a plurality of representative images (selection candidates) are displayed in the display area 1102 of the display unit 109. In the case of FIG. 11A, five representative images 1101 are displayed in the display area 1102.

[0126] The control unit 101 receives the selection of the representative image through an operation performed by the user in the display area 1102. For example, the user can select the representative image by touching or dragging the desired representative image. The control unit 101 arranges the selected representative image in one of the rectangular areas of the template 401 at the display position. The position where the selected representative image is arranged can be a position specified by the user and can be a rectangular area to which the user moves the representative image by a drag-and-drop operation.

[0127] In the case where the desired representative image is not displayed in the display area 1102, the user can change the candidates of the representative images displayed in the display area 1102. For example, the control unit 101 receives an instruction to display other representative images through a sliding operation performed by the user in the display area 1102. When the instruction is received from the user, the control unit 101 changes the representative images displayed in the display area 1102 to other representative images. Thus, even if the desired representative image is not currently displayed, the user can change the display of the display area 1102 and display the desired representative image.

[0128] The control unit 101 can change the display in the display area 1102 by receiving an operation that specifies the type (composition, tone, subject) of the label of the image to which the user assigns a priority. In this case, the control unit 101 displays a UI for receiving the specification of the label type on the display unit 109. By the user specifying the type of the label, the control unit 101 can display representative images having the specified label type in the display area 1102. Thus, in step S1004, the control unit 101 can present to the user the previously captured images having the labels (features of the images) to which the user assigns a priority.

[0129] In step S1002, the control unit 101 determines whether the user has selected a representative image. If a representative image is selected (step S1002: Yes), the process proceeds to step S1003. If no representative image is selected (step S1002: No), the process proceeds to step S1005. For example, the case where no representative image is selected is the case where the user does not perform the operation of selecting a representative image within a predetermined time.

[0130] Once a representative image is selected, for example, as Figure 11B shown, the selected representative image is arranged in the rectangular area 1105. If the user moves the representative image to any one of the rectangular areas of the template 401 by a drag-and-drop operation, the control unit 101 arranges the selected representative image in the rectangular area specified by the user.

[0131] In step S1003, the control unit 101 acquires relevant previously captured images from the recording unit 108 based on the representative image selected by the user. The relevant previously captured images are images related to the representative image selected by the user and are images presented to the user as candidates for the previously captured images to be arranged in the template 401. The relevant previously captured images can be previously captured images with the same marker value as the representative image.

[0132] When acquiring the relevant previously captured images, the control unit 101 may receive an operation for specifying the type (composition, tone, theme) of the label of the image to which the user assigns a priority. In this case, the control unit 101 displays a UI for receiving the specification of the label type on the display unit 109. The control unit 101 preferentially acquires relevant previously captured images having the same label value as the specified one. In the case where the user selects multiple labels, the control unit 101 acquires relevant previously captured images having the same values of the multiple selected labels. Thus, the control unit 101 can preferentially acquire relevant previously captured images from the recording unit 108 according to the type of the label to which the user assigns a priority.

[0133] In step S1004, the control unit 101 displays the relevant previously captured images acquired in step S1003 on the display unit 109 as candidates for the previously captured images to be arranged in the template 401.

[0134] Figure 11B Indicating Figure 11A An example of the display of the display unit 109 after the state in Figure 11B is changed in processing steps S1002 to S1004. In

[0135] The control unit 101 displays the relevant previously captured image 1103 obtained in step S1003 in the display area 1104 as a candidate for the previously captured image to be arranged in the template 401. In Figure 11B the case of, the control unit 101 displays five relevant previously captured images 1103 as candidates. Here, the control unit 101 removes from the candidates displayed in the display area 1104 the relevant previously captured images that have already been arranged in the rectangular area of the template 401, and displays in the display area 1104 the other relevant previously captured images that have not been arranged.

[0136] In the case where the desired relevant previously captured image is not displayed in the display area 1104, the user can change the relevant previously captured image displayed in the display area 1104. For example, the control unit 101 receives an instruction to display other relevant previously captured images through a sliding operation performed by the user in the display area 1104. When the instruction is received from the user, the control unit 101 changes the relevant previously captured image displayed in the display area 1104 to another relevant previously captured image. Thus, even if the desired relevant previously captured image is not currently displayed, the user can change the display in the display area 1104, and thus the desired relevant previously captured image can be displayed.

[0137] In the case where the number of relevant previously captured images obtained in step S1003 is less than the number of images that can be displayed in the display area 1104, the control unit 101 can additionally display other previously captured images. The other previously captured images to be added are, for example, previously captured images having the same value as any label attached to the representative image, or previously captured images with a shooting date and time or shooting location close to the representative image. Thus, even if the number of relevant previously captured images is less than the number of images that can be displayed in the display area 1104, the user can easily select other previously captured images. After the processing in step S1004 ends, the processing proceeds to step S1006.

[0138] In step S1005, since the user has not selected a representative image, the control unit 101 displays the previously captured images as candidates for the previously captured images to be arranged in the template 401. The candidates for the previously captured images in step S1005 can be displayed in any priority order. For example, the control unit 101 can display the previously captured images in the order of shooting date and time. The control unit 101 removes from the candidate display the previously captured images that have already been arranged in the rectangular area of the template 401, and displays in the display area 1104 the previously captured images that have not been arranged.

[0139] In step S1006, the control unit 101 receives an instruction from the user to arrange the (relevant) previously captured image to be displayed in the display area 1104 in steps S1004 and S1005 in the template 401. The process of arranging the selected (relevant) previously captured image in the template 401 is the same as step S304 of Embodiment 1.

[0140] Each time a (relevant) previously captured image is displayed in the template 401, the control unit 101 removes the (relevant) previously captured image arranged in the template 401 from the candidate display in the display area 1104. When a (relevant) previously captured image is arranged in the template 401, the control unit 101 displays another (relevant) previously captured image in the display area 1104.

[0141] In step S1007, the control unit 101 determines whether an instruction to end the arrangement of the previously captured image is received from the user. If an end instruction is received (step S1007: Yes), the process of arranging the previously captured image ends, and the process proceeds to Figure 9 step S305. If an end instruction is not received (step S1007: No), the process returns to step S1001.

[0142] In the case where a representative image has been selected and arranged in the template 401, the control unit 101 may return to step S1003 instead of returning to step S1001 and continue the process.

[0143] In Embodiment 3, the control unit 101 displays candidates for the previously captured images to be arranged in the template 401 based on the representative image selected by the user. Thereby, the imaging device 100 can reduce the steps that the user has to take to select the previously captured image to be compared with the live image.

[0144] In step S1006, for the candidates to be displayed in the display area 1104, the control unit 101 can change whether to display the relevant previously captured image of the representative image or the previously captured image under different conditions according to an instruction from the user. The previously captured images under different conditions are, for example, the previously captured images captured during a specific imaging period, the previously captured images captured at a predetermined imaging position, or all previously captured images. Therefore, even if the previously captured image desired by the user is not displayed as a candidate, the control unit 101 can present the previously captured images under different conditions to the user.

[0145] In addition, when arranging the relevant previously captured images selected by the user in the template 401 in step S1006, the control unit 101 may update the candidates of the relevant previously captured images to be displayed in the display area 1104. For example, the control unit 101 determines the tags having a common value by the tag of the representative image selected by the user in step S1002 and the tag of the relevant previously captured image selected by the user in step S1006. The control unit 101 preferentially displays, in the display area 1104, the previously captured images having the same value as the common tag as the new relevant previously captured images. Since the candidates to be displayed in the display area 1104 are updated based on the relevant previously captured images selected by the user, the control unit 101 can more easily display the relevant previously captured images desired by the user as candidates.

[0146] The relevant previously captured images are not limited to the previously captured images having the same value of the tag, but may be the previously captured images having the values within a similar range of the tag. The similar range of the values of the tag can be determined in advance. For example, in the tag of the composition, the three-division rule (vertical) and the two-division rule (vertical) can be defined as the similar range. In the hue tag, the color codes of the similar colors can be defined as the similar range.

[0147] Embodiment 4

[0148] In Embodiment 3, in order to reduce the steps that the user has to take to select the previously captured images to be compared with the live image, the relevant previously captured images related to the representative image selected by the user are displayed on the display unit 109 as candidates for the previously captured images. On the other hand, in Embodiment 4, based on the acquisition conditions for acquiring the previously captured images selected (specified) by the user, candidates for the previously captured images are acquired from the recording unit 108 and displayed as candidates on the display unit 109.

[0149] In Embodiment 4, the user specifies the acquisition conditions for acquiring the candidates for the previously captured images, whereby the candidates for the previously captured images are narrowed down based on the features (tags) of the images related to the acquisition conditions. Since the candidates corresponding to the specified acquisition conditions are displayed, the user can reduce the steps that have to be taken to select the previously captured images. The structure of the imaging device according to Embodiment 4 is the same as Figure 1 that of the imaging device 100 according to Embodiment 1 shown. In Embodiment 4, the tag attachment process is performed in the same manner as in Embodiment 3. The description of the processing steps the same as those in Embodiment 3 is omitted.

[0150] Imaging processing according to Embodiment 4: Refer to Figure 9 and 12 The imaging processing of the imaging device 100 according to Embodiment 4 will be described. Except for Figure 9Except for the processing in step S901, the imaging process of the imaging device 100 according to Embodiment 4 is the same as the imaging process according to Embodiment 3.

[0151] Figure 12 is a flowchart illustrating the arrangement process of the previously captured image according to Embodiment 4. The arrangement process of the previously captured image is a sub-flow of the imaging process. Figure 12 The arrangement process of the previously captured image in illustrates the detailed process of step S901. In Figure 12 each step of, the same processing steps as Figure 10 are denoted by the same reference numerals, and their descriptions are omitted.

[0152] In step S1201, the control unit 101 receives an operation from the user for specifying acquisition conditions for acquiring a previously captured image. The acquisition conditions for acquiring a previously captured image are, for example, conditions for the composition of the image, conditions for the color tone of the image, and conditions for the subject of the image, etc.

[0153] will refer to Figures 13A to 13D a UI that is displayed on the display unit 109 to receive a specification of acquisition conditions for acquiring a previously captured image from the user. Figure 13A A UI that instructs the user to specify each item of the acquisition conditions for acquiring a previously captured image. The selection box 1301 is a UI for specifying the conditions for the composition. In response to a click operation performed by the user on the selection box 1301, the control unit 101 displays Figure 13B the selection items for the conditions for the composition indicated in. The control unit 101 receives a selection of the conditions for the composition according to the click operation of the user.

[0154] The selection box 1302 is a UI for specifying the conditions for the color tone. In response to a click operation performed by the user, the control unit 101 displays Figure 13C the selection items for the conditions for the color tone shown in. The control unit 101 receives a selection of the conditions for the color tone according to the click operation of the user.

[0155] The selection box 1303 is a UI for specifying the conditions for the subject. In response to a click operation performed by the user, the control unit 101 displays Figure 13D the selection items for the conditions for the subject shown in. The control unit 101 receives a selection of the conditions for the subject according to the click operation of the user.

[0156] In this way, the control unit 101 can receive an instruction from the user for specifying acquisition conditions for acquiring a previously captured image. The acquisition conditions for acquiring a previously captured image, the UI for specifying the acquisition conditions, and the operation method of the user are not limited to the above examples. The instruction for the acquisition conditions for acquiring a previously captured image can be received by the user using any method.

[0157] For example, the conditions for selecting a composition can be selected not from the name of the composition as shown in Figure 13B but from the illustration indicating each composition. The conditions for selecting a color tone can be selected not by specifying RGB as shown in Figure 13C but from the illustration indicating each color tone. In addition, the conditions for selecting a subject can be selected not from the name of the subject as shown in Figure 13D but from the illustration indicating each subject.

[0158] In step S1202, the control unit 101 determines whether the user has specified acquisition conditions for acquiring a previous captured image. If the acquisition conditions for acquiring a previous captured image are specified (step S1202: Yes), the process proceeds to step S1203. If the acquisition conditions for acquiring a previous captured image are not specified (step S1202: No), the process proceeds to step S1005. A case where the acquisition conditions for acquiring a previous captured image are not specified is, for example, a case where the user does not perform an operation of specifying the acquisition conditions for acquiring a previous captured image within a predetermined time.

[0159] In step S1203, the control unit 101 acquires relevant previous captured images from the recording unit 108 based on the acquisition conditions for acquiring a previous captured image specified by the user. For the relevant previous captured images, the control unit 101 acquires previous captured images having the following label: the value of the label is the same as the acquisition conditions for acquiring a previous captured image specified by the user. The processing steps from step S1004 to step S1007 are the same as those shown in Figure 10 the same.

[0160] In Embodiment 4, the control unit 101 displays candidates for previous captured images to be arranged in the template 401 based on the acquisition conditions for acquiring a previous captured image specified by the user. Thus, the imaging device 100 can reduce the steps that the user has to take to select a previous captured image to be compared with the live image.

[0161] Embodiment 5

[0162] In Embodiment 3 and Embodiment 4, the representative image selected by the user or the relevant previous captured images related to the acquisition conditions for acquiring a previous captured image specified by the user are respectively displayed on the display unit 109 as candidates for previous captured images. On the other hand, in Embodiment 5, candidates for previous captured images are acquired from the recording unit 108 based on the features of the live image and displayed on the display unit 109 to reduce the steps that the user has to take to select a previous captured image to be compared with the live image.

[0163] Since the candidates of previously captured images are narrowed down based on the features of the live image, the user can reduce the steps required to select a previously captured image similar to the live image. The structure of the imaging device according to Embodiment 5 is the same as that of the imaging device 100 according to Embodiment 1 shown in Figure 1 In Embodiment 5, the tag attachment process is performed in the same manner as in Embodiment 3. The description of the processing steps that are the same as those in Embodiment 3 is omitted.

[0164] Imaging processing according to Embodiment 5: Refer to Figure 9 and 14 The imaging processing of the imaging device 100 according to Embodiment 5 will be described. Except for the processing in step S901 of Figure 9 , the imaging processing of the imaging device 100 according to Embodiment 5 is the same as the imaging processing according to Embodiment 3. Figure 14 is a flowchart illustrating the arrangement process of previously captured images according to Embodiment 5. The arrangement process of previously captured images is a sub-flow of the imaging process. Figure 14 The arrangement process of previously captured images in

[0165] illustrates the detailed processing of step S901. In step S1401, the control unit 101 receives an instruction from the user to update the display of the candidates of previously captured images. For example, the display unit 109 displays a UI such as an update button for receiving an instruction from the user to update the display of the candidates of previously captured images. The control unit 101 receives an instruction to update the display of the candidates of previously captured images through an operation in which the user clicks the update button.

[0166] In step S1402, the control unit 101 determines whether an instruction from the user to update the display of the candidates of previously captured images has been received. If an instruction from the user to update the display has been received (step S1402: Yes), the process proceeds to step S1403. If an instruction from the user to update the display has not been received (step S1402: No), the process proceeds to step S1005.

[0167] In step S1403, the control unit 101 extracts the features of the live image. The method of extracting features can be the same as the method of extracting features in the tag attachment process described in Embodiment 3. The control unit 101 can use the same method as the method of extracting features in the tag attachment process to obtain information related to the features of the live image (such as composition, tone, subject).

[0168] When the live image changes over time, the control unit 101 can analyze the features of the image using the live image at the timing when an instruction from the user to update the display of the candidates of previously captured images is received in step S1402.

[0169] In step S1404, the control unit 101 acquires relevant previously captured images from the recording unit 108 based on the features of the live image. For the relevant previously captured images, the control unit 101 acquires previously captured images having the following label: the value of the label is the same as the features of the live image extracted in step S1403.

[0170] In Embodiment 5, the control unit 101 displays candidates for the previously captured images to be arranged in the template 401 based on the live image. Thereby, the imaging device 100 can reduce the steps that the user has to take to select a previously captured image to be compared with the live image.

[0171] In the description of FIG. 14, the control unit 101 uses the live image at the timing when the user instructs to update the display of the candidates for the previously captured images to extract the features of the live image, but the present invention is not limited thereto. The control unit 101 may use the live image at the timing that satisfies a predetermined condition to extract the features of the live image. The timing that satisfies the predetermined condition is, for example, the timing when the amount of change of the live image due to the passage of time is equal to or less than a threshold value. Thereby, the imaging device 100 can reduce the steps that the user has to take to instruct to update the display of the candidates for the previously captured images. In addition, at the timing when the user determines the composition of the image to be captured, the features of the live image can be more easily extracted.

[0172] Embodiment 6

[0173] In Embodiment 3, the relevant previously captured images related to the representative image selected by the user are displayed as candidates for the previously captured images on the display unit 109. On the other hand, in Embodiment 6, the relevant previously captured images related to the representative image selected by the user are automatically arranged in the area of the template 401 to further reduce the steps that the user has to take to arrange the previously captured images.

[0174] In Embodiment 6, since the relevant previously captured images are automatically arranged in the area of the template 401, the user can reduce the steps that the user has to take to select and arrange the previously captured images. The structure of the imaging device according to Embodiment 6 is the same as that of the imaging device 100 according to Embodiment 1 Figure 1 shown. In Embodiment 6, the label attachment process and the representative image setting process are performed in the same manner as in Embodiment 3. The description of the processing steps that are the same as those in Embodiment 3 is omitted.

[0175] Imaging process according to Embodiment 6: Refer to Figure 9 and 15 The imaging process of the imaging device 100 according to Embodiment 6 will be described. Except for the processing in step S901 of Figure 9 , the imaging process of the imaging device 100 according to Embodiment 6 is the same as the imaging process according to Embodiment 3.Figure 15 This is a flowchart illustrating the arrangement process of previously captured images according to Embodiment 6. The arrangement process of previously captured images is a sub - flow of the imaging process. Figure 15 The arrangement process of previously captured images in [reference] illustrates the detailed process of step S901. In Figure 15 each step of [reference], the same processing steps as Figure 10 are denoted by the same reference numerals and their descriptions are omitted.

[0176] In step S1501, the control unit 101 arranges the relevant previously captured images obtained in step S1003 in the respective regions of the template 401. Specifically, among the relevant previously captured images, the control unit 101 acquires a plurality of relevant previously captured images that fill the open regions of the template 401, and arranges the acquired relevant previously captured images in the open regions of the template 401 respectively.

[0177] For example, in the case where there are three regions where previously captured images are not arranged, as in the case of the screen example in Figure 4C [reference], the control unit 101 acquires three relevant previously captured images from the recording unit 108 and arranges these images in the open regions.

[0178] The control unit 101 controls so that the live image arranged by the user is not automatically replaced by the relevant previously captured images. The control unit 101 arranges the relevant previously captured images in the regions of the template 401 where there is no live image arranged by the user.

[0179] The control unit 101 can control so that the previously captured images arranged by the user are not automatically replaced by the relevant previously captured images. The control unit 101 manages a flag to determine whether the currently arranged previously captured image is arranged by the user or automatically arranged by the control unit 101 in the process of step S1501. For example, if the currently arranged previously captured image is arranged by an instruction from the user in step S1006, the control unit 101 attaches a "0" flag, and if the image is arranged by the control unit 101 in step S1501, a "1" flag is attached. By referring to the value of the flag, the control unit 101 can control so that the previously captured images arranged by the user (flag == 0) are not replaced by the relevant previously captured images.

[0180] In addition, the control unit 101 can change the display format of the previously captured images in the current layout according to whether the previously captured images in the current layout were arranged by the user or by the control unit 101 in step S1501. By changing the display format, the user can easily distinguish whether the previously captured images in the current layout were arranged by the user or automatically arranged by the imaging device 100. For example, the control unit 101 can change the display format by the color of the frames of the respective rectangular regions of the template 401.

[0181] In the case where the number of relevant previously captured images is less than the number of open regions in the template 401, the control unit 101 can arrange only the relevant previously captured images obtained in step S1003. In this case, the control unit 101 can arrange the relevant previously captured images in order starting from the region closer to the live image. By arranging the relevant previously captured images in order starting from the region closer to the live image, the user can more easily compare the live image and the previously captured images.

[0182] In Embodiment 6, the control unit 101 automatically arranges the relevant previously captured images obtained based on the representative image selected by the user in the template 401. Thereby, the imaging device 100 can reduce the steps that the user has to take to select and arrange the previously captured images to be compared with the live image.

[0183] The relevant previously captured images arranged by the control unit 101 in the region of the template 401 in step S1501 can be replaced by other previously captured images for user operations in step S1006. Thereby, in the case where the relevant previously captured images arranged by the control unit 101 are not desired images, the user can replace the images with desired previously captured images.

[0184] Embodiment 7

[0185] When displaying images in a list, the image list can present a sense of consistency by arranging images with the same characteristics. If the images with the first characteristic and the images with the second characteristic are arranged in a regular order, the image list can also present a sense of consistency.

[0186] In Embodiment 6, the relevant previously captured images having the same characteristics as the representative image selected by the user are automatically arranged in the region of the template 401. On the other hand, in Embodiment 7, the previously captured images of multiple groups having mutually different characteristics are automatically arranged in a regular order.

[0187] In Embodiment 7, the previously captured images in multiple groups having different characteristics can be automatically arranged in a regular order. Therefore, the user can achieve various types of list displays with simple operations. The structure of the imaging device according to Embodiment 7 is the same as Figure 1It is the same as the imaging device 100 according to Embodiment 1 shown. The same processing steps as those in Embodiment 6 are denoted by the same reference numerals, and their description is omitted.

[0188] Imaging processing according to Embodiment 7: Refer to Figure 9 and 15 to describe the imaging processing of the imaging device 100 according to Embodiment 7. Aspects different from the imaging processing according to Embodiment 6 will be described in detail.

[0189] In Figure 9 at S302, the control unit 101 receives an instruction from the user to select a template in which the display positions of the previously captured images in the group having the first feature and the display positions of the previously captured images in the group having the second feature can be specified. Thereafter, the group having the first feature is referred to as the "first group", and the group having the second feature is referred to as the "second group".

[0190] Refer to Figure 16A and 16B to describe examples of the template for the display positions of the previously captured images. Figure 16A and 16B are examples of the templates for the display positions that the user can select. The multiple regions included in each template are divided into multiple groups and arranged in a regular order.

[0191] In Figure 16A and 16B , each rectangular region 1601 (filled rectangular region) is a rectangular region in the rectangular regions indicating the display positions of the template 401 where the images in the first group are arranged. Each rectangular region 1602 (hatched rectangular region) is a rectangular region in the rectangular regions indicating the display positions of the template 401 where the images in the second group are arranged.

[0192] In Figure 16A , the rectangular regions 1601 and the rectangular regions 1602 are arranged in a grid pattern in a regular order. In Figure 16B , the rectangular regions 1601 and the rectangular regions 1602 are arranged in a vertical stripe pattern in a regular order.

[0193] The control unit 101 displays the template of the display positions as shown in Figure 16A and 16B on the display unit 109 to present the template to the user. The control unit 101 receives the selection of the template from the user. The control unit 101 can make the display format of the rectangular region 1601 where the images in the first group are arranged different from the display format of the rectangular region 1602 where the images in the second group are arranged. Thus, the user can easily distinguish the group to which each rectangular region belongs.

[0194] In Figure 15In S1001, the control unit 101 receives a selection of a representative image from the user. For example, the user can move a desired representative image to one of the rectangular areas of the template 401 by a drag-and-drop operation. The control unit 101 detects the user's operation and arranges the representative image selected by the user in the area specified by the drag-and-drop operation. Thus, the control unit 101 can receive the user's selection and arrangement of the representative image.

[0195] In Figure 15 In step S1002, the control unit 101 determines whether the user has selected a representative image. If a representative image has been selected (step S1002: Yes), the control unit 101 records the feature group of the rectangular area in which the representative image is arranged in the recording unit 108, and the process proceeds to step S1003. If no representative image has been selected (step S1002: No), the process proceeds to S1005.

[0196] In step S1501 of FIG. 15, the control unit 101 automatically arranges the relevant previously captured image obtained in step S1003 in the rectangular area having the same feature group as the feature group of the rectangular area in which the representative image is arranged.

[0197] Reference will be made to Figures 17A to 17C describe an example of the live view display screen in which the relevant previously captured image is arranged. Figure 17A is an example of the template for the display position selected in step S302. Figure 16A In the state of step S303, the live image is arranged in the rectangular area 402 at the center of the template 401. In the state of step S1001, the representative image 1701 is arranged in the upper left rectangular area of the template 401.

[0198] In step S1501, the control unit 101 reads the feature group of the rectangular area in which the representative image 1701 is arranged from the recording unit 108. In Figure 17A the case where the rectangular area in which the representative image 1701 is arranged belongs to the first group. The control unit 101 arranges the relevant previously captured image obtained in step S1003 in the rectangular areas at other display positions belonging to the first group, which is the same feature group as the representative image.

[0199] Figure 17B Illustrated in Figure 17A the state where the control unit 101 arranges the relevant previously captured image in the rectangular areas at other display positions belonging to the first group. The control unit 101 controls so that the representative image and the live image arranged by the user are not replaced by the relevant previously captured image.

[0200] Figure 17CThis is an example of repeating the processing steps from step S1001 to step S1501 again, and Figure 17B The representative image and the related previously captured images in are also arranged in the rectangular areas in the second group. In this way, the control unit 101 can configure a live view display screen in which the features of the images in each feature group are consistent.

[0201] In the seventh embodiment, the control unit 101 can easily arrange the previously captured images in the multiple groups having mutually different features in the template 401. Thereby, the imaging device 100 can reduce the steps that the user has to take to arrange the previously captured images in the multiple feature groups in a regular order for comparison with the live image.

[0202] The control unit 101 can determine the conditions for the features of the images arranged in the rectangular areas in the second group based on the features of the images in the first group. The conditions for the images arranged in the rectangular areas in the second group are, for example, the condition that the subject in the composition is smaller than the composition of the images in the first group, or the condition that the hue is compatible with the hue of the images in the first group in terms of the color scheme.

[0203] The control unit 101 can also select the previously captured images belonging to each feature group based on the information about the tags related to the features of the images. Thereby, the imaging device 100 can reduce the steps that the user has to take to arrange the previously captured images in the multiple feature groups in a regular order for comparison with the live image.

[0204] Embodiment 8

[0205] In the eighth embodiment, the sizes of the respective display areas (the respective areas of the template) of the previously captured images and the live image are adjusted so that the details (such as focus, viewing angle) of the live image can be more easily checked. While checking the details of the live image, the user can adjust the composition, brightness, and hue of the live image based on the previously captured images before capturing the live image.

[0206] In addition to Figure 1 the structure of the imaging device 100 shown, the structure of the imaging device according to the eighth embodiment further includes an acceleration sensor for measuring the moving speed of the imaging device 100. Reference will be made to Figure 18 and Figures 19A to 19F to describe the characteristic processing according to the eighth embodiment, and the description of the operations and processing steps the same as those in the first embodiment is omitted.

[0207] Imaging processing according to the eighth embodiment: Reference will be made to Figure 18 to describe the imaging processing of the imaging device 100 according to the eighth embodiment. Except for step S1801, the processing steps of the imaging processing according to the eighth embodiment are the same as those in Figure 3The flowchart is the same as that in [reference], so the same reference numerals are used to denote the same processing steps, and the description thereof is omitted.

[0208] In step S1801, the control unit 101 receives an operation for adjusting the imaging conditions so that the previously captured image and the live image arranged have the same composition, brightness, and color tone. The control unit 101 also receives an operation for the user to check details (such as focus and perspective) of the live image.

[0209] The imaging conditions are image processing conditions for changing composition, exposure conditions, color, and grayscale, etc. As an example of adjusting the imaging conditions, the method of adjusting the composition will be described with reference to Figure 5A and 5B Description of the method for adjusting the composition.

[0210] Figure 5A and 5B are examples of the live view display screen in the case where eight previously captured images are selected and arranged in step S304. In Figure 5A and 5B The zoom bar 501 is a UI for changing the zoom factor of the live image and is a UI for capturing an image by pressing or touching the imaging button 502.

[0211] In Figure 5A In the live view display screen, the toast in the live image arranged in the rectangular area 402 is displayed in a size smaller than the toast of the previously captured images surrounding the live image. If an image is captured in this state, the subject sizes between the live image and the previously captured images are different, so the list display of these images does not provide a sense of consistency, which is not desirable.

[0212] To match the subject size of the live image with the subject size of the previously captured images, the user performs an operation to increase the zoom factor of the live image. In the case of the live view display screen in Figure 5B The user can increase the zoom factor by moving the tab of the zoom bar 501 in the + direction to increase the subject size of the live image. By the zoom operation, Figure 5A The subject size of the live image in Figure 5B (which is smaller than the subject size of the previously captured images surrounding the live image) is increased to the same subject size as the subject size of the previously captured images surrounding the live image as shown in

[0213] By changing the zoom factor, the subject sizes of multiple images become similar, and the list display of multiple images on the live view display screen after imaging becomes a list display with a sense of consistency.

[0214] As an example of checking details of the live image, it will be described with reference to Figures 19A to 19CDescribe a method for checking the focus of a live view image. Figure 19A The following is an example of a live view display screen: The live image is arranged in the center, and eight previously captured images are arranged around the live image. In Figure 19A , the display areas of the live image and the previously captured images are the same size. Making the display areas of the live image and the previously captured images the same size is suitable for adjustment to provide a sense of consistency.

[0215] However, at each size of the equally divided area, it is difficult to check whether the live image is in focus, and the user may capture out-of-focus images. To easily check the focus of the live image, for example, the control unit 101 can receive a Pinch-Out operation from the user and increase the rectangular area 402 to display the live image, as shown in Figure 19B and 19C in the live view display screen. Therefore, it becomes easier to check whether the live image is in focus.

[0216] When checking the focus, the user performs a pinch operation. When a pinch operation is received from the user, the control unit 101 returns the size of the rectangular area 402 that displays the live image to the original size, as shown in Figure 19A in the live view display screen. By changing the display size of the live image according to the user operation, the user can easily check the focus.

[0217] A method for checking the viewing angle will be described with reference to Figure 19B and 19C . For example, in the case where the imaging device 100 is moved by the user at a predetermined threshold speed or higher, the control unit 101 determines that the user is checking the general viewing angle of the live image. The speed of the imaging device 100 can be measured by an acceleration sensor included in the imaging device 100.

[0218] If it is determined that the user is checking the viewing angle of the live image, the control unit 101 automatically increases the size of the rectangular area 402 that displays the live image, as shown in Figure 19B and 19C . By changing the display size of the live image, the user can easily check the viewing angle.

[0219] When the speed of the imaging device 100 becomes less than the predetermined threshold, the control unit 101 determines that the user's check of the viewing angle has ended. If it is determined that the user's check of the viewing angle has ended, then as shown in Figure 19A in the live view display screen, the control unit 101 automatically returns the size of the rectangular area 402 that displays the live image to the original size. By changing the size of the rectangular area 402 that displays the live image according to the speed of the imaging device 100, the user can easily check the general viewing angle.

[0220] In Embodiment 8, the control unit 101 adjusts the size of the rectangular region 402 that displays the live image in step S1801 so that the user can check the details of the live image. Thus, the user can easily check the details of the live image (such as focus and general perspective, etc.).

[0221] In the case of increasing the size of the rectangular region 402 that displays the live image, the number of previously captured images and their arrangement in the live view display screen in Figure 19B and 19C are the same as those in Figure 19D However, the present invention is not limited thereto. For example, in the example of Figure 19D , the live image is displayed in an enlarged state, and a part of the previously captured images is displayed side by side in a reduced state below the live image. In the example of Figure 19E , the live image is displayed in an enlarged state in the center, and the previously captured images are displayed side by side in a reduced state above and below the live image. In the example of Figure 19F , the live image is displayed in an enlarged state in the center, and the previously captured images are not displayed. As shown in Figure 19D and 19F , the number of previously captured images and their arrangement can be freely changed according to the size of the live image.

[0222] Other Embodiments

[0223] Embodiments 1 to 8 (including modification examples) are merely examples, and structures achieved by appropriately modifying or changing the structures of the respective embodiments within the scope of the essence of the present invention are included in the present invention. Results achieved by appropriately combining the structures of the respective embodiments are also included in the present invention.

[0224] For example, Embodiments 1 to 8 can be combined with Embodiment 2. In this case, the imaging device 100 displays the live image and the previously captured images obtained from an external server (e.g., SNS) on the live view display screen during imaging.

[0225] Specifically, in the case of combining Embodiment 8 and Embodiment 2, the imaging device 100 displays the previously captured images obtained from the external server together with the live image on the display unit 109. The imaging device 100 adjusts the size of the region that displays the live image so that the user can check the details of the live image. The user can adjust the composition, brightness, and color tone of the live image according to the previously captured images obtained from the external server, and capture the live image.

[0226] Other Embodiments

[0227] Embodiments of the present invention can also be implemented by the following method, that is, software (program) that executes the functions of the above embodiments is provided to a system or device through a network or various storage media, and a method in which a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0228] According to the present invention, it is possible to easily generate an image group including a plurality of images with a sense of consistency.

[0229] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the following claims should be given the broadest interpretation to cover all such modifications as well as equivalent structures and functions.

Claims

1. An imaging device, comprising at least one memory and at least one processor, the at least one memory and the at least one processor being configured to function as: A display unit configured to display a live image of a subject and a plurality of previously captured images on a display; An imaging unit configured to capture an image of the subject; A changing unit configured to change an imaging condition of the subject based on the plurality of previously captured images displayed together with the live image of the subject on the display, wherein the display unit displays the live image of the subject captured under the imaging condition changed by using the changing unit together with the plurality of previously captured images on the display; And A recording unit configured to record the captured image of the subject captured by the imaging unit and the plurality of previously captured images in association with layout information in a recording medium, the layout information including a display position of the captured image of the subject and a display position of each of the plurality of previously captured images.

2. The imaging device according to claim 1, wherein, The at least one memory and the at least one processor are further configured to function as: A determining unit configured to determine respective display positions of the live image of the subject and the previously captured images.

3. The imaging device according to claim 1, wherein The display unit is configured to adjust the live image of the subject according to the imaging condition changed by the changing unit and display the adjusted live image on the display.

4. The imaging device according to claim 1, wherein The imaging condition includes at least one of the following items: a condition related to a focal length, a condition related to a subject distance, a condition related to a depth of focus, and a condition related to an exposure.

5. The imaging device according to claim 1, wherein The display unit is configured to display the captured image of the subject and the plurality of previously captured images on the display based on the layout information.

6. The imaging device according to claim 1, wherein, The at least one memory and the at least one processor are further configured to function as: An uploading unit configured to upload the captured image of the subject and the plurality of previously captured images together with the layout information recorded in the recording medium to an external server.

7. The imaging device according to claim 1, wherein The display position of the captured image of the subject and the display position of each of the plurality of previously captured images are regions respectively selected from a plurality of pre-divided regions.

8. The imaging device according to claim 1, wherein The plurality of previously captured images are images selected from images obtained from the recording medium.

9. The imaging device according to claim 1, wherein The plurality of previously captured images are images selected from images obtained from an external server.

10. The imaging device according to claim 9, wherein, The at least one memory and the at least one processor are further configured to function as: A transmitting unit configured to transmit the layout information to the external server, wherein the layout information includes a display position of the captured image of the subject and a display position of each of the plurality of previously captured images.

11. The imaging device according to claim 10, wherein, the display unit is configured to display the captured image of the subject and the plurality of previously captured images on the display based on the arrangement information acquired from the external server.

12. The imaging device according to claim 1, wherein, the display unit is configured to display relevant previously captured images as candidates for the plurality of previously captured images, the relevant previously captured images being acquired based on the features of the representative image selected by the user.

13. The imaging device according to claim 1, wherein, the display unit is configured to display relevant previously captured images as candidates for the plurality of previously captured images, the relevant previously captured images being acquired based on the acquisition conditions specified by the user.

14. The imaging device according to claim 1, wherein, the display unit is configured to display relevant previously captured images as candidates for the plurality of previously captured images, the relevant previously captured images being acquired based on the features of the live image of the subject.

15. The imaging device according to claim 12, wherein, the display unit is configured to display a template on the display that is divided into a plurality of regions for displaying the live image of the subject and the plurality of previously captured images, and automatically arrange previously captured images selected from the relevant previously captured images in the regions of the plurality of regions where the live image of the subject and the plurality of previously captured images are not arranged.

16. The imaging device according to claim 15, wherein, the plurality of regions of the template are divided into a plurality of groups, and the display unit is configured to automatically arrange the previously captured images selected from the relevant previously captured images acquired for each of the groups in the corresponding regions of the group.

17. The imaging device according to claim 1, wherein, the display unit is configured to change the display size of the live image of the subject based on a user operation or the moving speed of the imaging device.

18. A control method for an imaging device, the control method comprising: a display step of displaying a live image of a subject and a plurality of previously captured images on a display; an imaging step of capturing an image of the subject; a change step of changing the imaging conditions of the subject based on the plurality of previously captured images displayed together with the live image of the subject on the display, wherein the live image of the subject captured by using the imaging conditions changed in the change step is displayed together with the plurality of previously captured images on the display; and a recording step of recording the captured image of the subject captured in the imaging step and the plurality of previously captured images in association with arrangement information in a recording medium, the arrangement information including the display position of the captured image of the subject and the display position of each of the plurality of previously captured images.

19. A computer-readable storage medium storing a program which, when executed by a computer, implements the steps of the method according to claim 18.

20. A computer program product comprising a program which, when executed by a computer, implements the steps of the method according to claim 18.

Images