Imaging apparatus, imaging method, and program

Abstract

An imaging apparatus includes: a first acquisition unit configured to acquire a captured image captured by an imaging unit and a depth image indicating a subject distance; an analysis unit configured to analyze a recommended composition recommended by the imaging unit for capturing the subject image in a three-dimensional space defined by the depth image acquired by the first acquisition unit; a creation unit for creating a directional reference object as an object in the three-dimensional space so that the reference object is set in a direction and position capable of providing guidance for photographing an image using the recommended composition analyzed by the analysis unit; and a superimposing unit configured to superimpose and display the reference object on the captured image displayed on the display unit as a preview image.

Description

Technical Field

[0001] The present application relates to an imaging device, an imaging method and a program. Background Art

[0002] For imaging devices such as smartphones, there are existing systems that automatically determine a recommended composition for imaging through a predetermined algorithm and provide guidance on imaging direction, etc. for achieving the composition.

[0003] For example, the following scheme is disclosed in the prior art: after displaying the position where the recommended composition can be used to take an image in an overlaid manner on the display screen and roughly presenting the standing position of the photographer, guidance is provided through a plane virtual frame to achieve the recommended composition (for example, see patent document 1). In addition, a technology is also disclosed: identifying a model composition sample similar to the frame image in the layout state of the subject from multiple model composition samples, and guiding the user to move to the position of the subject in the model composition sample through guidance information such as lines or plane symbols to change the recommended composition (for example, see patent document 2). Furthermore, a technology is also disclosed: collecting and analyzing imaging data through a camera, analyzing the number and position of faces, etc., matching the analyzed number of faces with the faces contained in one or more pre-registered recommended posture photos, and judging whether the face position is consistent with the recommended posture photo (for example, see patent document 3).

[0004] Prior art literature:

[0005] [Patent Document 1] Japanese Patent Application No.: 2017-085433;

[0006] [Patent Document 2] Japanese Patent No.: 4844657;

[0007] [Patent Document 3] Japanese Patent No.: 6101397. Summary of the Invention

[0008] Conventional technologies aim to provide guidance by displaying flat symbols or by overlaying recommended compositions on images. However, such guidance does not take into account the three-dimensional movement of the imaging device, which makes it difficult to provide users with intuitive guidance on the position and imaging direction of the imaging device.

[0009] The present application is proposed in view of the above-mentioned problems, and its purpose is to provide an imaging device, an imaging method and a program that can provide users with intuitive guidance on the position and imaging direction of the imaging device.

[0010] In order to solve the above problems and achieve the purpose of the present application, the present application provides an imaging device, which includes: a first acquisition unit, configured to acquire a captured image captured by an imaging unit and a depth image indicating a distance to a subject; an analysis unit, configured to analyze, in a three-dimensional space defined by the depth image acquired by the first acquisition unit, a recommended composition recommended by the imaging unit for capturing the image of the subject; a creation unit, configured to create a directional reference object as an object in the three-dimensional space, so that the object is set in a direction and position that can provide guidance for capturing an image using the recommended composition analyzed by the analysis unit; and an overlay unit, configured to overlay the reference object on the captured image displayed on a display unit as a preview image.

[0011] According to one aspect of the present application, it is possible to provide a user with intuitive guidance of the position and imaging direction of an imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is an example diagram explaining a guidance method of a conventional imaging device;

[0013] Figure 2 is an example diagram explaining a guidance method of a conventional imaging device;

[0014] Figure 3 is an exemplary diagram showing a hardware configuration of an imaging device according to an embodiment;

[0015] Figure 4 is a diagram showing an example of a functional block configuration of an imaging device according to the present embodiment;

[0016] Figure 5 is a schematic diagram explaining a guidance method of the imaging apparatus according to the present embodiment, in which a reference object is displayed to achieve a recommended composition;

[0017] Figure 6 is a schematic diagram explaining a guidance method of the imaging apparatus according to the present embodiment, in which a reference object is displayed to achieve a recommended composition;

[0018] Figure 7 is a flowchart showing an example of the flow of automatic composition guidance processing by the imaging device according to the present embodiment;

[0019] Figure 8 is a schematic diagram explaining a guidance flow of the imaging apparatus according to the present embodiment, in which an animal is created and displayed as a reference object;

[0020] Figure 9 is a schematic diagram explaining a guidance flow of the imaging apparatus according to the first modification, in which a frame is created and displayed as a reference object;

[0021] Figure 10 is a schematic diagram for explaining a guiding operation of an imaging device according to a second modification example, in which an imaging device is created and displayed as a reference object; and

[0022] Figure 11 is a schematic diagram for explaining a directing operation of the imaging apparatus according to the third modification example, in which a source object and a target object are created and displayed. DETAILED DESCRIPTION

[0023] The following will refer to Figures 1 to 11 , detailed description of embodiments of the imaging device, imaging method, and program according to the present application. Furthermore, the present application is not limited to the following embodiments, and the constituent elements in the following embodiments include elements readily apparent to those skilled in the art, substantially identical elements, and so-called equivalent elements. Furthermore, various omissions, substitutions, changes, and combinations of the constituent elements may be made without departing from the scope of the following embodiments.

[0024] Guidance methods for traditional imaging equipment

[0025] Figure 1 and Figure 2 This is an example diagram explaining the guidance method of traditional imaging equipment. Figure 1 and Figure 2 Describes how conventional imaging equipment guides recommended composition.

[0026] For traditional imaging devices such as smartphones, when a predetermined algorithm automatically determines the recommended imaging composition and provides guidance on the position and imaging direction of the imaging device to achieve the composition, flat two-dimensional symbols (such as straight lines, arrows, etc.) are usually displayed as guidance information to guide the user's imaging device position and imaging direction.

[0027] like Figure 1 As shown, when a conventional imaging device 51, such as a smartphone, takes an object OB1 as a subject and determines a composition of photographing the object OB1 from the front as a recommended composition, it is assumed that a preview image 2001a is displayed on the display of the imaging device 51 before the imaging device 51 moves to achieve the composition. In this case, the object OB1a is displayed in the preview image 2001a as a photographed image of the object OB1, but since the recommended composition of photographing the object OB1 from the front has not yet been achieved, the imaging device 51 displays a guiding indicator object 10B1, such as an arrow shape, as a flat two-dimensional symbol on the preview image 2001a. Figure 1 As shown, the object OB1a is located at a lower position in the preview image 2001a, and therefore, the guidance indication object 10B1 is drawn as an upward arrow to guide the object OB1a to move upward in the preview image 2001a.

[0028] However, in reality, Figure 1 As shown, the user will move imaging device 51 downward, causing the presentation of object OB1a to transition to a frontal shooting state, as shown by object OB1b in preview image 2001b, which aligns with the recommended composition. That is, although the guidance indicator object 10B1 is displayed as an upward arrow on the preview screen, the imaging device 51 is actually moving downward, the two directions being opposite. This makes it difficult for the user to intuitively understand the direction in which imaging device 51 should be moved. In other words, although the flat, two-dimensional symbol displayed on the preview image is intended to provide guidance for shooting object OB1 from the front, the user is left to determine how to actually move imaging device 51. Therefore, the user is unable to accurately and intuitively convey the position and shooting direction to which imaging device 51 should be moved.

[0029] In addition, if Figure 2 As shown, when the conventional imaging device 51 takes the object OB1 as the subject and determines the composition of photographing the object OB1 from the front as the recommended composition, it is assumed that the preview image 2001c is displayed on the display of the imaging device 51 before the imaging device 51 moves to the position where the composition is realized. In this case, the object OB1c will be displayed in the preview image 2001c as the photographed image of the object OB1; however, since the recommended composition of photographing the object OB1 from the front is not realized, the imaging device 51 will display the guiding indicator object 10B2, such as an arrow, as a flat two-dimensional symbol on the preview image 2001c. Figure 2 As shown, the object OB1c is located at a higher position in the preview image 2001c, and therefore, the guidance indicator object 10B2 is drawn as a downward arrow to guide the object OB1c to move downward in the preview image 2001c.

[0030] Then, the user Figure 2 As shown, the imaging device 51 is moved downward and the imaging direction is adjusted upward, so that the presentation of the object OB1c is changed to a state of being photographed from the front, as shown in the preview image 2001d of the figure, which is consistent with the recommended composition. Figure 2 In the case shown, with Figure 1 Compared with the situation shown in FIG. 1 , the guide indicator object 10B2 in the preview image is displayed as a downward arrow, and the imaging device 51 actually moves downward. Figure 1 and Figure 2 The imaging direction of the imaging device 51 before moving is different, resulting in different directions of the guide indicator objects displayed on the preview image, while the actual moving direction of the imaging device 51 is downward. Therefore, the user may be confused about the direction in which the imaging device 51 should move, and it is difficult to intuitively understand.

[0031] In the embodiment described below, an imaging device will be explained that solves the problems of the traditional imaging device 51 caused by the above-mentioned guidance method, namely, it is difficult for the user to intuitively understand the direction in which the imaging device 51 should move, and the position and imaging direction to which the imaging device 51 should move cannot be correctly and intuitively conveyed to the user.

[0032] Hardware configuration of imaging equipment

[0033] Figure 3 1 is a diagram showing an example of the hardware configuration of the imaging device according to the present embodiment. Figure 3 The hardware configuration of the imaging device 1 in this embodiment is described.

[0034] like Figure 1 As shown, the imaging device 1 is an imaging device equipped with a camera function, such as a smart phone or a tablet terminal, etc. It should be noted that in this embodiment, the imaging device 1 is described by taking a smart phone as an example.

[0035] like Figure 1 As shown, the imaging device 1 includes an imaging module 501 , an image signal processor (ISP) 502 , a central processing unit (CPU) 503 , a random access memory (RAM) 504 , a flash memory 505 , an input module 506 and a display 507 .

[0036] Imaging module 501 includes, for example, two cameras. Operating as a stereo camera, imaging module 501 captures images captured by the two cameras as stereo images. Imaging module 501 then uses the parallax of the stereo images to create a depth image through stereo matching based on triangulation principles. It should be noted that imaging module 501 corresponds to an "imaging unit."

[0037] It should be noted that the process of creating a depth image from a stereo image can also be performed by the ISP 502 described later. In addition, the imaging module 501 is not limited to a configuration including two cameras. For example, it can also be configured to include a single camera, or a combination of a ranging sensor and a camera. In the case of a single camera configuration, a depth image can be acquired by treating two captured images captured at a predetermined distance apart as quasi-stereo images. Furthermore, in the case of a combination of a ranging sensor and a camera, a depth image can be generated using distance information indicating the distance to each object as a subject obtained by the ranging sensor and the captured image captured by the camera, or a depth image can be acquired using only the distance information obtained by the ranging sensor.

[0038] The ISP 502 is a processor responsible for performing various image processing on the images captured by the imaging module 501. For example, the ISP 502 performs various image processing tasks, including exposure adjustment, shading correction, white balance, gamma correction, etc.

[0039] The CPU 503 is an arithmetic device that controls the operation of the entire imaging device 1. The RAM 504 is a nonvolatile storage medium that serves as a work area for the CPU 503. It should be noted that the CPU 503, ROM 504, etc. can be configured by a system on chip (SoC) mounted on a circuit board.

[0040] The flash memory 505 is a non-volatile storage medium used to store programs executed by the CPU 503, image data captured by the imaging module 501, and various setting value data.

[0041] The input module 506 is a touch panel module, and the user can operate the imaging device 1 through touch operation. The input module 506 operates based on a capacitive sensing method, a resistive film method, or the like.

[0042] The display 507 and the input module 506 together form a unit, which can be a liquid crystal display, an organic electroluminescent (EL) display, etc., for displaying the image of the subject and various icons. It should be noted that the display 507 corresponds to the "display unit".

[0043] like Figure 1 As shown, the imaging device 1 further includes a codec 508 , a speaker 509 , a microphone 510 , a wireless communication module 511 , an antenna 511 a , a global navigation satellite system (GNSS) module 512 , an antenna 512 a , and an inertial measurement unit (IMU) 513 .

[0044] Codec 508 is a module for encoding the voice collected by microphone 510, decoding the voice data, and outputting it through speaker 509. Speaker 509 is an acoustic device that outputs sound based on the voice data decoded by codec 508. Microphone 510 is a sound collection device that collects and receives external sounds.

[0045] The wireless communication module 511 is a communication module that uses an antenna 511a to communicate wirelessly with an external device such as a server 2 via a network using a communication method such as Wi-Fi (registered trademark), the fourth generation mobile communication system (4G) or the fifth generation mobile communication system (5G).

[0046] The GNSS module 512 is a receiving device that receives positioning signals from positioning satellites based on the Global Navigation Satellite System (GNSS) through an antenna 512 a.

[0047] The IMU 513 is a sensor unit that integrates, for example, an accelerometer, a gyroscope, and a magnetic field sensor into a single package. The IMU 513 can acquire detection information (hereinafter referred to as "attitude information") such as the angle (imaging direction), movement speed, acceleration, and rotation rate of the imaging device 1.

[0048] like Figure 3 As shown, the ISP 502, CPU 503, RAM 504, flash memory 505, input module 506, display 507, codec 508, wireless communication module 511, GNSS module 512 and IMU 513 in the above-mentioned units are interconnected through a bus 515 so that they can communicate with each other.

[0049] It should be noted that Figure 1 The hardware configuration of the imaging device 1 shown is merely an example. Therefore, the imaging device may not necessarily include all of these components and may also include other components. For example, the imaging device 1 may include a graphics processing unit (GPU) for performing real-time image processing, a near-field communication circuit for performing near-field wireless communication, such as NFC (near field communication) or Bluetooth (registered trademark), and the like.

[0050] Configuration and operation of imaging device functional blocks

[0051] Figure 4 is a diagram showing an example of a functional block configuration of the imaging apparatus according to the present embodiment. Figure 5 and Figure 6 1 is a schematic diagram for explaining the guiding method of the imaging device according to the present embodiment, wherein a reference object is displayed to achieve a recommended composition. Figures 4 to 6 , the configuration and operation of the functional blocks of the imaging device 1 of this embodiment will be described.

[0052] like Figure 4 As shown, the imaging device 1 includes a first acquisition unit 101, a second acquisition unit 102, a third acquisition unit 103, a composition analysis unit 104, a reference object creation unit 105, a reference object superposition unit 106, a feedback unit 107, and a completion determination unit 108. It should be noted that the composition analysis unit 104, the reference object creation unit 105, the reference object superposition unit 106, and the completion determination unit 108 correspond to the "analysis unit," "creation unit," "superposition unit," and "determination unit," respectively.

[0053] The first acquisition unit 101 is a functional unit configured to acquire an image captured by the imaging module 501 and generate a depth image from the captured image. Furthermore, the first acquisition unit 101 displays the captured image as a preview image on the display 507 in response to user operations performed via the input module 506. It should be noted that when the imaging module 501 is configured to include a combination of a ranging sensor and a camera, the first acquisition unit 101 can acquire the captured image from the camera of the imaging module 501 and generate a depth image from the distance information obtained by the ranging sensor.

[0054] The second acquiring unit 102 is a functional unit configured to acquire position information based on the positioning signal received by the GNSS module 512. The position information here indicates the position of the imaging device 1 in a three-dimensional space.

[0055] The third acquisition unit 103 is a functional unit for acquiring posture information of the imaging device 1 detected by the IMU 513 .

[0056] The composition analysis unit 104 is a functional unit for analyzing the composition (recommended composition) recommended by the imaging device 1 for photographing a subject, such as an object, a person, or a scene. Specifically, the composition analysis unit 104 calculates the position and angle (imaging direction) of the imaging device 1 in the three-dimensional space defined by the depth image acquired by the first acquisition unit 101, based on the position information acquired by the second acquisition unit 102 and the posture information acquired by the third acquisition unit 103. Subsequently, the composition analysis unit 104 analyzes the recommended composition of the imaging device 1 based on the calculated position and angle (imaging direction) of the imaging device 1 in the three-dimensional space. For example, the composition analysis unit 104 analyzes the recommended composition according to predetermined composition rules by calculating the edges or subject features in the captured image or depth image; alternatively, the composition analysis unit 104 uses the three-dimensional spatial information defined by the depth image to distinguish between distant and near scenes, thereby analyzing a recommended composition with a well-balanced layout of each subject. It should be noted that the composition analysis unit 104 can also use other known methods to analyze the recommended composition. Therefore, through analysis of the recommended composition by the composition analysis unit 104, the position and angle (imaging direction) at which the imaging device 1 should capture the image in three-dimensional space can be determined. It should be emphasized that the composition analysis unit 104 does not necessarily need to use the position information acquired by the second acquisition unit 102 and the posture information acquired by the third acquisition unit 103 to analyze the recommended composition. The analysis can also be performed based solely on the depth image acquired by the first acquisition unit 101, without using the position information or posture information.

[0057] The reference object creation unit 105 is a functional unit for creating a directional reference object to provide guidance for capturing an image using the recommended composition analyzed by the composition analysis unit 104. The reference object created by the reference object creation unit 105 is set in the three-dimensional space defined by the depth image acquired by the first acquisition unit 101. For example, Figure 5 In the illustrated example, when the imaging device 1 uses object OB as the subject and, through analysis by the composition analysis unit 104, sets a composition that captures object OB from the front as the recommended composition, it is assumed that a preview image 1001a containing object OBa is displayed on the display 507, where object OBa is a captured image of object OB. Here, the reference object creation unit 105 creates a directional reference object SOBa, represented as a dog-shaped 3D object, as an example of a reference object. This allows the user to capture reference object SOBa from the front when searching for a recommended composition, i.e., the user can face the dog-shaped 3D object. In other words, since the recommended composition is a composition that captures object OB from the front, the reference object creation unit 105 creates the orientation and position of reference object SOBa so that the dog-shaped 3D object, i.e., reference object SOBa, faces forward, with its back positioned in front of object OB.

[0058] The reference object superimposing unit 106 is a functional module for superimposing the reference object created by the reference object creating unit 105 on the preview image so that the reference object is in the direction and position determined by the reference object creating unit 105. In this case, the preview image is the captured image acquired by the first acquiring unit 101 and is displayed based on the position information of the imaging device 1 acquired by the second acquiring unit 102 and the posture information acquired by the third acquiring unit 103. Figure 5 As shown, the reference object superimposing unit 106 superimposes and displays the dog-shaped three-dimensional object SOBa created by the reference object creating unit 105 on the preview image 1001 a .

[0059] Therefore, the reference object creation unit 105 creates a directional reference object for guiding the use of the recommended composition when capturing an image, and the reference object superimposition unit 106 superimposes and displays the reference object on the preview image. With this configuration, the user can move the imaging device 1 under guidance to capture the reference object superimposed on the preview image from the front, thereby facilitating intuitive movement of the imaging device 1.

[0060] In the above Figure 5 In the example shown, the user appears to be looking down at the dog-shaped stereoscopic reference object SOBa superimposed and displayed on the preview image 1001a from an upper position. Therefore, the user can intuitively move the imaging device 1 downward to photograph the dog-shaped stereoscopic object from the front. The user can thus set the imaging device 1 to a position and angle (imaging direction) that enables the user to photograph an image using the recommended composition. Specifically, Figure 5 As shown, when the user intuitively moves the imaging device 1 downward, in the preview image 1001b, the dog-shaped three-dimensional object appears to be photographed from the front of the reference object SOBb, thereby achieving the same view as the recommended structure. Figure 1 A consistent composition in which the presentation of the object OB looks like the object OBb is photographed from the front.

[0061] In addition, Figure 6 In the illustrated example, when the imaging device 1 captures an object OB as the subject and, through analysis by the composition analysis unit 104, sets a composition that captures object OB from the front as the recommended composition, it is assumed that a preview image 1001c containing object OBc is displayed on the display 507. Object OBc is the captured image of object OB. Here, the reference object creation unit 105 creates a directional reference object SOBa, represented as a dog-shaped 3D object, as an example of a reference object so that the user can capture reference object SOBc from the front when searching for a recommended composition. In other words, since the recommended composition captures object OB from the front, the reference object creation unit 105 creates the orientation and position of reference object SOBc so that the dog-shaped 3D object, i.e., reference object SOBc, faces forward, with its back positioned in front of object OB. Subsequently, the reference object superimposition unit 106 superimposes and displays the dog-shaped 3D object SOBc created by the reference object creation unit 105 on the preview image 1001c.

[0062] In this case, although the object OBc is located in the upper area of the preview image 1001c, the user looks down at the dog-shaped stereoscopic reference object SOBc superimposed and displayed on the preview image 1001c from an upper position. Therefore, the user can intuitively move the imaging device 1 downward while adjusting the imaging direction upward to photograph the dog-shaped stereoscopic object from the front. Therefore, the user can set the imaging device 1 to a position and angle (imaging direction) that can capture an image using the recommended composition. Specifically, as Figure 6 As shown, when the user intuitively moves the imaging device 1 downward and adjusts the imaging direction upward, the dog-shaped three-dimensional object in the preview image 1001d looks like the reference object SOBd photographed from the front, thereby achieving the same appearance as the recommended structure. Figure 1 A consistent composition in which the object OB is presented as if the object OBd is photographed from the front.

[0063] It should be noted that although Figure 5 and Figure 6The example uses a dog-shaped 3D object as a reference object, but the reference object is not limited to this. The reference object can be any object with directionality, such as other animals, people, or objects. This application does not assume that a non-directional spherical object is a reference object. However, if a pattern or coating is added to the surface of a spherical object to identify directionality, such an object can be used as a reference object.

[0064] The feedback unit 107 is a functional unit that, when the reference object displayed superimposed by the reference object superimposing unit 106 guides the user to move the imaging device 1, displays a message or other information on the display 507 to provide auxiliary guidance instructions. In this case, the feedback unit 107 can change the content of the message displayed on the display 507 based on the distance from the position where the recommended composition is achieved or the imaging direction. In addition, when the completion determination unit 108 determines that the imaging device 1 is at a position and angle (imaging direction) that can capture an image with the recommended composition, the feedback unit 107 displays this information on the display 507. It should be noted that the feedback unit 107 does not always need to display a message to assist in the guidance instructions.

[0065] The completion determination unit 108 is a functional unit for determining whether the imaging device 1 is in a position and angle (imaging direction) at which an image can be captured with the recommended composition. For example, when the composition analysis unit 104 analyzes the recommended composition, it stores information about the position and angle (imaging direction) at which the imaging device 1 can capture the image with the recommended composition in the RAM 504 or flash memory 505. The completion determination unit 108 can be configured to determine whether the position and angle of the imaging device 1, as identified by the position information acquired by the second acquisition unit 102 and the posture information acquired by the third acquisition unit 103, are consistent with the position and angle stored in the RAM 504 and flash memory 505, respectively, to achieve the recommended composition calculated by the analysis of the composition analysis unit 104. In this case, the completion determination unit 108 is not required to determine that the position and angle of the imaging device 1 are exactly the same as the position and angle that achieve the recommended composition. The completion determination unit 108 can determine that the imaging device 1 is in a position and angle at which an image can be captured with the recommended composition when the position and angle of the imaging device 1 are within a predetermined range centered on the position and angle that achieve the recommended composition. It should be noted that the above determination method of the completion determination unit 108 is only an example, and other methods may also be used for determination.

[0066] like Figure 4 The first acquisition unit 101, the second acquisition unit 102, the third acquisition unit 103, the composition analysis unit 104, the reference object creation unit 105, the reference object superposition unit 106, the feedback unit 107 and the completion determination unit 108 in the imaging device 1 are all obtained by Figure 1The CPU 503 shown executes a program to implement the above. It should be noted that at least one of the above functional units can be implemented by hardware such as an integrated circuit. In addition, at least one functional unit can be implemented by the processing of the ISP 502.

[0067] It should be noted that Figure 4 The functional units of the imaging device 1 shown conceptually represent functions and are not limited to this configuration. For example, Figure 4 Multiple functional units shown as separate functional units may be integrated into a single functional unit; conversely, Figure 4 The one functional unit shown in can also be split into two or more parts by processing the functions of the module, thereby being configured as multiple functional units.

[0068] Automatic composition guidance processing for imaging devices

[0069] Figure 7 : is a flowchart showing an example of the flow of automatic composition guidance processing by the imaging apparatus according to the present embodiment. Figure 8 1 is a schematic diagram explaining the guidance flow of the imaging device according to the present embodiment, in which an animal is created and displayed as a reference object. Figure 7 and Figure 8 The flow of the automatic composition guidance process of the imaging device 1 of this embodiment will be described.

[0070] Step S11

[0071] The first acquisition unit 101 of the imaging device 1 acquires images captured by the imaging module 501 and a depth image generated from the captured images. In response to user operations performed via the input module 506, the first acquisition unit 101 displays the acquired images as preview images on the display 507. The second acquisition unit 102 of the imaging device 1 acquires position information based on positioning signals received by the GNSS module 512. The third acquisition unit 103 acquires attitude information of the imaging device 1 detected by the IMU 513.

[0072] Figure 8 (a) shows a state where the captured image acquired by the first acquisition unit 101 is displayed as a preview image on the display 507 of the imaging device 1. The preview image includes the object OB10 as the subject of the image. In addition to the preview image, the display 507 also includes an operation button 1011 and an automatic composition button 1012.

[0073] The operation button 1011 is a button for releasing the shutter. It should be noted that the operation button 1011 is not limited to obtaining a static image by releasing the shutter, and the operation button 1011 can also start or end video imaging.

[0074] The automatic composition button 1012 is a button for switching the recommended composition analysis and guidance processing using a reference object between active and inactive states.

[0075] It is assumed here that the user sets the recommended composition analysis and the guidance processing using the reference object to be active by operating the automatic composition button 1012. The flow then proceeds to step S12.

[0076] Step S12

[0077] When the recommended composition analysis and guidance processing using a reference object are activated via the auto composition button 1012, the composition analysis unit 104 of the imaging device 1 performs a process (i.e., composition analysis processing) of analyzing a composition (recommended composition) recommended by the imaging device 1 for capturing an image of a subject such as an object, person, or scene using the aforementioned method. The process then proceeds to step S13.

[0078] Step S13

[0079] The reference object creation unit 105 of the imaging device 1 creates a directional reference object to provide guidance for capturing an image using the recommended composition analyzed by the composition analysis unit 104. Specifically, Figure 8 As shown in FIG. 5( b ), the reference object creation unit 105 creates a dog-shaped 3D object SOB10 as an example of a reference object so that the user can photograph the reference object SOB10 from the front when searching for the recommended composition analyzed by the composition analysis unit 104. That is, the user can face the dog-shaped 3D object. The process then proceeds to step S14.

[0080] Step S14

[0081] The reference object superimposing unit 106 of the imaging device 1 superimposes and displays the reference object created by the reference object creating unit 105 on the preview image, so that the reference object is in the direction and position determined by the reference object creating unit 105. Specifically, Figure 8 As shown in step (b), reference object superimposing unit 106 superimposes reference object SOB10 created by reference object creation unit 105 on the preview image, with reference object SOB10 positioned in the orientation and position determined by reference object creation unit 105. In this case, reference object superimposing unit 106 displays reference object SOB10 and removes the dog-shaped 3D object displayed on auto-compose button 1012. This allows the user to recognize that the dog-shaped 3D object displayed on auto-compose button 1012 appears to have been moved into the preview image as reference object SOB10. The process then proceeds to step S15.

[0082] Step S15

[0083] The feedback unit 107 of the imaging device 1 may display a message or the like on the display 507 to provide auxiliary guidance instructions when the user is guided to move the imaging device 1 by the reference object superimposed and displayed by the reference object superimposed display unit 106. Figure 8 As shown in (b), the feedback unit 107 displays an auxiliary message for guiding the imaging device 1 to reach the position and angle for achieving the recommended composition in the guidance message display area 1013. When the user moves the imaging device 1, the reference object SOB10 is photographed from the front thereof, and the position of the imaging device 1 is a certain distance away from the position for achieving the recommended composition, the feedback unit 107 may be as follows: Figure 8 As shown in FIG. 10(c), an auxiliary message is displayed in the guidance message display area 1013, instructing the user to approach (move forward) the reference object SOB 10. This, combined with the intuitive guidance provided to the user by the displayed reference object SOB 10, more accurately guides the user to the position that achieves the recommended composition. The process then proceeds to step S16.

[0084] Step S16

[0085] The completion determination unit 108 of the imaging device 1 determines whether the imaging device 1 is in a position and angle (imaging direction) at which it can take an image with the recommended composition. When the imaging device 1 is set to a position and angle (imaging direction) at which it can take an image with the recommended composition, it determines that the guidance is completed (Yes in step S16). In this case, to indicate that the guidance is completed, for example, the completion determination unit 108 indicates this state by lighting the operation button 1011, as shown in FIG. Figure 8 (d) As shown. In this way, the user can recognize that the imaging device 1 has been moved to the position where the recommended composition is achieved, and the user can release the shutter or start video imaging using the recommended composition by clicking the operation button 1011. It should be noted that when the completion determination unit 108 confirms that the imaging device 1 is in a position and angle where it can take an image with the recommended composition, the imaging device 1 can automatically release the shutter or automatically start video imaging. In addition, after the completion determination unit 108 determines that the guidance has been completed, as shown in FIG. Figure 8 As shown in (d), the reference object SOB10 can be deleted from the preview image, and the dog-shaped 3D object can be displayed again on the automatic composition button 1012. At the same time, the recommended composition analysis and guidance process using the reference object are set to inactive. It should be noted that after the completion determination unit 108 determines that the guidance is complete, the user can also manually set the recommended composition analysis and guidance process using the reference object to inactive by clicking the automatic composition button 1012.

[0086] On the other hand, if the imaging device 1 is not set to a position and angle (imaging direction) at which an image can be captured using the recommended composition (No in step S16), the process returns to step S11. It should be noted that once the composition analysis unit 104 has analyzed the recommended composition and determined the position and angle (imaging direction) at which an image can be captured using the recommended composition, no further analysis is required; therefore, the subsequent processing of step S12 can be skipped. Furthermore, once the reference object creation unit 105 has created a directional reference object based on the recommended composition analyzed by the composition analysis unit 104, the reference object superimposition unit 106 can superimpose and display the reference object on subsequently captured images; therefore, the subsequent processing of step S13 can be skipped.

[0087] As described above, in the imaging device 1 according to this embodiment: the first acquisition unit 101 acquires a depth image generated based on the captured image captured by the imaging module 501; the composition analysis unit 104 analyzes the recommended composition for capturing the subject image within the three-dimensional space defined by the depth image; the reference object creation unit 105 creates a directional reference object in the three-dimensional space, as an object in the three-dimensional space, and places it in a direction and position that can provide guidance for capturing an image using the recommended composition analyzed by the analysis unit 104; and the reference object superimposition unit 106 displays the reference object as a superimposed preview image on the captured image on the display 507. This configuration allows the user to be provided with intuitive guidance on the position and imaging direction of the imaging device 1.

[0088] The first modification example

[0089] The following describes an imaging device 1 according to a first modified example, specifically explaining the differences from the imaging device 1 in the aforementioned embodiment. In the aforementioned embodiment, an example was described of creating and displaying a dog-shaped three-dimensional object as a directional reference object for guiding image capture using a recommended composition. In this modified example, however, the operation of creating and displaying a three-dimensional object of a different shape as a directional reference object will be described.

[0090] Figure 9 1 is a diagram explaining the imaging device guidance process according to the first modification, in which a frame is created and displayed as a reference object. Figure 9 Explanation: Creates and displays a frame-like three-dimensional object as a directional reference object, providing guidance for taking images with recommended compositions.

[0091] Figure 9 (a) shows a state where the captured image acquired by the first acquisition unit 101 is displayed as a preview image on the display 507 of the imaging device 1. The preview image includes the object OB11 as the subject. In addition to the preview image, the display 507 also includes an operation button 1011 and an automatic composition button 1012a.

[0092] The automatic composition button 1012 a is a slide switch for switching the recommended composition analysis and the guidance process using the reference object between activation and inactivation.

[0093] The user can activate the recommended composition analysis and guidance processing using a reference object by operating the automatic composition button 1012a. When the recommended composition analysis and guidance processing using a reference object are set to be activated by the automatic composition button 1012a, the composition analysis unit 104 of the imaging device 1 analyzes the recommended composition by the above-mentioned method. The reference object creation unit 105 of the imaging device 1 creates a directional reference object to provide guidance for capturing an image using the recommended composition analyzed by the composition analysis unit 104. In this modification, as shown in FIG. Figure 9 As shown in (b), the reference object creation unit 105 creates a directional reference object SOB11, which is expressed as a frame-shaped three-dimensional object, as an example of a reference object so that the user can capture the reference object SOB11 from the front when looking for a recommended composition. The frame-shaped three-dimensional object is an object that imitates the outer edge of the display area of the display 507. Figure 9 As shown in (b), the reference object superimposing unit 106 of the imaging device 1 superimposes and displays the reference object SOB11 created by the reference object creating unit 105 on the preview image, thereby setting the reference object SOB11 in the direction and position determined by the reference object creating unit 105. Figure 9 As shown in (b), the feedback unit 107 of the imaging device 1 displays an auxiliary message on the guidance message display area 1013 for guiding the imaging device 1 to a position and angle required to achieve the recommended composition.

[0094] Furthermore, when the user moves the imaging device 1 so that the reference object SOB11 of the frame-shaped three-dimensional object is photographed from the front thereof, and the position of the imaging device 1 is kept at a certain distance from the position where the recommended composition can be achieved, the feedback unit 107 may display an auxiliary message in the guidance message display area 1013, instructing the user to move closer to the reference object SOB11 (move forward), such as Figure 9 It should be noted that in this modified example, it is not always necessary to display auxiliary information in the guidance information display area 1013. Since the reference object SOB11 of the frame-shaped three-dimensional object is displayed as a reference object, it provides guidance to enable the user to intuitively align the frame with the outer edge of the display 507.

[0095] The completion determination unit 108 of the imaging device 1 determines whether the imaging device 1 is set at a position and angle (imaging direction) at which an image can be taken with the recommended composition. When the imaging device 1 is set at a position and angle (imaging direction) at which an image can be taken with the recommended composition, it is determined that the guidance is completed. In this case, to indicate that the guidance is completed, that is, the imaging device 1 is at a position and angle at which it can take a picture with the recommended composition, the completion determination unit 108 determines that the guidance is completed by, for example, Figure 9 (d) The operation button 1011 is lit to indicate this state. With this configuration, the user can release the shutter or start video imaging using the recommended composition by clicking the operation button 1011. It should be noted that when the completion determination unit 108 determines that the imaging device 1 is in a position and angle where it can capture an image with the recommended composition, the shutter can be automatically triggered or video imaging can be automatically started. In addition, as Figure 9 As shown in FIG. 5(d), after the completion determination unit 108 determines that the guidance is complete, the reference object SOB11 may be deleted from the preview screen, and the status of the auto-composition button 1012a may be set to inactive for the recommended composition analysis and guidance using the reference object. It should be noted that after the completion determination unit 108 determines that the guidance is complete, the user may also manually inactive the recommended composition analysis and guidance using the reference object by clicking the auto-composition button 1012.

[0096] As described above, in the imaging device 1 of this modification, the reference object creation unit 105 creates a frame-shaped solid object as a reference object that simulates the outer edge frame of the display area of the display 507. This can guide the user to intuitively align the frame-shaped solid object with the outer edge of the display 507.

[0097] Second Modification

[0098] The following describes an imaging device 1 according to a second modification, particularly different from the imaging device 1 in the above embodiment. In this modification, an operation example of creating and displaying a solid object representing the imaging device 1 itself as a directional reference object will be explained.

[0099] Figure 10 is a schematic diagram for explaining the guiding operation of the imaging device according to the second modification example, in which the imaging device is created and displayed as a reference object. Figure 10 It is explained how to create and display a three-dimensional object representing the imaging device 1 itself as a reference object that guides the user in achieving the directionality of the recommended composition.

[0100] The reference object creation unit 105 of the imaging device 1 can create a reference object SOB12 that simulates the three-dimensional shape of the imaging device 1 (here, a smartphone) as a directional reference object to guide the user to shoot according to the recommended composition analyzed by the composition analysis unit 104. Figure 10 As shown, the reference object superimposing unit 106 of the imaging device 1 superimposes the reference object SOB12 created by the reference object creating unit 105 with the preview image and displays it on the display 507 , thereby setting the reference object SOB12 in the direction and position determined by the reference object creating unit 105 .

[0101] Through this configuration, the user can be guided to align the imaging device 1 carried by the user with the three-dimensional object representing the imaging device 1 itself displayed on the display 507, thereby achieving more intuitive guidance.

[0102] The third modification

[0103] The following describes an imaging device 1 according to a third modified example, particularly the differences from the imaging device 1 of the aforementioned embodiment. This modified example explains the operation of creating and displaying a three-dimensional object having the same shape as the reference object in addition to creating and displaying a directional reference object before the imaging device 1 moves.

[0104] Figure 11 1 is a schematic diagram for explaining the guiding operation of the molding apparatus according to the third modification, in which a source object and a target object are created and displayed. Figure 11 An operation of creating and displaying a three-dimensional object (source object) having the same shape as that of a reference object (target object) in addition to creating and displaying a directional reference object (target object) before moving the imaging device 1 is explained.

[0105] Figure 11 1. A state is shown in which the captured image acquired by the first acquisition unit 101 is displayed as a preview image on the display 507 of the imaging device 1. The preview image includes the object OB13a and the object OB13b as subjects.

[0106] like Figure 11 As shown, the reference object creation unit 105 of the imaging device 1 creates a target reference object SOB13b as a directional reference object for providing guidance for capturing an image using the recommended composition analyzed by the composition analysis unit 104. Figure 11 In the illustrated example, for simplicity of explanation, it is assumed that the target reference object SOB13b is formed as a plate-like solid object overlapping one side of the object OB13b shaped like a cube.

[0107] In addition, if Figure 11As shown, the reference object creation unit 105 creates a source object SOB13a before the imaging device 1 is moved to a position where the recommended composition is achieved. The source object SOB13a is a three-dimensional object having the same shape as the target reference object SOB13b. In this case, the reference object creation unit 105 creates the source object SOB13a so that the display of the source object SOB13a is the same as the display of the target object SOB13b on the display 507 when the imaging device 1 is moved to a position where the recommended composition is achieved.

[0108] Subsequently, the reference object superimposing unit 106 of the imaging device 1 superimposes the target object SOB 13b and the source object SOB 13a created by the reference object creating unit 105 on the preview image, and displays them on the display 507 so that these objects are arranged in the direction and position determined by the reference object creating unit 105. In this case, the reference object superimposing unit 106 displays the source object SOB 13a, and the display of the source object SOB 13a on the display 507 is fixed even if the imaging device 1 moves. In other words, even if the imaging device 1 moves, the source object SOB 13a is displayed on the display 507 in a fixed manner.

[0109] With this configuration, the user can intuitively move the imaging apparatus 1 so that the source object SOB 13 a overlaps the target object SOB 13 b for providing guidance for capturing an image with the recommended composition, thereby achieving more intuitive guidance.

[0110] It should be noted that each function in the above-mentioned embodiments and modifications can be implemented by one or more processing circuits. Here, "processing circuit" includes: a processor programmed by software to perform the function, such as a processor implemented by an electronic circuit; and devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules, which are designed to perform the above-mentioned functions.

[0111] The program executed by the imaging device 1 in the above-described embodiment and modifications may be configured to be provided as being built in a ROM (Read Only Memory) or the like in advance.

[0112] Furthermore, according to the above-described embodiments and modifications, the program executed by the imaging device 1 may be configured to be provided in the form of a computer program product, recorded in a computer-readable storage medium such as a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), a CD-R (Compact Disc Recordable), and a DVD (Digital Versatile Disk) in an installable or executable file format.

[0113] Furthermore, the program executed by the imaging device 1 according to the above-described embodiment and the modified example may be configured to be stored on a computer connected to a network such as the Internet and provided by downloading the computer via the network. Furthermore, the program executed by the imaging device 1 according to the above-described embodiment may also be provided or distributed via a network such as the Internet.

[0114] Furthermore, the program executed by the imaging device 1 according to the above embodiment has a modular configuration including the above functional components. From the perspective of actual hardware, the CPU (processor) reads and executes the program from the ROM, so that the above functional components are loaded into the main memory and generated in the main memory.

[0115] Description of Reference Numerals

[0116] 1: Imaging device; 2: Server; 51: Imaging device; 101: First acquisition unit; 102: Second acquisition unit; 103: Third acquisition unit; 104: Composition analysis unit; 105: Reference object creation unit; 106: Reference object superposition unit; 107: Feedback unit; 108: Completion determination unit; 501: Imaging module; 502: Image signal processor (ISP); 503: Central processing unit (CPU); 504: Random access memory (RAM); 505: Flash memory; 506: Input module; 507: Display; 508: Codec; 509: Speaker; 510: Microphone; 511: Wireless communication module; 511a: Antenna; 512: Global Navigation Satellite System (GNSS) module; 512a: Antenna; 513: Inertial Measurement Unit (IMU); 515: Bus; 1001a to 1001d: Preview image; 1011: Operation button; 1012: Automatic composition button; 1013: Guidance message display area; 2001a to 2001d: Preview image; IOB1, IOB2: Guidance indicating objects; SOB10 to SOB12: Reference objects; SOB13a: Source object; SOB13b: Target object; SOBa to SOBd: Reference object; OB, OBa to OBd: Object; OB10, OB11, OB13a, OB13b: Object.

Claims

1. An imaging device comprising: a first acquiring unit, configured to acquire an image captured by the imaging unit and a depth image indicating a distance to a subject; an analyzing unit, configured to analyze, in a three-dimensional space defined by the depth image acquired by the first acquiring unit, a recommended composition for capturing the image of the subject recommended by the imaging unit; a creating unit configured to create a directional reference object as an object in the three-dimensional space, and to place the object in a direction and position that can provide guidance for capturing an image using the recommended composition analyzed by the analyzing unit; as well as The superimposition unit is configured to superimpose the reference object on the captured image displayed on the display unit to serve as a preview image.

2. The imaging device according to claim 1, further comprising: A feedback unit is configured to enable the display unit to display auxiliary guidance instructions when the reference object displayed by the superimposing unit guides the imaging device to move.

3. The imaging device according to claim 1 or 2, further comprising: a second acquiring unit, configured to acquire position information indicating a position of the imaging device; a third acquiring unit, configured to acquire posture information indicating a posture of the imaging device; as well as A determining unit is configured to determine whether the imaging device is in a position and imaging direction capable of capturing an image with the recommended composition based on the position information acquired by the second acquiring unit and the posture information acquired by the third acquiring unit.

4. The imaging device according to claim 3, wherein When the determination unit determines that the imaging device is in a position and imaging direction in which an image can be captured with the recommended composition, the determination unit causes the display unit to display this state.

5. The imaging device according to any one of claims 1 to 3, characterized in that The creation unit creates a frame-shaped three-dimensional object that simulates the outer edge frame of the display area of the display unit as the reference object.

6. The imaging device according to any one of claims 1 to 3, characterized in that The creation unit creates an object simulating a three-dimensional shape of the imaging device itself as the reference object.

7. The imaging device according to any one of claims 1 to 3, characterized in that The creation unit further creates a source object before the imaging device moves to a position where the recommended composition is achieved, so that the presentation of the source object is the same as the presentation of the reference object on the display unit when the imaging device is assumed to move to the position where the recommended composition is achieved; as well as The superimposing unit displays the source object superimposed on the preview image so that even if the imaging device moves, the representation of the source object on the display unit is fixed.

8. The imaging device according to any one of claims 1 to 7, characterized in that The imaging unit includes a single camera, dual cameras, or a combination of a ranging sensor and a camera.

9. An imaging method of an imaging device, comprising: acquiring a captured image captured by an imaging unit and a depth image indicating a distance to a subject; analyzing, in a three-dimensional space defined by the acquired depth image, a recommended composition for capturing the image of the subject recommended by the imaging unit; creating a directional reference object as an object in the three-dimensional space, and placing it in a direction and position that can provide guidance for capturing an image using the analyzed recommended composition; as well as The reference object is superimposed and displayed on the captured image displayed on the display unit as a preview image.

10. A computer-implemented program comprising: acquiring a captured image captured by an imaging unit and a depth image indicating a distance to a subject; analyzing, in a three-dimensional space defined by the acquired depth image, a recommended composition for capturing the image of the subject recommended by the imaging unit; creating a directional reference object as an object in the three-dimensional space, and placing it in a direction and position that can provide guidance for capturing an image using the analyzed recommended composition; as well as The reference object is superimposed and displayed on the captured image displayed on the display unit as a preview image.

Images