Mobile terminal device

Abstract

To provide an easy-to-use portable terminal device equipped with a foldable display which allows a user to easily confirm how the user looks in a picture without making unnecessary displays to save electricity.SOLUTION: A portable terminal device 100 with a camera for taking an image of an imaging target person 130 and a flexible foldable display has also a vision-of-line detection unit for detecting a line of vision of the imaging target person 130. The portable terminal device 100 displays a mirror image in the display unit on the imaging target person side 130 in a case where a direction of the line of vision of the imaging target person 130 detected by the vision-of-line detection unit is determined to be facing the display unit on an imaging target person side or the camera or to be facing at least the portable terminal device 100 itself when the portable terminal device 100 is being folded at the time of imaging.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a mobile terminal device having an imaging function for imaging a subject.

Background Art

[0002] Mobile terminal devices having functions such as a camera, a telephone, and information management such as memo and schedule can generally display images taken with a palm-sized camera and various information on a display screen, and can also make calls, and are widely used because of their good usability.

[0003] Conventional mobile terminal devices are provided with a flat panel display made of liquid crystal or the like on the surface of the mobile terminal device. In recent years, however, mobile terminal devices using a flexible display that is flexible and can be bent and folded have emerged. A mobile terminal device equipped with a foldable display can be used as a single display in an open state, enabling the display screen to be enlarged. As the flexible display, organic electro-luminescence (Organic EL) using a plastic substrate that is easy to bend is adopted instead of a glass substrate that is difficult to bend and is used in a liquid crystal panel or the like. As for the shape, in addition to the inward folding type in which there are displays on both inner sides of the folded mobile terminal device, an outward folding type in which there are displays on both outer sides when the mobile terminal device is folded has also appeared, and further spread is expected.

[0004] In addition, the high functionality of the imaging function by the camera is progressing more and more, and has achieved a leap-forward evolution such as an increase in the number of pixels, a triple lens configuration, and an increase in the magnification of the optical zoom, and anyone can easily take beautiful camera shots.

[0005] However, although there are many cases of being photographed by people or opportunities to photograph people, it is sometimes difficult to know how one is being photographed during the shooting, and it may not be known until it is shown after the shooting.

[0006] Regarding this point, there is a known technique of displaying the captured image of the camera on both the main display unit on the front side that the photographer views and the sub-display unit on the back side, and the subject checks the appearance by looking at the sub-display unit on the back side.

[0007] For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2004-15770) describes "In a portable terminal device equipped with a camera including a lens for imaging a subject and an imaging element, a first screen provided on the same side as the side where the lens is provided, a second screen provided on a side different from the side where the lens is provided, and means for displaying an image captured by directing the lens at the subject on the first screen and the second screen." Also, Patent Document 2 (Japanese Patent Application Laid-Open No. 2004-187182) describes "A first camera and a first display unit are provided on the front surface of the housing, a second camera and a second display unit are provided on the back surface of the housing, and when selecting an image captured by the first camera or the second camera and displaying it on the first display unit and the second display unit, a mirror image display is performed on the display unit arranged on the same side as the selected camera." That is, it is a technique of providing a sub-display unit visible to the subject so that the subject can visually recognize their own appearance.

Prior Art Documents

Patent Documents

[0008]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0009] In Patent Document 1 and Patent Document 2 described above, although it is described that a sub-display unit visible to the person being photographed is provided so that the person being photographed can visually confirm their own appearance, there is no description or suggestion regarding a portable terminal device equipped with a foldable display. Further, even when the person being photographed is not looking at the sub-display unit, the captured image of the camera is unnecessarily displayed on the sub-display unit, which causes an unnecessary increase in power consumption and shortens the operating time of the portable terminal device more than necessary. In addition, when the captured image is displayed on the sub-display unit visible from the person being photographed, the captured image will be shown to people who have nothing to do with the person being photographed.

[0010] An object of the present invention is to provide a portable terminal device having an imaging function, which can reduce power consumption without performing unnecessary display, and which is user-friendly in that the person being photographed can easily visually confirm how they are being imaged while being photographed.

Means for Solving the Problems

[0011] The portable terminal device includes an imaging unit configured to image a person being photographed, a first display unit configured to display an image captured by the imaging unit, a second display unit configured to display a processed image obtained by processing the image, a line-of-sight direction detection unit configured to detect the line-of-sight direction of the person being photographed, a determination unit configured to determine whether or not the person being photographed is facing the direction of the existence of the portable terminal device based on the detection result by the line-of-sight direction detection unit, and a control unit configured to control the display / non-display of the processed image on the second display unit based on the determination result of the determination unit. Here, the first display unit is provided at a position visible to the photographer, and the second display unit is provided at a position visible to the person being photographed. The control unit is configured to display the processed image on the second display unit when the determination unit determines that the person being photographed is facing the direction of the existence of the portable terminal device, while the control unit is configured to make the processed image non-displayed when the determination unit determines that the person being photographed is not facing the direction of the existence of the portable terminal device.

Effects of the Invention

[0012] According to one embodiment, in a mobile terminal device equipped with a foldable display, it is possible to reduce power consumption without performing unnecessary display, and the subject being photographed can easily visually recognize how they look and their appearance while being photographed, realizing a user-friendly mobile terminal device.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Modes for Carrying Out the Invention

[0014] In all the diagrams for explaining the embodiments, the same members are generally given the same reference numerals, and repeated explanations thereof are omitted. Note that, for the sake of clarity of the drawings, hatching may be added even in the case of a plan view.

[0015] (Embodiment 1) <External Configuration of the Portable Terminal Device> FIG. 1 is a diagram schematically showing the external configuration of the portable terminal device.

[0016] In FIG. 1, the portable terminal device 100 has a camera (imaging unit) 101, a depth sensor 102, a first left gaze detector 103a, a first right gaze detector 104a, a camera operation switch 105, and a foldable display 106.

[0017] The camera 101 is configured to photograph surrounding people and scenery.

[0018] The depth sensor 102 is a sensor that captures the shape of an object such as a person or an object in three dimensions and is configured to measure the distance and angle to the object.

[0019] The first left gaze detector 103a and the first right gaze detector 104a each have a function of detecting the movements of the left and right eyes of the subject 130 photographed by the camera 101, and can detect the gazes of both the left and right eyes to three-dimensionally identify the gaze destination. The arrangement of the first left gaze detector 103a and the first right gaze detector 104a is provided so as to be in the same arrangement direction as the arrangement direction of the left and right eyes of a generally upright subject when photographed by the camera 101, enabling more accurate gaze detection. Note that there may be one gaze detector, and when the subject is relatively far from the portable terminal device 100, performance close to binocular gaze detection can be obtained.

[0020] The camera operation switch 105 is a switch that the photographer presses when starting or stopping shooting with the camera 101.

[0021] When the portable terminal device 100 is folded for shooting, the camera 101 is folded in the direction facing the subject. The display 106 is divided into a display unit (second display unit) 107 visible from the subject side, a display unit (first display unit) 108 visible from the photographer side, and a non-display unit 109 that does not display when in the folded state at the folding hinge portion.

[0022] FIG. 2 is a diagram showing an overview seen from the photographer side when the portable terminal device is in a folded state. In the state shown in FIG. 2, when the photographer 140 operates the camera operation switch 105 to start shooting, the image 113 photographed by the camera 101 is displayed as an upright image as it is on the display unit 108 on the photographer 140 side of the display 106.

[0023] FIG. 3 is a diagram showing an overview as viewed from the subject side when the portable information terminal is in a folded state. Of the display 106, a processed image obtained by image processing the image 113 is displayed on the display portion 107 on the subject 130 side. Specifically, a mirror image 115 that is a left-right reversed mirror image of the image 113 is displayed on the display portion 107. From this, in a state where the portable terminal device 100 is folded, the photographer 140 can confirm the photographed subject 130 and the like by viewing the image 113 displayed on the display portion 108 on the photographer 140 side (see FIG. 2). On the other hand, the subject 130 can view his or her own image being photographed by viewing the mirror image 115 displayed on the display portion 107 on the subject 130 side. Therefore, the subject 130 can easily visually recognize how he or she appears while being photographed.

[0024] Note that instead of the camera operation switch 105 that starts shooting with the camera 101, a software button (not shown) displayed on the display portion 108 on the photographer 140 side may be used. It is also possible to display the image 113 instead of the mirror image 115 on the display portion 107 on the subject 130 side, and it may be possible to switch by input settings.

[0025] <Configuration of Portable Terminal Device> FIG. 4 is a block diagram showing a configuration example of the portable terminal device.

[0026] In FIG. 4, the mobile terminal device 100 is appropriately configured using a camera 101, a depth sensor 102, a first left eye gaze detector 103a, a second left eye gaze detector 103b, a first right eye gaze detector 104a, a second right eye gaze detector 104b, a camera operation switch 105, a display 106, a folding state detector 201, an acceleration sensor 202, a gyro sensor 203, a geomagnetic sensor 204, a control unit 205, a memory 208 storing a program 206 and information data 207, a touch panel 209, a microphone 210, a speaker 211, a vibration generator 212, a communication interface 213, and a transmission / reception antenna 214. Each component is interconnected via a bus 215 except for the transmission / reception antenna 214.

[0027] The camera 101 captures the visual field state of the front surroundings and can capture a subject in front regardless of the folding state.

[0028] The depth sensor 102 is a sensor that can capture the shape of an object such as a person or an object in three dimensions and can measure the distance and angle to the object. Examples of sensors used include LiDAR (Light Detection and Ranging) that irradiates an object with laser light such as infrared light, measures the scattered light that bounces back, and analyzes and detects the distance to an object at a long distance and the state of the object, a TOF (Time of Flight) sensor that measures the reflection time of pulsed light irradiated on a subject for each pixel to perform distance measurement, and a millimeter-wave radar that emits millimeter-wave radio waves, captures the reflected wave, and detects the distance to the object that is reflecting and the state of the object.

[0029] The depth sensor 102 is a configuration example of one of the person detection and both-eye identification processing means for detecting a person and identifying both eyes from the face of the detected person. This makes it possible to capture the subject as a person while the subject is being photographed by the camera 101 and to find and identify the left eye and the right eye in the face. The first left eye gaze detector 103a, the second left eye gaze detector 103b, the first right eye gaze detector 104a, and the second right eye gaze detector 104b detect the gaze by capturing the movement and direction of the right eye and the left eye, respectively.

[0030] Note that, for the process of detecting the line of sight, a technique generally used as eye-tracking processing can be utilized. For example, as a method using corneal reflection, an infrared LED (Light Emitting Diode) irradiates the face and is photographed by an infrared camera. Based on the position of the reflected light emitted by irradiating the infrared LED on the cornea (corneal reflection) as a reference point, a technique for detecting the line of sight based on the position of the pupil relative to the position of the corneal reflection is known. Also, a method is known in which by photographing the eyes with a visible light camera, using the inner corner of the eye as a reference point and the moving point as the iris, and detecting the line of sight based on the position of the iris relative to the inner corner of the eye.

[0031] As shown in FIG. 3, when the display 106 of the mobile terminal device 100 is folded horizontally, the first left-eye line-of-sight detector 103a and the first right-eye line-of-sight detector 104b that are horizontally arranged on the mobile terminal device 100 are generally in the same direction as the arrangement of both eyes of the subject photographed in the upright posture, and more accurate line-of-sight detection can be obtained.

[0032] The folding state detector 201 is composed of, for example, a pair of Hall elements and a pair of magnets, and detects whether or not both end faces of the mobile terminal device 100 are close to each other and the mobile terminal device 100 is in a folded state. The Hall element is a sensor that measures the magnetic flux density using the Hall effect, and when a magnet approaches, it can measure a strong magnetic flux density and detect the approach of the magnet. Also, as the folding state detector 201, in addition to the configuration using Hall elements, for example, a magnetoresistive element that utilizes the magnetoresistive effect in which the resistance value changes when a magnetic field is applied can also be used.

[0033] FIG. 5 is a diagram for explaining the detailed structure of the folding state detector 201.

[0034] As shown in FIG. 5, the mobile terminal device 100 includes a Hall element 301 and a Hall element 302 on one end face portion to be folded, and a magnet 303 and a magnet 304 on the other end face portion. The Hall element is a sensor that measures the magnetic flux density using the Hall effect, and can detect the approach of a magnet by measuring a strong magnetic flux density when the magnet approaches. When the mobile terminal device 100 is folded, both end faces of the mobile terminal device 100 approach each other. Thereby, the Hall element 301 measures a strong magnetic flux density from the magnet 303 and detects the approach of both end faces of the mobile terminal device 100. Similarly, the Hall element 302 measures a strong magnetic flux density from the magnet 304 and detects the approach of both end faces of the mobile terminal device 100. Therefore, the Hall elements 301 and 302 and the magnets 303 and 304 can detect whether both end faces of the mobile terminal device 100 are in an approaching state, that is, whether the mobile terminal device 100 is in a folded state.

[0035] Note that as the folding state detector 201 for detecting whether both end faces of the mobile terminal device 100 are close to each other and the mobile terminal device 100 is in a folded state, a configuration other than the configuration using the Hall element described in FIG. 5 can also be used. For example, a magnetoresistive element using the magnetoresistive effect in which the resistance value changes when a magnetic field is applied can also be used, or an electric switch incorporated in the hinge portion of the mobile terminal device 100 can also be used.

[0036] In FIG. 4, the acceleration sensor 202 is a sensor that captures movements, vibrations, impacts, etc., and can detect the inclination and direction of the mobile terminal device 100.

[0037] The gyro sensor 203 is a sensor that detects the angular velocity in the rotational direction and can capture the states of vertical, horizontal, and diagonal postures.

[0038] The geomagnetic sensor 204 is a sensor that detects the earth's magnetic force and detects the direction in which the mobile terminal device 100 is facing. In addition, a three-axis type that detects geomagnetism in the vertical direction as well as the front-rear and left-right directions is used to detect the movement of the mobile terminal device 100 by capturing changes in geomagnetism with respect to the movement of the mobile terminal device 100.

[0039] The control unit 205 is composed of a CPU or the like, and controls each component by executing programs 206 such as an operating system (OS) and operation control applications stored in the memory 208, thereby performing operation control processing for the entire mobile terminal device 100.

[0040] For example, when the camera 101 starts a shooting operation while the mobile terminal device 100 is in a folded state, if it is identified that the line of sight of the subject detected by the line-of-sight detector is directed at the display unit 107 on the subject side, the camera 101, or at least the mobile terminal device 100 itself, control is performed to display a mirror image 115 obtained by image processing the image captured by the camera 101 on the display unit 107 on the subject side. On the other hand, if it is identified that the line of sight of the subject is not directed at the display unit 107 on the subject side, the camera 101, or the mobile terminal device 100 itself, control is performed to turn off the display unit 107 on the subject side. In addition, control is performed to display the camera image 113 of the subject captured by the camera 101 as a normal image on the display unit 108 on the photographer side.

[0041] The memory 208 is composed of a flash memory or the like, and stores programs 206 such as an operating system and operation control applications used by the control unit 205. In addition, it stores information data 207 such as images, sounds, and signals handled by the mobile terminal device 100. Examples of the information data 207 include image data captured by the camera 101.

[0042] The display 106 is composed of a flexible display panel that can be folded and is arranged on both outer sides when the mobile terminal device 100 is folded. As the flexible display panel, an organic EL panel using a plastic substrate that is easy to bend is adopted, etc.

[0043] In the folded state of the mobile terminal device 100, separate images are displayed on the display unit 107 visible to the person being photographed and the display unit 108 visible to the photographer, respectively. On the other hand, in the opened state, it becomes a single display and a large display unit can be realized. Also, the display 106 performs the display of images and videos, the display of a screen indicating an input operation, the display of notification information to the user such as the battery remaining amount, various alarms, time, etc., and the icon display of applications launched within the display screen.

[0044] The touch panel 209 is composed of a highly transparent material and detects the position and area on the touched screen with a finger. The touch panel 209 is provided on the surface of the mobile terminal device 100 so as to overlap the upper layer of the display 106 and can detect the position touched on the display surface of the display 106.

[0045] Also, the touch panel 209 constitutes an input operation unit that can perform an input operation by touching the touch panel 209 while looking at the display object displayed on the display 106, and can detect an approach or contact operation by a finger or a touch pen as an operation input. For example, when a plurality of people being photographed are shown on the display unit 107 on the side of the person being photographed, the display position of a specific person being photographed can be touched and taken in as input information. Also, an icon indicating instruction information, an application, or an icon of a file on the display screen can be easily selected and specified as an input by a touch operation.

[0046] The microphone 210 collects external voices and the user's own voice.

[0047] Speaker 211 outputs sound externally and notifies the photographer or the like holding the mobile terminal device 100 of notification information or the like by voice. For example, when the subject's line of sight is detected and identified by the line-of-sight detector as being focused on the display unit on the subject side, the photographer can easily recognize the subject's line-of-sight state by being notified of this state by voice.

[0048] Vibration generator 212 generates vibration according to a control instruction from the control unit 205, converts the notification information transmitted by the mobile terminal device 100 into vibration, and can notify the photographer holding the mobile terminal device 100 of the notification information by vibration. For example, when the line-of-sight detector detects that the subject is looking at the display unit on the subject side, the photographer can surely recognize the subject's line-of-sight state by being notified of this state by vibration.

[0049] Communication interface 213 performs wireless communication with a server device or the like at another location by short-range wireless communication, wireless LAN, or base station communication. When performing wireless communication, it transmits and receives image data captured by the camera, control signals, etc. to and from the server device or the like via the transmission / reception antenna 214.

[0050] As the short-range wireless communication, for example, an electronic tag, Bluetooth (registered trademark), Wi-Fi (registered trademark), etc. can be used. Also, as the base station communication, long-range wireless communication such as LTE (Long Term Evolution) or GSM (Global System for Mobile communications, registered trademark) can be used. Although not shown, the communication interface 213 may use other methods such as optical communication or communication by sound waves as the means of wireless communication. In that case, an optical emission / reception unit or a sound wave output / sound wave input unit will be used instead of the transmission / reception antenna 214 respectively. For example, by wireless communication with the mobile terminal device 100 held by the subject, it is possible to transmit and receive various information such as the line-of-sight detection result, thereby improving the convenience of the photographer and the subject.

[0051] According to the mobile terminal device 100 configured as described above, when the mobile terminal device 100 is folded, based on the camera image captured by the camera 101, a processed image (for example, a mirror image) obtained by image-processing the camera image can be displayed on the display unit on the photographed person side. As a result, according to the mobile terminal device 100, the photographed person can view their own appearance displayed on the display unit on the photographed person side, and thus can visually confirm their own appearance during shooting.

[0052] <Functional Block Configuration of Mobile Terminal Device> Subsequently, in the mobile terminal device 100 having the above-described configuration, a functional block configuration focused on the functions realized by the mobile terminal device 100 will be described.

[0053] FIG. 6 is a diagram showing a functional block configuration of a mobile terminal device.

[0054] Here, in the functional block diagram shown in FIG. 6, not only the parts related to the first embodiment but also the parts related to other embodiments are described. The parts related to other embodiments will be described in the relevant embodiments.

[0055] The imaging unit 401 is built in the mobile terminal device 100 and is composed of, for example, a camera that images a photographed person. The image captured by the imaging unit 401 is stored in the data storage unit 415.

[0056] The first display unit 402 is configured to display the image captured by the imaging unit 401 and is provided at a position where the photographer can view it. For example, the first display unit 402 corresponds to the display unit 108 in FIG. 2.

[0057] The second display unit 403 is configured to display a processed image obtained by processing the image captured by the imaging unit 401 and is provided at a position where the photographed person can view it. For example, the second display unit 403 corresponds to the display unit 107 in FIG. 3.

[0058] In this specification, "image" means an image as captured by the imaging unit 401. On the other hand, "processed image" means an image obtained by subjecting the image captured by the imaging unit 401 to image processing. For example, the "processed image" corresponds to the image that has been subjected to image processing by the image processing unit 410. That is, the image processing unit 410 is configured to perform image processing on the image captured by the imaging unit 401. For example, in the image processing unit 410, a mirror image that is a left-right reversed image of the image captured by the imaging unit 401 is generated. The processed image that has been subjected to image processing by the image processing unit 410 is also stored in the data storage unit 415 in the same manner as the image captured by the imaging unit 401. That is, the data storage unit 415 is configured to store data such as images and processed images.

[0059] Basically, an image is displayed on the first display unit 402, but a processed image may also be displayed. Similarly, basically, a processed image is displayed on the second display unit 403, but an image may also be displayed.

[0060] The portable terminal device 100 is configured to be flexible and foldable, for example, as shown in FIG. 1. When the portable terminal device is not folded, the display unit 108 constituting the first display unit 402 and the display unit 107 constituting the second display unit 403 constitute a flat integrated display unit (see FIG. 1). On the other hand, when the portable terminal device 100 is folded, the display unit 108 constituting the first display unit 402 and the display unit 107 constituting the second display unit 403 are arranged at positions facing each other (see FIGS. 2 and 3).

[0061] Therefore, in the mobile terminal device 100, in order to be able to change the display form depending on whether it is folded or not, the mobile terminal device 100 is provided with a folding state detection unit 409 configured to detect whether the mobile terminal device 100 is folded or not. As a result, when the folding state detection unit 409 detects that the mobile terminal device 100 is not in a folded state, the mobile terminal device 100 can be configured to display an image on the integrated display unit. Further, when the folding state detection unit 409 detects that the mobile terminal device 100 is in a folded state, the mobile terminal device 100 can be configured to display an image on the display unit 108 constituting the first display unit 402 and to display a processed image on the display unit 107 constituting the second display unit 403. The mobile terminal device 100 is configured as described above.

[0062] <Operation of the mobile terminal device> The mobile terminal device 100 in the first embodiment is configured as described above, and the operation of the mobile terminal device 100 will be described below.

[0063] FIG. 7 is a flowchart for explaining the operation of the mobile terminal device.

[0064] In FIG. 7, when the camera operation switch 105 is pressed by the photographer and the camera 101 starts imaging surrounding people or scenery (S101), the folding state detection unit 409 detects whether the mobile terminal device 100 is in a folded state (S102).

[0065] When the folding state detection unit 409 detects that the mobile terminal device 100 is in a folded state, an image is displayed upright on the first display unit 402 (display unit 108) on the photographer side (S103). On the other hand, on the second display unit 403 (display unit 107) on the subject side, a processed image (mirror image) generated by performing image processing on the image by the image processing unit 410 is displayed (S104).

[0066] In this state, as an instruction to end without continuing the image display, it is determined whether it is set and input on the touch panel 209 by the touch operation of the photographer, or whether the camera operation switch 105 is pressed by the photographer (S106). When it is instructed to stop shooting, imaging is stopped (S107), the display of the image on the display unit 108 on the photographer side and the display of the processed image on the display unit 107 on the subject side are stopped (S108), and the sequence ends.

[0067] On the other hand, when it is not instructed to stop shooting (S106), the process returns to the determination sequence S102 as to whether the mobile terminal device 100 is in a folded state.

[0068] In addition, when the folding state detection unit 409 detects that the mobile terminal device 100 is in an open state rather than a folded state (S102), an image is displayed in a normal image on the entire display 106 (integrated display unit) (S105). That is, the image is displayed in a display area that combines the display unit 108 that is always displayed on the photographer side, the display unit 107 on the subject side that is displayed when folded, and the non-display unit 109 that is not displayed when folded at the folding part.

[0069] Next, the process proceeds to S106, and the subsequent operations are the same as those described above.

[0070] In this way, when imaging is started with the mobile terminal device 100 in a folded state, the image of the subject is displayed on the display unit 107 on the subject side. As a result, the subject can see their own image on the display unit 107 on the subject side, and thereby the subject can visually confirm how they are being photographed while being photographed.

[0071] In addition, while the imaging is continued, it is possible to shift from the folded state of the mobile terminal device 100 to the unfolded state, or from the unfolded state of the mobile terminal device 100 to the folded state. For example, if the photographer turns on the camera operation switch 105 to try to take a picture while folding the device, but wants to check the image on the large screen before taking a picture with the camera 101, the mobile terminal device 100 can be unfolded and opened. Furthermore, while the photographer is taking a picture with the camera 101 with the mobile terminal device 100 folded, the photographer can also check the image on the large screen in the unfolded state of the mobile terminal device 100 once without stopping the imaging operation.

[0072] (Embodiment 2) In this Embodiment 2, an example of controlling the display / non-display of a processed image on the display unit on the subject side by detecting the line of sight of the subject will be described.

[0073] <Overview of Display Control by Line-of-Sight Detection> In this Embodiment 2, when the mobile terminal device 100 is in the folded state and the photographer operates the camera operation switch 105 to start imaging with the camera 101, an image captured by the camera 101 is displayed on the display unit 108 on the photographer side. On the other hand, the display unit 107 on the subject side remains in the non-display state.

[0074] The depth sensor 102 detects the subject as a person while the subject is being photographed by the camera 101, and discriminates the left eye and the right eye from the face.

[0075] The first left-eye line-of-sight detector 103a and the first right-eye line-of-sight detector 104a respectively capture the movements and directions of the left eye and the right eye identified by the depth sensor 102, and detect the line-of-sight destination 114 (see FIG. 3) of the subject.

[0076] When the mobile terminal device 100 detects that the line-of-sight destination 114 detected by the first left line-of-sight detector 103a and the first right line-of-sight detector 104a is directed at the display unit 107 on the subject side, the camera 101, or at least the direction of the mobile terminal device 100 itself, the mirror image 115 is displayed on the display unit 107.

[0077] In this way, in the folded state of the mobile terminal device 100, only when the subject is gazing at the mobile terminal device 100 during imaging, the image of the subject being imaged can be displayed on the display unit 107 on the subject side as a mirror image 115 that is easy for the subject to view.

[0078] Therefore, unnecessary displays when the subject is not looking at the mobile terminal device 100 can be suppressed. For this reason, without wasting power consumption, the subject can easily and conveniently view how they look while being photographed.

[0079] Even if the depth sensor 102 detects the subject as a person, if the left and right eyes in the face cannot be found and identified, the subject is in a state of not looking at the mobile terminal device 100. In this case, the display unit 107 on the subject side is left in a non-display state with the display turned off, and no unnecessary display is performed.

[0080] Also, when the line-of-sight destination 114 detected by the first left line-of-sight detector 103a and the first right line-of-sight detector 104a is not directed at the display unit 107 on the subject side, the camera 101, or at least the direction of the mobile terminal device 100 itself, similarly, the display unit 107 on the subject side is left in a non-display state with the display turned off, and no unnecessary display is performed.

[0081] In the second embodiment, the case where the depth sensor 102 is used as a component for realizing the function of identifying both eyes from the face of the detected person has been described, but other components may be used.

[0082] For example, an image captured by the camera 101 may be used to identify a person or face from the image showing the person being photographed using an image recognition algorithm for the person or face, and the left eye and the right eye may be detected. Further, in this case, the gaze detector may also capture the eyes from the image and detect the gaze based on the position of the iris with respect to the eye corners, so that both the depth sensor 102 and the gaze detector can be all performed by software processing of the image. Also, the left eye gaze detector 103a and the right eye gaze detector 104a themselves may directly detect the left eye and the right eye.

[0083] <Functional block configuration> Subsequently, the functional block configuration for realizing the display control by gaze detection will be described with reference to FIG. 6.

[0084] In order to realize the display control by gaze detection, the mobile terminal device 100 includes a gaze direction detection unit 404, a determination unit 407, and a display control unit 408.

[0085] The gaze direction detection unit 404 is configured to detect the gaze direction of the person being photographed. For example, the identification unit 405 configured to identify the eyes of the person being photographed and the gaze direction detector 406 configured to detect the gaze direction of the person being photographed based on the movement of the eyes identified by the identification unit 405. In particular, the identification unit 405 includes the depth sensor 102 configured to three-dimensionally capture the shape, and the identification unit 405 is configured to identify the eyes of the person being photographed based on the output of the depth sensor 102.

[0086] Here, the depth sensor 102 constituting the identification unit 405 detects the person being photographed as a person in the state where the person being photographed is photographed by the camera 101, and identifies the left eye and the right eye from the face. However, the depth sensor 102 does not need to identify both the left eye and the right eye, and may be configured to detect only one of the left eye and the right eye.

[0087] The line-of-sight direction detector 406 is composed of, for example, a first left-eye line-of-sight detector 103a and a first right-eye line-of-sight detector 104a. These first left-eye line-of-sight detector 103a and first right-eye line-of-sight detector 104a are configured to capture the movements and directions of the left eye and the right eye identified by the depth sensor 102 respectively, and to detect the line-of-sight destination 114 (see FIG. 3) of the subject.

[0088] However, the line-of-sight direction detector 406 does not necessarily have to be composed of both the first left-eye line-of-sight detector 103a and the first right-eye line-of-sight detector 104a, and it may be composed of either one of the first left-eye line-of-sight detector 103a and the first right-eye line-of-sight detector 104a.

[0089] That is, the identification unit 405, which is a component of the line-of-sight direction detection unit 404, only needs to be configured to identify the eyes of the subject (either both eyes or one eye), and the line-of-sight direction detector 406, which is a component of the line-of-sight direction detection unit 404, only needs to be configured to detect the line-of-sight direction of the subject based on the movement of the eyes (either both eyes or one eye) identified by the identification unit 405.

[0090] Thus, not only when the line-of-sight direction detection unit 404 is composed of hardware, but also the line-of-sight direction detection unit 404 may be configured to detect the line-of-sight direction of the subject from the image software-wise by analyzing the image captured by the camera 101 through the image processing by the image processing unit 410.

[0091] Next, the determination unit 407 is configured to determine whether the subject is facing the direction of the presence of the mobile terminal device 100 based on the detection result by the above-described line-of-sight direction detection unit 404. For example, the determination unit 407 is configured to determine whether the line-of-sight direction of the subject detected by the line-of-sight direction detection unit 404 is the direction of the display unit 107 on the subject side, the camera 101, or at least the direction of the mobile terminal device 100 itself. Then, the determination unit 407 is configured to output the determination result regarding whether the subject is facing the direction of the presence of the mobile terminal device 100 to the display control unit 408.

[0092] Subsequently, the display control unit 408 is configured to control the display / non-display of the processed image (mirror image) on the display unit 107 on the photographed person side based on the determination result of the determination unit 407. For example, when the determination unit 407 determines that the photographed person is facing the direction of the presence of the mobile terminal device 100, the display control unit 408 displays the processed image on the display unit 107 on the photographed person side. On the other hand, when the determination unit 407 determines that the photographed person is not facing the direction of the presence of the mobile terminal device 100, the display control unit 408 is configured to turn off the display of the processed image.

[0093] The mobile terminal device 100 is configured as described above.

[0094] <Operation of the mobile terminal device> The mobile terminal device 100 in the second embodiment is configured as described above, and the operation of the mobile terminal device 100 will be described below.

[0095] FIG. 8 is a flowchart for explaining the operation of the mobile terminal device.

[0096] When the camera operation switch 105 is pressed by the photographer and the camera 101 starts shooting the surrounding people and scenery (S201), the folding state detection unit 409 detects whether the mobile terminal device 100 is in a folded state (S202).

[0097] When the folding state detection unit 409 detects that the mobile terminal device 100 is in a folded state, the mobile terminal device 100 directly displays the image captured by the camera 101 as a positive image on the first display unit 402 (display unit 108) on the photographer side (S203). Thereby, the photographer can confirm the image being captured on the display unit 108 on the photographer side.

[0098] Subsequently, an attempt is made to capture and detect the subject as a person using the depth sensor 102, and to identify the left and right eyes in the face thereof (S204). When both eyes can be identified, the first left eye gaze detector 103a and the first right eye gaze detector 104a, or the second left eye gaze detector 103b and the second right eye gaze detector 104b are used to detect the gaze destination 114 (gaze direction) towards which the left and right eyes are directed (S205). Based on the detected gaze destination 114 of the subject 130, it is identified whether the subject is gazing at the display unit 107 on the subject side, the camera 101, or at least the portable terminal device 100 itself (S206).

[0099] When the subject is gazing at the display unit 107 on the subject side, the camera 101, or at least the portable terminal device 100 itself, a processed image (mirror image) obtained by image processing the image captured by the camera 101 is displayed on the display unit 107 on the subject side (S207). Thereby, when the subject gazes at the display unit 107 on the subject side, the camera 101, or at least the portable terminal device 100 itself, the subject can see their own figure as a mirror image on the display unit 107 on the subject side. As a result, according to the second embodiment, the subject can easily visually recognize their own appearance as if looking at themselves in a mirror.

[0100] If the eyes of the subject cannot be identified in S204, or if the subject is not gazing at the display unit 107 on the subject side, the camera 101, or at least the portable terminal device 100 itself in S206, nothing is displayed on the display unit 107 on the subject side (S208). As a result, when the subject is not looking at the display unit on the subject side and there is no need to display, by stopping the display of the processed image, it is possible to suppress wasteful power consumption due to unnecessary display.

[0101] In this state, as an instruction to end without continuing imaging, it is determined whether it is set and input on the touch panel 411 by the touch operation of the photographer, or whether the camera operation switch 105 is pressed by the photographer (S209). When an instruction to stop imaging is given, after stopping imaging (S210), the display of the image on the display unit 108 on the photographer side and the display of the processed image on the display unit 107 on the subject side are stopped (S211), and the sequence ends. If an instruction to stop shooting is not given in S209, the process returns to the determination of whether the mobile terminal device 100 is in a folded state.

[0102] On the other hand, when the folding state detection unit 409 detects that the mobile terminal device 100 is in an open state rather than a folded state, the image captured by the camera 101 is displayed upright on the entire display 106 (integrated display unit) (S212). That is, the image is displayed in a display area that combines the display unit 108 that is always displayed on the photographer side, the display unit 107 on the subject side that is displayed when folded, and the non-display unit 109 that is not displayed when folded at the folding part. Then, the process proceeds to S209, and the subsequent operations are the same as those described above.

[0103] By the above operations, in the folded state of the mobile terminal device, only when the subject gazes at the mobile terminal device during shooting, the image of the subject captured by the camera 101 can be displayed on the display unit 107 on the subject side as a mirror image (processed image) that is easy for the subject to view. As a result, unnecessary display when the subject is not looking is suppressed, and without consuming unnecessary power, the subject can easily and conveniently visually recognize how they look during imaging.

[0104] Also, in the above description, an example of an image captured by the camera 101, that is, a still image such as a photograph, is shown. However, even for a moving image such as a video, the subject can see themselves moving on the display unit on the subject side. In particular, even when performing display control based on gaze detection, the subject is reflected on the display unit on the subject side only when the subject looks at the camera 101 while moving, and the subject can visually recognize their own appearance.

[0105] (Embodiment 3) The mobile terminal device 100 in Embodiment 3 will be described.

[0106] The flexible and foldable display 106 of the mobile terminal device 100 in Embodiment 1 was of a horizontal folding structure as shown in FIGS. 2 and 3, but the flexible and foldable display 106 of the mobile terminal device 100 in Embodiment 3 is of a vertical folding structure.

[0107] FIG. 9 is a diagram showing an overview of the mobile terminal device in Embodiment 3 as viewed from the photographer's side. FIG. 10 is a diagram showing an overview of the mobile terminal device 100 in Embodiment 3 as viewed from the subject side.

[0108] In FIG. 9, the parts marked with the same reference numerals as in FIG. 2 have the same configuration as the configuration already described in FIG. 2, so their detailed description is omitted. Similarly, in FIG. 10, the parts marked with the same reference numerals as in FIG. 3 have the same configuration as the configuration already described in FIG. 3, so their detailed description is omitted.

[0109] The difference between the configuration of Embodiment 3 and the configuration of Embodiment 1 is that instead of the first left gaze detector 103a and the first right gaze detector 104a, a second left gaze detector 103b and a second right gaze detector 104b are provided. The arrangement of the second left gaze detector 103b and the second right gaze detector 104b is provided in the same arrangement direction as the arrangement direction of the left eye and the right eye of the subject 130 when photographed by the camera 101, similar to the case of the arrangement of the first left gaze detector 103a and the first right gaze detector 104a shown in FIG. 3.

[0110] The control to display a processed image (mirror image) obtained by image-processing the image captured by the camera 101 on the display unit 107 on the subject side when the mobile terminal device 100 is folded is the same as that in Embodiment 1 described above, and its description is omitted.

[0111] When the display control by line-of-sight detection is performed, when the imaging device 100 is in a folded state and the photographer 140 operates the camera operation switch 105 to start imaging with the camera 101, the image captured by the camera 101 is displayed on the display unit 108 on the photographer side. On the other hand, the display unit 107 on the subject side remains in a non-display state. Then, the depth sensor 102 captures the subject 130 as a person and detects it while the subject 130 is being photographed by the camera 101, and finds and identifies the left eye and the right eye from the face. The second left-eye line-of-sight detector 103b and the second right-eye line-of-sight detector 104b capture the movements and directions of the left eye and the right eye detected by the depth sensor 102 respectively, and detect the line-of-sight destination 114 of the subject 130.

[0112] When the mobile terminal device 100 detects that the line-of-sight destination 114 detected by the second left-eye line-of-sight detector 103b and the second right-eye line-of-sight detector 104b is directed towards the display unit 107 on the subject side, the camera 101, or at least the direction of the mobile terminal device 100 itself, it displays a mirror-image image 115 that is reversed left and right as if reflected in a mirror on the display unit 107 on the subject side.

[0113] Thereby, even when the display 106 of the mobile terminal device 100 has a vertical folding structure, while more accurately detecting the line of sight of both eyes of the subject 130, in the folded mobile terminal device 100, only when the subject 130 gazes at the mobile terminal device 100 during imaging, the image of the subject being imaged can be displayed on the display unit 107 on the subject side as a mirror-image image 115 that is easy for the subject to view.

[0114] Note that even in the case of the mobile terminal device 100 whose display 106 has a vertical folding structure, line-of-sight detection may be performed using the first left-eye line-of-sight detector 103a and the first right-eye line-of-sight detector 104a instead of the second left-eye line-of-sight detector 103b and the second right-eye line-of-sight detector 104b.

[0115] Also, in the case of the mobile terminal device 100 with either a horizontal fold structure or a vertical fold structure for the display 106, a plurality of line-of-sight detectors such as the first left line-of-sight detector 103a, the second left line-of-sight detector 103b, the first right line-of-sight detector 104a, and the second right line-of-sight detector 104b may be provided, and an optimal line-of-sight detector may be selected or used in combination according to the positional relationship between the mobile terminal device 100 and the photographed person 130. In this case, regardless of the positional relationship between the mobile terminal device 100 and the photographed person 130, more accurate line-of-sight detection can be performed.

[0116] Furthermore, in the case of the mobile terminal device 100 having a pair of the first left line-of-sight detector 103a and the first right line-of-sight detector 104a and a pair of the second left line-of-sight detector 103b and the second right line-of-sight detector 104b, the posture state of the mobile terminal device 100 may be detected, and a pair that can perform more accurate line-of-sight detection may be selected according to the posture. The posture state of the mobile terminal device 100 can be detected by sensors such as an acceleration sensor 202, a gyro sensor 203, and a geomagnetic sensor 204, for example.

[0117] (Embodiment 4) The mobile terminal device 100 in this Embodiment 4 will be described.

[0118] FIG. 11 is a diagram schematically showing an overview of the mobile terminal device when the photographed person is photographed small in the image captured by the camera. FIG. 12 is a diagram showing the display unit on the photographed person side in the case of FIG. 11. In FIGS. 11 and 12, parts denoted by the same reference numerals as those in the first embodiment have the same configurations as those already described in the first embodiment, and thus their detailed descriptions are omitted.

[0119] As shown in FIG. 11, when the subject 500 appears far away or the entire landscape is photographed within the imaging range of the camera 101, if the image captured by the camera 101 is directly displayed on the display unit 107 on the subject side, the subject 500 will be displayed as a mirror image 510 with a relatively small figure in the image. In this case, it becomes difficult for the subject 500 to easily visually recognize their own appearance even when looking at the display unit 107 on the subject side.

[0120] Therefore, the mobile terminal device 100 in the fourth embodiment is provided with the following functions to address the above-mentioned difficulties.

[0121] As shown in FIG. 6, the mobile terminal device 100 in the fourth embodiment has an image processing unit 410 configured to perform image processing on the image captured by the camera 101. This image processing unit 410 is configured to expand the image area of the subject 500 included in the image captured by the camera 101 and generate a processed image with mirror processing applied. Then, the second display unit 403 is configured to display the processed image generated by the image processing unit 410.

[0122] Thereby, in the mobile terminal device 100 in the fourth embodiment, as shown in FIG. 12, the display unit 107 on the subject side displays the processed image (mirror image) 510A in which the subject 500 is greatly enlarged. The mobile terminal device 100 may, for example, temporarily save the image captured by the camera 101 in the information data 207 in the memory 208, and the image processing unit 410 reads out the subject 510 centered from the saved image and displays the greatly enlarged processed image 510A on the display unit 107 on the subject side.

[0123] Also, as shown in FIG. 13, the entire image is displayed in a positive image on the display unit 108 on the photographer side. At this time, it is also possible to perform a framing 520 display such that the display range indicating the figure of the subject can be seen. By this framing 520 display, the subject 500 to be enlarged and displayed on the display unit 107 on the subject side can be specified.

[0124] As described above, according to the mobile terminal device 100 in the fourth embodiment, the photographed person 500 can see his or her own image that is greatly enlarged and displayed on the display unit 107 on the photographed person side. As a result, the photographed person 500 can easily and firmly visually recognize how he or she looks during shooting.

[0125] (Embodiment 5) The mobile terminal device 100 in the fifth embodiment will be described.

[0126] FIGS. 14 and 15 are diagrams schematically showing an overview of the mobile terminal device when there are a plurality of photographed persons in the image captured by the camera. FIGS. 16 and 17 are diagrams showing the display unit on the photographer side in the case of FIG. 14.

[0127] In FIGS. 14 to 17, parts denoted by the same reference numerals as those in the first embodiment have the same configuration as the configuration already described in the first embodiment, and thus their detailed descriptions are omitted.

[0128] The mobile terminal device 100 in the fifth embodiment is provided with the following functions. That is, in FIG. 6, the mobile terminal device 100 has an image processing unit 410 and an input operation unit 411. This input operation unit 411 is configured to be able to specify a partial image area in which a specific photographed person is imaged among the image areas of an image in which a plurality of photographed persons are imaged. For example, the input operation unit 411 includes a first touch panel unit 412 that is on the first display unit 402 and is configured to be able to specify a partial image area from among the image areas of the image based on the position touched by the photographer. Further, the image processing unit 410 is configured to generate a processed image obtained by enlarging the partial image area specified by the input operation unit 411 and performing mirror processing. Then, the second display unit 403 displays the processed image generated by the image processing unit 410.

[0129] Specific description will be given below.

[0130] As shown in FIG. 14, when there are a plurality of subjects 600, 601, and 602 in the image captured by the camera 101, if this image is directly displayed on the display unit 107 on the subject side, the subjects 600, 601, and 602 will be displayed as relatively small processed images 610, 611, and 612 in the image. Therefore, in the fifth embodiment, as shown in FIG. 16, when the photographer performs an input operation to specify the subject to be enlarged and displayed, as shown in FIG. 15, the specified subject 602 is displayed on the display unit 107 on the subject side as a processed image 612A that is enlarged. The input operation can use the touch panel 209 (the first touch panel unit 412). As shown in FIG. 16, the entire image of the positive image captured by the camera 101 is displayed on the display unit 108 on the photographer side. Processed images 620, 621, and 622 of a plurality of subjects that can be detected as enlargement targets are displayed, and the processed image 622 of the specified subject is touched and selected on this display unit 108. The touch panel 209 detects that the position of the subject specified by the photographer on the display unit 108 on the photographer side where a plurality of subjects are displayed has been touched by the photographer, and the mobile terminal device 100 takes in the subject at the position detected by the touch panel 209 as input information, and enlarges and displays the processed image 612A of the specified subject on the display unit 107 on the subject side as shown in FIG. 15.

[0131] Also, as shown in FIG. 16, a frame 630 for selecting the subject to be enlarged may be displayed. At that time, the line of sight of the subject within the frame 630 is detected by the line-of-sight detector, and only when the line-of-sight destination (line-of-sight direction) of the subject is directed toward the display unit 107 on the subject side, the camera 101, or at least the mobile terminal device 100, the processed image of the subject within the frame 630 is enlarged and displayed on the display unit 107 on the subject side. It is meaningless to display a subject who is not looking at the camera 101, and only the subject looking at the camera 101 can be enlarged and displayed so that the subject can visually confirm how they look. That is, it becomes possible to assist the photographer's selection by the "function of determining the target of the frame display" such as "frame display" and line-of-sight detection.

[0132] Furthermore, after a subject to be enlarged and displayed is specified by an input operation, as shown in FIG. 17, an enlarged image 622A of the image of the subject specified by the selection may also be displayed on the display unit 108 on the photographer side. With this display, the photographer can surely visually confirm that the subject specified by the selection is enlarged and displayed without fail after the selection.

[0133] As described above, when there are a plurality of subjects, the photographer can specify the subject to be enlarged and displayed on the display unit 107 on the subject side. From this, the photographer can specify the subject whose appearance the photographer most wants to visually confirm, and can provide a processed image that is enlarged and easily visible for the specified subject. Conversely, in response to a request from the subject whose appearance the photographer most wants to visually confirm, the photographer can enlarge and display the requested subject.

[0134] In addition, as another example of a specific input operation, as an input operation unit for performing the input operation, the microphone 210 may be used to capture the voice from the photographer indicating the specified subject into the mobile terminal device 100 by the microphone 210. Furthermore, a line-of-sight detector on the photographer side for detecting the line of sight of the photographer may be provided on the surface on the photographer side, and the line-of-sight detector on the photographer side can detect which subject among the plurality of subjects displayed on the display unit on the photographer side the photographer is trying to specify, and can determine the specified subject. Also, the photographer may confirm with the subject whether to display or not. For example, when a person who happened to be captured in the image is looking at the camera 101, the photographer may choose not to display.

[0135] (Embodiment 6) The mobile terminal device 100 in this Embodiment 6 will be described.

[0136] FIG. 18 schematically shows an overview of a mobile terminal device when the hand of the photographer covers the photographed person displayed on the display unit on the photographed person side, resulting in the photographed person being unable to see their entire body. FIGS. 19 and 20(a) to 20(c) are diagrams showing the display unit on the photographed person side in the case shown in FIG. 18. FIG. 21 is a diagram showing the display unit on the photographer side in the case shown in FIG. 18. In FIGS. 18 to 21, portions assigned the same reference numerals as those in the first embodiment have the same configuration as the configuration already described in the first embodiment, and thus their detailed description is omitted.

[0137] The mobile terminal device 100 in the sixth embodiment is provided with the following functions. That is, in FIG. 6, the mobile terminal device 100 has an image processing unit 410, an input operation unit 411, and an overlap determination unit 414. The input operation unit 411 is on the second display unit 403 and includes a second touch panel unit 413 configured to detect an overlapping area between the second display unit 403 and the hand of the photographer. Further, the overlap determination unit 414 is configured to determine whether or not the overlapping area detected by the second touch panel unit 413 and the image area of the photographed person included in the processed image displayed on the second display unit 403 overlap planarly. Furthermore, when the overlap determination unit 414 determines that the overlapping area and the image area of the photographed person overlap planarly, the image processing unit 410 is configured to perform image processing on the processed image so that the image area of the photographed person does not overlap the overlapping area planarly. Then, the second display unit 403 displays the processed image that has been image-processed by the image processing unit 410.

[0138] Specific description will be given below.

[0139] As shown in FIG. 18, when the subject 800 is within the shooting range of the camera 101 and the image area 810 in which the subject is shown is included in the processed image displayed on the display unit 107 on the subject side, the handle 700 of the photographer holding the mobile terminal device 100 may partially cover the image area 810 of the subject displayed on the display unit 107. In this case, it becomes very difficult for the subject 800 to see his or her own image that is being captured. Therefore, in the mobile terminal device 100 according to the sixth embodiment, the overlapping area between the display unit 107 on the subject side and the handle 700 of the photographer is detected by the touch panel 209 (second touch panel unit 413), and as shown in FIG. 19, a device is provided to move and display the image area 810 of the subject to a location other than the detected overlapping area.

[0140] This feature will be described in detail with reference to FIGS. 20(a) to 20(c). First, as shown in FIG. 20(a), the touch panel 209 detects the shielding range 700A covered by the handle. In FIG. 20(a), the shielding range 700A covers the image area 810 in which the subject is shown on the display unit 107 on the subject side. Note that the area 820 indicated by the dotted line indicates the image display range in which the processed image is displayed on the display unit 107 on the subject side. In the case of FIG. 20(a), the area 820 is almost the entire range of the display unit 107 on the subject side.

[0141] On the other hand, for example, as shown in Fig. 20(b), the entire processed image is reduced, and the reduced entire processed image is displayed in the area 830 outside the shielding range 700A. In this case, by ensuring that the area 830 and the shielding range 700A do not overlap, it is possible to display the entire processed image while ensuring that the image area 810A in which the subject is captured is not hidden by the shielding range 700A. Also, as shown in Fig. 20(c), the entire processed image can be moved and the moved processed image can be displayed in the area 840 outside the shielding range 700A detected by the touch panel 209. In this case, although there will be parts that are not displayed outside the image area 810B in which the subject is captured, the image area 810B in which the subject is captured without being blocked by the shielding range 700A can be displayed in its original size. Further, for example, as shown in Fig. 12 of the above-described Embodiment 4, the processed image of the subject can be enlarged and displayed outside the shielding range 700A that is not blocked by the handle.

[0142] Furthermore, as shown in Fig. 21, the shielding range 700A by the handle may be displayed on the display unit 108 on the photographer's side. Thereby, the photographer can recognize that the image area 810C in which the subject is captured within the image display area 850 indicated by the dotted line is covered by the shielding range 700A by the handle. Therefore, according to the portable terminal device 100 in this Embodiment 6, the subject can clearly visually recognize their entire self-image without being blocked by the handle.

[0143] (Embodiment 7) The portable terminal device 100 in this Embodiment 7 will be described.

[0144] In this Embodiment 7, an example will be described in which the camera 101 and the depth sensor 102 are arranged on the inner surface of the portable terminal device 100 in the folded state.

[0145] In addition, in FIG. 22, parts with the same reference numerals as those in the first embodiment have the same configurations as those already described in the first embodiment, and thus their detailed descriptions are omitted.

[0146] For example, in the mobile terminal device 100 in the first embodiment, an example in which a camera 101 and a depth sensor 102 are arranged on the outer surface of the folded mobile terminal device 100 has been described. In this case, for example, as shown in FIG. 3, the front fields of view of the camera 101 and the depth sensor 102 are not blocked, and the photographing and detection / identification of the subject 130 by the camera 101 and the depth sensor 102 can be sufficiently performed. That is, in the configuration of the mobile terminal device 100 in the first embodiment, the camera 101 (imaging unit 401) is provided at a position where the subject 130 can be imaged even when the mobile terminal device 100 is folded, and the line-of-sight direction detection unit 404 is also provided at a position where the line-of-sight direction of the subject 130 can be detected.

[0147] On the other hand, for example, when the camera 101 and the depth sensor 102 are arranged on the inner surface of the folded mobile information terminal 100, depending on the housing structure of the folded mobile terminal device 100, the front fields of view of the camera 101 and the depth sensor 102 may be blocked, and there may be a case where the photographing and detection / identification of the subject 130 by the camera 101 and the depth sensor 102 cannot be sufficiently performed.

[0148] Regarding this point, in the seventh embodiment, as shown below, a device is made for the housing structure of the mobile terminal device 100. That is, as shown in FIG. 22, even in the folded state, the housing in front of the camera 901 and the depth sensor 902 on the inner surface of the mobile terminal device 100 is opened, etc., so that the photographing range of the camera 901 from direction 903 to direction 904 and the observation range of the depth sensor 902 from direction 905 to direction 906 are not blocked.

[0149] That is, the mobile terminal device 100 in the seventh embodiment has a housing structure that does not hinder imaging by the imaging unit 401 and detection of the line-of-sight direction by the line-of-sight direction detection unit 404 even when the mobile terminal device 100 is folded.

[0150] Accordingly, according to the mobile terminal device 100 in the seventh embodiment, in the folded state, the camera 901 can satisfactorily photograph all the people and landscapes within the field of view, and the depth sensor 902 can detect the subject 130 as a person and can identify the eyes from the faces of the detected people without any problem.

[0151] (Eighth Embodiment) The mobile terminal device 100A in the eighth embodiment will be described.

[0152] FIG. 23 is a diagram schematically showing an overview of the mobile terminal device in the eighth embodiment, and FIG. 24 is a diagram schematically showing the appearance of the mobile terminal device shown in FIG. 23 from another angle. In FIGS. 23 and 24, the portions denoted by the same reference numerals as those in the first embodiment have the same configurations as those already described in the first embodiment, and thus their detailed descriptions are omitted.

[0153] In the mobile terminal device 100A shown in FIG. 23, as the display, instead of the flexible display panel employed in the first embodiment, a flat panel made of a glass substrate and incapable of being bent is used. As shown in FIGS. 23 and 24, this mobile terminal device 100A includes a main display 1001 on the side of the photographer 140 (see FIG. 24) and a sub-display 1002 on the side of the subject 130 (see FIG. 23).

[0154] In FIG. 24, when the subject 130 appears within the shooting range of the camera 101 on the main display 1001 which is the display unit on the photographer 140 side, the image 113 of the subject shot by the camera 101 is displayed as the original upright image, and the photographer 140 can confirm the subject with the image 113. On the other hand, as shown in FIG. 23, when the subject 130 appears, the first left gaze detector 103a and the first right gaze detector 104a each capture the movement and direction of the left and right eyes of the subject 130 detected and identified by the depth sensor 102, and when it is detected that the line of sight 114 of the subject 130 is directed towards the sub-display 1002 which is the display unit on the subject 130 side, the camera 101, or at least the mobile terminal device 100A itself, a mirror image 115 that is reversed left and right as if reflected in a mirror is displayed on the sub-display 1002.

[0155] Therefore, even in the mobile terminal device 100A in the eighth embodiment that has displays on both the front and back surfaces, only when the subject 130 gazes at the mobile terminal device 100A during imaging, the image of the subject 130 being imaged can be displayed on the sub-display 1002 on the subject 130 side as a mirror image 115 that is easy for the subject to view. From this, it is possible to suppress unnecessary display when the subject 130 is not looking and suppress the consumption of wasted power, and the subject 130 can easily and conveniently visually recognize how they look while being photographed.

[0156] Subsequently, as another example of the eighth embodiment, an example in which a flexible display is provided on the inner surface of the folded mobile terminal device 100B will be described.

[0157] FIG. 25 is a diagram schematically showing the appearance of a configuration having a flexible display on the inner surface of the foldable mobile terminal device 100B, and FIG. 26 is a diagram schematically showing an overview of the folded mobile terminal device.

[0158] In FIGS. 25 and 26, parts with the same reference numerals as those in the first embodiment have the same configurations as those already described in the first embodiment, and thus their detailed descriptions are omitted.

[0159] As shown in FIG. 25, the mobile terminal device 100B has a first display 1001 made of a flexible panel on the inner surface that can be folded, and as shown in FIG. 26, a second display 1102 made of a flat panel composed of a glass substrate on the outer surface on the side of the person to be photographed 130 and cannot be bent. When the first display 1001 is opened, the photographer 140 can view an image from the mobile terminal device 100B on a large screen. On the other hand, only when the line of sight of the person to be photographed 130 being imaged by the camera 101 faces the second display 1102 visible to the person to be photographed 130, the camera 101, or at least the mobile terminal device 100B itself, a mirror image 115 that is reversed left and right as if reflected in a mirror is displayed on the second display 1102 visible to the person to be photographed 130.

[0160] Therefore, even in the mobile terminal device 100B having the first display 1101 made of a flexible panel on the inner surface, by providing the second display 1102 on the outer surface on the side of the person to be photographed 130, when the person to be photographed 130 gazes at the mobile terminal device 100B during imaging, the image of the person to be photographed 130 being imaged can be displayed as a mirror image 115 that is easy for the person to be photographed 130 to view on the second display 1102 on the side of the person to be photographed 130.

[0161] This suppresses unnecessary displays when the person to be photographed 130 is not looking and suppresses wasteful power consumption. At the same time, the person to be photographed 130 can easily and conveniently visually recognize how they are being photographed during imaging. Additionally, it is also possible to provide a display on the outer surface located on the side of the photographer 140 in the folded state so that the photographer 140 can view the image during photography.

[0162] As described above, the invention made by the present inventor has been specifically described based on its embodiments. However, it goes without saying that the present invention is not limited to the above embodiments and can be variously modified without departing from the gist thereof.

[0163] For example, the above-described embodiments have been described in detail in order to explain the technical idea of the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Also, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace other configurations for a part of the configuration of each embodiment.

[0164] Each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing part or all of them with an integrated circuit. However, each of the above configurations, functions, etc. may also be realized by software in which a processor interprets and executes a program for realizing each function. Information such as programs, tables, and files for realizing each function can be placed in a memory, a recording device such as a hard disk or an SSD (Solid State Drive), or a recording medium.

[0165] The control lines and information lines show those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In fact, it may be considered that almost all the components are interconnected.

Explanation of Reference Numerals

[0166] 100 Portable terminal device 100A Portable terminal device 100B Portable terminal device 101 Camera 102 Depth sensor 103a First left gaze detector 103b Second left gaze detector 104a First right gaze detector 104b Second right gaze detector 105 Camera operation switch 106 Display 107 Display section 108 Display section 109 Non-display section 113 Image 114 Line of sight destination 115 Mirror image 130 Person to be photographed 140 Photographer 201 Folding state detector 202 Acceleration sensor 203 Gyro sensor 204 Geomagnetic sensor 205 Control unit 206 Program 207 Information data 208 Memory 209 Touch panel 210 Microphone 211 Speaker 212 Vibration generator 213 Communication interface 214 Transceiver antenna 215 Bus 301 Hall element 302 Hall element 303 Magnet 304 Magnet 401 Imaging unit 402 First display section 403 Second display section 404 Line of sight direction detection section 405 Identification section 406 Line of sight direction detector 407 Judgment section 408 Display control section 409 Folding state detection section 410 Image processing section 411 Input operation section 412 First touch panel section 413 Second touch panel section 414 Duplication judgment section 415 Data storage section 500 Person to be photographed 510 Mirror Image 510A Processed Image 520 Framing 600 Subject 601 Subject 602 Subject 610 Processed Image 611 Processed Image 612 Processed Image 612A Processed Image 620 Processed Image 621 Processed Image 622 Processed Image 622A Processed Image 630 Frame 700 Handle 700A Shielding Range 800 Subject 810 Image Area 810A Image Area 810B Image Area 810C Image Area 820 Area 830 Area 840 Area 850 Area 901 Camera 902 Depth Sensor 903 Direction 904 Direction 905 Direction 906 Direction 1001 Main Display 1002 Sub - Display

Claims

1. An imaging unit configured to capture an image of a subject; A first display unit configured to display an image captured by the imaging unit; a second display configured to display a processed image obtained by processing the image; a determination unit configured to determine whether the person to be photographed is facing in the direction in which a mobile terminal device is present; a display control unit configured to control display / non-display of the processed image on the second display unit based on a determination result of the determination unit; The mobile terminal device comprises: the first display unit is provided at a position visible to a photographer, the second display unit is provided at a position visible to the subject, the determination unit determines whether the person being photographed is facing a direction in which the mobile terminal device is located by analyzing the image; A mobile terminal device, wherein the display control unit is configured to display the processed image on the second display unit when the judgment unit judges that the person being photographed is facing in the direction in which the mobile terminal device is located, and to not display the processed image when the judgment unit judges that the person being photographed is not facing in the direction in which the mobile terminal device is located.

2. 2. The mobile terminal device according to claim 1, A mobile terminal device, wherein the processed image is a mirror image of the image.

3. 2. The mobile terminal device according to claim 1, The mobile terminal device is configured to be flexible and foldable, When the mobile terminal device is not folded, the first display unit and the second display unit form a flat integrated display unit, When the mobile terminal device is folded, the first display unit and the second display unit are disposed in positions facing each other.

4. 4. The mobile terminal device according to claim 3, The mobile terminal device comprises a folded state detection unit configured to detect whether the mobile terminal device is folded or not.

5. 5. The mobile terminal device according to claim 4, when the folding state detection unit detects that the portable terminal device is in an unfolded state, the portable terminal device is configured to display the image on the integrated display unit, A portable terminal device, wherein when the folded state detection unit detects that the portable terminal device is in a folded state, the portable terminal device is configured to be able to display the image on the first display unit, while being configured to be able to display the processed image on the second display unit.

6. 2. The mobile terminal device according to claim 1, the portable terminal device has an image processing unit configured to enlarge an image area of ​​the subject included in the image and generate the processed image by performing mirror image processing; The second display unit is configured to display the processed image generated by the image processing unit.

7. 2. The mobile terminal device according to claim 1, The mobile terminal device an input operation unit configured to be able to specify a partial image area in which a specific person is captured among image areas of the image in which a plurality of people are captured; an image processing unit configured to enlarge the partial image area designated by the input operation unit and generate the processed image by performing mirror image processing; having The second display unit is configured to display the processed image generated by the image processing unit.

8. 8. The mobile terminal device according to claim 7, A mobile terminal device, wherein the input operation unit is a first touch panel unit located on the first display unit and configured to be able to specify the partial image area from within the image area of ​​the image based on the position touched by the photographer.

9. 2. The mobile terminal device according to claim 1, The mobile terminal device a second touch panel unit disposed on the second display unit and configured to detect an overlapping area between the second display unit and a contact object; an overlap determination unit configured to determine whether or not the overlap area detected by the second touch panel unit and an image area of ​​the subject included in the processed image displayed on the second display unit overlap in a planar manner; an image processing unit configured to, when it is determined by the overlap determination unit that the overlapping area and the image area of ​​the subject overlap in a planar manner, perform image processing on the processed image so that the image area of ​​the subject does not overlap in a planar manner with the overlapping area; having A mobile terminal device, wherein the second display unit is configured to display the processed image that has been image-processed by the image processing unit.

10. 2. The mobile terminal device according to claim 1, The imaging unit is provided at a position where it can capture an image of the person to be photographed even when the portable terminal device is folded.

11. 2. The mobile terminal device according to claim 1, The portable terminal device has a housing structure that does not impede imaging by the imaging unit even when the portable terminal device is folded.

Images