Information processing device, image display device, captured image transferring display system and captured image transferring display method

Abstract

To assist camera capturing operations under a condition that a display screen of an information processing device such as a smartphone and the like is difficult to be seen.SOLUTION: An information processing device comprises a camera, a display screen for displaying a captured image, a gaze sensor for detecting gaze of a user and outputting gaze information, a communication apparatus for communicating with another image display device, and a processor. While the camera is capturing an image, the processor determines whether or not a gaze destination of the user is on the display screen based on the gaze information, and if it is determined that the gaze destination of the user is not on the display screen, transfers the captured image displayed on the display screen from the communication apparatus to the other image display device, and transmits a display start signal to cause the other image display device to display the transferred captured image.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an image display apparatus, a photographed image transfer display system, and a photographed image transfer display method, and more particularly to an information processing apparatus, an information processing system, and an information processing method for switching a terminal for displaying an image.

Background Art

[0002] Patent Document 1 describes that "a digital camera includes a camera body with a built-in image sensor and a head-mounted viewfinder worn on the head of a photographer. When in use, the head-mounted viewfinder is worn on the photographer's head so that the display unit of the head-mounted viewfinder is located in front of the photographer's eyes. Then, when the camera body held in one hand of the photographer is directed at a subject for shooting, the image is displayed on the display unit of the head-mounted viewfinder. (Summary excerpt)".

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An information processing apparatus having a camera function, a display function, etc., especially a portable one, is called a smartphone (hereinafter referred to as a smartphone), and generally has a palm-sized display screen for displaying an image taken by a camera and various information, and is widely popular for its good usability.

[0005] When taking a camera shot using a smartphone, for example, when there are many people in front, it is difficult to hold the smartphone in front of the eyes and take a camera shot, and it becomes a situation where one has to stretch one's hand to take a camera shot. Also, depending on the position of the camera shooting object, there may be a situation where one has to take a shot with the display screen of the smartphone being difficult to see.

[0006] Therefore, it is conceivable to apply the technology of Patent Document 1 to a smartphone and transfer a captured image to another display device. However, even if an attempt is made to view the captured image on the display screen mounted on the smartphone, if it is transferred to another display device, there remains a problem that it goes against the user's intention.

[0007] The present invention has been made in view of the above problems, and an object thereof is to assist a camera shooting operation in a situation where the display screen of an information processing device such as a smartphone is difficult to view.

Means for Solving the Problems

[0008] As means for solving the above problems, the technology described in the claims is used. For example, the present invention is an information processing device including a camera, a display screen for displaying a captured image captured by the camera, a gaze sensor that detects a user's gaze and outputs gaze information indicating the user's gaze direction, a communicator that communicates with another image display device, and a processor connected to each of the camera, the display screen, the gaze sensor, and the communicator. The processor determines whether the user's line of sight is on the display screen based on the gaze information while the camera is shooting. When it is determined that the user's line of sight is not on the display screen, the captured image displayed on the display screen is transferred from the communicator to the other image display device, and a display start signal for causing the other image display device to display the transferred captured image is transmitted.

Effects of the Invention

[0009] According to the present invention, it is possible to assist a camera shooting operation in a situation where the display screen of an information processing device such as a smartphone is difficult to view. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0010]

Figure 1A

Figure 1B

Figure 1C

Figure 1D

Figure 1E

Figure 1F

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0011] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. The same components throughout the figures are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] <First Embodiment> The first embodiment is a photographed image transfer system 1 that cooperates an information processing apparatus having a camera function and an image display apparatus configured separately from the information processing apparatus, and transfers a photographed image to the image display apparatus as needed.

[0013] Figures 1A to 1F are diagrams for schematically explaining the appearance of the photographed image transfer system 1 according to the first embodiment. In the following description, a smartphone 100 is taken as a specific example of the information processing apparatus, and a head-mounted display (hereinafter referred to as HMD) 200 is taken as a specific example of the image display apparatus for explanation.

[0014] In Figure 1A, the HMD 200 is an image display apparatus that uses augmented reality (AR) technology to reflect and expand a virtual object (virtual object) on a virtual space composed of other images or images created by computer graphics (CG) into the real space, and displays the virtual object while recognizing a physical object in the real space in three dimensions.

[0015] The HMD 200 includes a type (optical see-through type) that displays a virtual object on a display unit while visually recognizing a physical object in front of the eyes, and a type (video see-through type) that displays a physical object or virtual object in the real space on a display unit provided on the front of the head. The user 10 can closely view the physical object or virtual object being gazed at.

[0016] User 10 holds the smartphone 100 in hand and wears the HMD 200 on the head. The smartphone 100 is provided with an in-camera 102 and an out-camera 103 for photographing the surrounding view on the display screen 101 side and the back side thereof, respectively.

[0017] Furthermore, on the display screen 101 side of the smartphone 100, a depth sensor 104, a left eye gaze sensor 105, and a right eye gaze sensor 106 are provided.

[0018] On the display screen 101, there is a camera shooting mode button 107 which, when touched, activates the camera and switches to the shooting display mode.

[0019] The in-camera 102 photographs surrounding people and scenery within the shooting range (angle of view) from direction 110 to direction 111.

[0020] The out-camera 103 shoots within the shooting range from direction 112 to direction 113.

[0021] The depth sensor 104 measures the distance and angle from the smartphone 100 to the object and outputs distance information indicating at what distance and at what angle the object is from the depth sensor 104. Furthermore, the depth sensor 104 is a sensor that can capture the shape of an object such as a person or an object in a three-dimensional manner within the observation range from direction 114 to direction 115. The depth sensor 104 can detect and identify the user 10 looking at the smartphone 100.

[0022] The left eye gaze sensor 105 and the right eye gaze sensor 106 respectively detect the movement and direction of the left eye and the right eye of the face of the user 10 detected and identified by the depth sensor 104, and can detect both the gaze lines 141 and 142 of the left eye and the right eye to capture the gaze destination 116 three-dimensionally. The gaze directions output by the left eye gaze sensor 105 and the right eye gaze sensor 106 correspond to gaze information, and each point on the gaze direction corresponds to the gaze destination.

[0023] The HMD200 is worn on the head of the user 10, and the display 265 is arranged in front of the eyes of the user 10. An image is displayed as a virtual object on the display screen 132 (see FIG. 1C) of the display 265. When the display 265 is a transmissive type, the virtual object is superimposed and displayed on the real object in front of the eyes. If the HMD200 is an immersive type, the user can experience augmented reality by superimposing the virtual object on the image captured by the out-camera 203 (see FIG. 3) of the HMD200 and displaying it.

[0024] When the smartphone 100 is powered on, the display screen in the normal menu display mode shown in FIG. 1A is displayed. In the menu display mode, the display screen 101 displays the camera shooting mode button 107, various app icons, etc. In this state, when the user 10 touches the camera shooting mode button 107, the display screen 101 changes from the normal menu display mode to the shooting display mode.

[0025] In the shooting display mode shown in FIG. 1B, the shooting image captured by the out-camera 103 is displayed on the display screen 101. Also, on the display screen 101, a shutter button 120 for inputting an instruction to record the shooting image in the memory 170 (a form of storage) of the smartphone 100 is displayed. Usually, in order to display the shooting image in as wide an area as possible, the shutter button 120 is displayed in a small area that can be touched with a fingertip.

[0026] When the user 10 is looking at the display screen 101, while sufficiently viewing the displayed shooting image, the user can touch the shutter button 120 with a small area display in the scene to be recorded.

[0027] On the other hand, when the display screen 101 is difficult to view or cannot be seen at all, the shooting image displayed on the display screen 101 cannot be sufficiently viewed. Therefore, it becomes difficult to perform a touch operation on the shutter button 120 with a small area display well at the moment of the scene to be recorded.

[0028] This embodiment is for coping with such difficult situations. When detecting the visual state of the user 10 and detecting that the user 10 is not looking at the display screen 101, as shown in FIG. 1C, the captured image of the out-camera 103 is transferred from the smartphone 100 to the HMD 200. In FIG. 1C, the arrow 130 indicates the transfer direction of the captured image.

[0029] In the HMD 200, the transferred captured image 131 is displayed as a virtual object on the display screen 132 of the HMD 200. The user 10 can visually recognize the transferred captured image 131 as a virtual object together with the physical object in front of the eyes by using the HMD 200.

[0030] The processor 167 (see FIG. 3) of the smartphone 100 identifies the left eye and the right eye in the face of the user 10 based on the depth sensor information output by the depth sensor 104, detects the direction of the line of sight 116 based on the respective left line-of-sight information and right line-of-sight information of the left line-of-sight sensor 105 and the right line-of-sight sensor 106, and determines whether the display screen 101 is on the extension of the line of sight 116. If the display screen 101 is on the line of sight 116, it is determined that the user 10 is looking at the display screen 101. Note that there may be only one line-of-sight sensor.

[0031] On the other hand, when the depth sensor 104 cannot identify the left eye and the right eye in the face of the user 10, or even if it can identify them but the line of sight is not the display screen 101, the processor 167 determines that the user 10 is not looking at the display screen 101. Also, the line of sight of the left eye and the right eye in the face may be detected from the captured image of the in-camera 102 using the image processing algorithm for face recognition.

[0032] The communication between the smartphone 100 and the HMD 200 is set as a connected state in advance so that the captured image can be instantly displayed on the HMD 200 when it enters the transfer mode. Note that even when connected, the HMD 200 does not necessarily need to constantly display the captured image from the smartphone 100. Also, on the display screen 101 of the smartphone 100, in addition to the captured image, a shutter button 120 and various operation buttons are also displayed. However, as the transfer image transferred from the smartphone 100 to the HMD 200, the necessity of displaying the shutter button 120 and various operation buttons is quite low in the normal usage state. Therefore, it is not necessarily required to be a screen mirroring, and only the captured image may be sufficient. The transfer judgment of the captured image is performed by the processor 167 installed in the smartphone 100. When the processor 167 determines, based on the sensor information of each of the depth sensor 104, the left gaze sensor 105, and the right gaze sensor 106, that the gaze destination 143 is not facing the display screen 101, it transfers the captured image to the HMD 200. As another example, the processor 267 of the HMD 200 may determine, based on the sensor information of each of the depth sensor 204, the left gaze sensor 205, and the right gaze sensor 206 installed in the HMD 200, that the gaze destination 143 is not facing the display screen 101, transmit a transfer request signal for the captured image to the smartphone 100, and the smartphone 100 may respond thereto and perform the transfer of the captured image. Further, the sensor information of each of the depth sensor 204, the left gaze sensor 205, and the right gaze sensor 206 installed in the HMD 200 may be transmitted to the smartphone 100, and the processor 167 of the smartphone 100 may perform the transfer judgment based on the received sensor information. The transfer judgment in the following embodiments is the same.

[0033] Moreover, even when it is difficult to view the display screen 101 of the smartphone 100 or when in a shooting state where the display screen 101 cannot be seen at all, and it is impossible to sufficiently visually recognize the captured image displayed on the display screen 101, the user 10 can clearly visually recognize the captured image captured by the outer camera 103 of the smartphone 100 by looking at the captured image 131 displayed on the HMD 200.

[0034] Furthermore, while the processor 167 of the smartphone 100 determines that the user 10 is not looking at the display screen 101 and transfers the captured image taken by the out-camera 103 to the HMD 200, the display screen 101 is made non-display, and an extended shutter button 134 that can detect a touch operation and perform a shutter button operation is provided in the display area of the captured image on the display screen 101.

[0035] Therefore, even when the user 10 has difficulty seeing or cannot see the display screen 101, a touch operation on the shutter button 120 can be performed in a wide operation area of the display screen 101, and it becomes possible to surely and conveniently capture a scene that the user wants to record. That is, when the captured image is transferred to the HMD 200, even if the image display area of the display screen 101 of the smartphone 100 other than the shutter button 120 is touched, the shutter can be released, and the effect that the shutter button operation can be performed even when the user 10 is not looking at the display screen 101 of the smartphone 100 is obtained.

[0036] Furthermore, unnecessary and wasteful display of captured images in a state where the user 10 cannot see them can be eliminated, and as a result, power consumption can be reduced. Note that the area that can be touched as the shutter button 120 does not have to be the entire display screen. For example, the extended shutter button 134 may be provided in an area inside a predetermined margin area from the edge of the display screen 101 so as to avoid the holding position of the smartphone 100. In FIG. 1C, the extended shutter button 134 is provided only in a limited area inside the display screen 101, not the entire display screen 101. In this case, it becomes possible to prevent the shutter button 120 from reacting to the holding hand of the smartphone 100.

[0037] On the other hand, when it is detected that the user 10 is looking at the display screen 101, as shown in FIG. 1D, the transfer of the captured image from the smartphone 100 to the HMD 200 is stopped and not performed. The image transferred to the display screen 132 of the HMD 200 becomes non-display, and the user 10 can directly view the display screen 101 in front of the eyes.

[0038] In addition, in the HMD 200 equipped with the depth sensor 204, the left line-of-sight sensor 205, and the right line-of-sight sensor 206, as shown in FIG. 1E, the depth sensor 204 identifies the display screen of the front smartphone 100, and the left line-of-sight sensor 205 and the right line-of-sight sensor 206 detect the left line of sight 141 and the right line of sight 142, respectively, to detect whether the line-of-sight destination 143 is facing the display screen 101 of the smartphone 100. When it is detected that the line-of-sight destination 143 is on the display screen 101, the transfer of the captured image from the smartphone 100 to the HMD 200 and the display of the transferred image on the HMD 200 are not performed. That is, the same operation as in the case shown in FIG. 1D is performed.

[0039] On the other hand, when it is detected that the line-of-sight destination 143 is not facing the display screen 101, as shown in FIG. 1F, the HMD 200 transmits this detection result information to the smartphone 100 (illustrated by the arrow 151 for transfer). In the smartphone 100, in response to the transmitted detection result, the captured image captured by the out-camera 103 is transferred from the smartphone 100 to the HMD 200 (illustrated by the arrow 130 for transfer), and in the HMD 200, the transferred captured image 131 is displayed on the display screen 132.

[0040] That is, the processor 267 mounted on the HMD 200 determines, based on the sensor information of the depth sensor 204, the left line-of-sight sensor 205, and the right line-of-sight sensor 206, that the line-of-sight destination 143 is not facing the display screen 101. When it is determined that it is not facing, a transfer request signal for the captured image is transmitted to the smartphone 100, and the smartphone 100 responds by transferring the captured image.

[0041] Alternatively, the sensor information of the depth sensor 204, the left line-of-sight sensor 205, and the right line-of-sight sensor 206 mounted on the HMD 200 may be transmitted to the smartphone 100, and the processor 167 of the smartphone 100 may determine whether transfer of the captured image is necessary based on this sensor information.

[0042] When taking a picture of the captured image with respect to the HMD 200, as in FIGS. 1A to 1D, on the smartphone 100, the display screen 101 is set to a non-display state, and a shutter button operation can be performed by touching any area of the display screen 101.

[0043] Therefore, even when detecting on the HMD 200 side whether the user 10 is looking at the display screen 101 of the smartphone 100, the same operations and effects can be obtained as when detecting on the smartphone 100 side whether the user 10 is looking at the display screen 101 of the smartphone 100.

[0044] Note that there may be one line-of-sight sensor. When the HMD 200 is equipped with a line-of-sight sensor, even when the user 10 is relatively far from the smartphone 100, performance close to binocular line-of-sight detection can be obtained.

[0045] In the above operations, the information processing device exemplified by the smartphone 100 may control the operation of the image display device exemplified by the HMD 200 in an instruction control manner, or an information processing system composed of the information processing device and the image display device may be configured. Thus, even in a situation where the display screen 101 of an information processing device such as the smartphone 100 is difficult to view for the user 10, the user 10 can clearly confirm the captured image of the information processing device, and further, the shutter button operation becomes easier, improving the usability.

[0046] Next, referring to FIG. 2, the hardware configurations of the smartphone 100 and the HMD 200 will be described. FIG. 2 is a hardware configuration diagram of the smartphone 100 and the HMD 200.

[0047] The smartphone 100 includes an in-camera 102, an out-camera 103, a depth sensor 104, a left-eye line-of-sight sensor 105, a right-eye line-of-sight sensor 106, an acceleration sensor 161, a gyro sensor 162, a geomagnetic sensor 163, a touch panel 164, a display 165, a telephone network communicator 166, a processor 167 (first processor), a memory 170 in which a program 168 and information data 169 are stored, a vibrator 171, a microphone 172, a speaker 173, and an inter-device communicator 174 (first communicator), and each component is interconnected via a bus 180.

[0048] The HMD200 includes an out-camera 203, a depth sensor 204, a left eye gaze sensor 205, a right eye gaze sensor 206, an acceleration sensor 261, a gyro sensor 262, a geomagnetic sensor 263, an operation input interface 264, a display 265, a processor 267 (second processor), a program 268, and a memory 270 for storing information data 269, a vibrator 271, a microphone 272, a speaker 273, and an inter-device communicator 274 (second communicator), and each component is interconnected via a bus 280.

[0049] The in-camera 102 converts the light incident from the lens into an electrical signal by an imaging element, and captures the user 10 looking at the display 165, etc.

[0050] The out-cameras 103 and 203 capture the visual field state of the front surroundings, and convert the light incident from the lens into an electrical signal by an imaging element to receive a captured image.

[0051] The out-camera 103 mounted on the smartphone 100 captures the back side of the display 165 of the smartphone 100, and the user 10 can hold the smartphone 100 facing the outer circumference and capture the surrounding visual field.

[0052] The out-camera 203 mounted on the HMD200 captures the visual field visible from the HMD200 worn by the user 10.

[0053] The depth sensors 104 and 204 measure the distance and angle to an object, and capture the shape of the object such as a person or an object in three dimensions. As the depth sensors 104 and 204, a LiDAR (Light Detection and Ranging) that irradiates the object with laser light such as infrared light, measures the scattered light that bounces back, and detects the distance to an object at a long distance, a TOF (Time Of Flight) sensor that measures the reflection time of the pulsed light irradiated to the subject for each pixel to perform distance measurement, a millimeter wave radar that emits millimeter wave radio waves and captures the reflected wave to detect the distance to the reflector, etc. are used.

[0054] The depth sensor 104 of the smartphone 100 can detect the user 10 when the user 10 is looking at the display screen 101 and identify both eyes from the face of the detected user 10. For example, the overall shape of the user's face can be grasped based on the depth information of the depth sensor 104, and the positions of both eyes can be estimated from the overall shape.

[0055] In addition, the depth sensor 204 of the HMD 200 can identify the distance to the display screen 101 when the display screen 101 of the smartphone 100 is in front of the user 10.

[0056] The left eye gaze sensors 105, 205 and the right eye gaze sensors 106, 206 detect the gazes 141, 142 by capturing the movement and direction of the right eye and the left eye, respectively. The process of detecting the gazes 141, 142 may utilize a well-known technology commonly used as eye tracking processing. For example, as a method using corneal reflex, a technology is known in which an infrared LED (Light Emitting Diode) is irradiated onto the face and photographed with an infrared camera, and the position on the cornea of ​​the reflected light (corneal reflex) produced by the infrared LED irradiation is used as a reference point, and the gazes 141, 142 are detected based on the position of the pupil relative to the position of the corneal reflex. In addition, a method is also known in which the eyes are photographed with a visible light camera (which may also serve as the in-camera 102 in the smartphone 100), and the gazes 141, 142 are detected based on the position of the iris relative to the inner corner of the eye, with the reference point being the inner corner of the eye and the moving point being the iris. The direction of the user's 10 gaze can be identified from the gazes 141 and 142 of both eyes detected by the left eye gaze sensors 105 and 205 and the right eye gaze sensors 106 and 206 .

[0057] The acceleration sensors 161 and 261 are sensors that detect acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, and the like.

[0058] The gyro sensors 162 and 262 are sensors that detect angular velocity in the rotational direction, and can detect vertical, horizontal, and oblique posture states.

[0059] Using the acceleration sensor 161 and gyro sensor 162 of the smartphone 100, it is possible to detect the posture of the smartphone 100, such as its tilt and direction.

[0060] Similarly, using the acceleration sensor 261 and gyro sensor 262 of the HMD 200, it is possible to detect the posture of the HMD 200 worn by the user 10, such as its tilt and direction.

[0061] The geomagnetic sensors 163 and 263 are sensors that detect the earth's magnetic force and detect the direction in which the smartphone 100 and HMD 200 are facing. By using a three-axis type that detects geomagnetism in the vertical direction in addition to the front-back and left-right directions, it is also possible to detect the movement of the smartphone 100 and HMD 200 by capturing changes in geomagnetism with respect to their movement.

[0062] With these sensor groups (a general term for acceleration sensors, gyro sensors, and geomagnetic sensors), it is possible to detect the tilt (posture) of the smartphone 100 and the movement of the head of the user 10 wearing the HMD 200. Therefore, the sensor group mounted on the smartphone 100 corresponds to a posture sensor that outputs posture information indicating the tilt of the smartphone 100. Also, the sensor group mounted on the HMD 200 corresponds to a motion detection sensor that detects the movement of the head of the user 10 and outputs motion amount information.

[0063] When the display screen of the display 165 of the smartphone 100 is difficult to see, etc., the smartphone 100 tends to be in a tilted state, and using the detected and discriminated posture state, it is possible to correct the captured image taken while tilted so that the object to be photographed is in an upright state.

[0064] The processors 167 and 267 are composed of a CPU, etc., and by executing programs 168 and 268 such as an operating system (OS) and operation control applications stored in the memories 170 and 270, they control each component and perform overall operation control processing of the smartphone 100 and HMD 200.

[0065] In addition, the processor 167 of the smartphone 100 may control the operation of the HMD 200 via device - to - device communication (wireless communication).

[0066] For example, when the smartphone 100 detects that the user 10 is not looking at the display screen 191 of the smartphone 100, the processor 167 may transfer the captured image to the HMD 200 and further control the HMD 200 to display the transferred captured image.

[0067] Also, when the smartphone 100 detects that the user 10 is looking at the display screen 101 of the smartphone 100, the processor 167 stops the transfer of the captured image and further sends a display stop signal to stop the display of the captured image that has been transferred to the HMD 200.

[0068] The memories 170, 270 are one form of storage, and are composed of, for example, flash memory, and store programs 168, 268 such as operating systems and operation control applications used by the processors 167, 267.

[0069] Also, they store information data 169, 269 such as images, voices, and signals handled by the smartphone 100 and the HMD 200. Examples of the information data 169, 269 include image data captured by the in - camera 102, the out - camera 103, 203, and when a touch operation of the shutter button 120 or the extended shutter button 134 is detected, the captured image is stored.

[0070] The display 165 of the smartphone 100 is composed of liquid crystal or the like, and a highly transparent touch panel 164 is laminated. The display 165 displays the display content via the touch panel 164.

[0071] The display 165 displays captured images taken by the out - camera 103 and the in - camera 102, various operation buttons such as the camera shooting mode button 107 and the shutter button 120, icons for startup applications and various status displays, and notification information to the user 10.

[0072] The touch panel 164 has, for example, a pressure-sensitive sensor that detects the pressure of contact, and detects a touch operation on the display screen 101, as well as the position and area touched by the hand when holding the display screen 101. The touch panel 164 is not limited to a pressure-sensitive sensor, and there is also a type that has a capacitance sensor that senses a weak current generated when touching the display screen 101, that is, a change in capacitance, or an ultrasonic surface acoustic wave sensor that detects the position of the screen touched by a finger or the like by the attenuation of ultrasonic surface acoustic waves. The same operation can be obtained with any type.

[0073] The touch panel 164 corresponds to an operation input device for the user 10 to input information to the smartphone 100. As another example of an operation input device, an input device such as a keyboard or key button, or a voice input device based on voice input from the microphone 172 may also be used.

[0074] In the case of the optical see-through type HMD 200, the display 265 of the HMD 200 may include, for example, a projection unit that projects virtual objects such as captured images transmitted from the smartphone 100 and notification information to the user 10, and a transparent half mirror that forms an image of the projected virtual objects or the like in front of the eyes.

[0075] Thereby, the user 10 can visually recognize the imaged virtual object in a floating form together with the real object in the visual field range in front of the eyes.

[0076] In the case where the HMD 200 is a video see-through type HMD 200, the display 265 is configured using, for example, a liquid crystal panel that displays the real object in front of the eyes captured by the out camera 203 and virtual objects or the like together. Thereby, the user 10 can visually recognize the real object and virtual objects or the like in the visual field image in front of the eyes in an overlapping manner.

[0077] For the operation input interface 264 of the HMD 200, for example, an input device such as a keyboard, key button, or touch key is used.

[0078] The operation input interface 264 may be provided at a position and in a form that makes it easy for the user 10 to perform input operations within the HMD 200, or may be separated from the main body of the HMD 200 and connected in a wired or wireless form.

[0079] Alternatively, an input operation screen may be displayed within the display screen of the display 265, and input operation information may be captured based on the position on the input operation screen where the visual lines 141 and 142 detected by the left eye visual line sensor 205 and the right eye visual line sensor 206 are directed.

[0080] Furthermore, a pointer may be displayed on the input operation screen, and the pointer may be operated by the operation input interface 264 to capture input operation information.

[0081] Alternatively, the user 10 may utter a voice indicating an input operation, and the microphone 272 may collect the sound and capture the input operation information.

[0082] The microphones 172 and 272 collect external voices and the voices uttered by the user 10 himself / herself, and capture them into the smartphone 100 and the HMD 200.

[0083] When performing voice input, the user 10 may utter voice information indicating the execution of a shutter button operation while looking at the captured image displayed on the HMD 200 without looking at the display screen 101 of the smartphone 100, and the microphones 172 and 272 may collect the voice information, and the shutter button operation may be performed on the smartphone 100.

[0084] The speakers 173 and 273 emit various output information within the smartphone 100 and the HMD 200 as voices, and can notify the user 10 of notification and instruction information by voice. For example, it may notify the user 10 that it has been detected that the user 10 is not looking at the display screen 101 of the smartphone 100, or that the extended shutter button 134 has been set on the display screen 101. Furthermore, it may notify the setting position of the extended shutter button 134, for example, whether it is the entire display screen 101 or is set around the center of the display screen 101.

[0085] Speakers 173 and 273 are one form of voice output devices, and can be used in the same way as speakers 173 and 273 even in earphones and headphones.

[0086] Vibrators 171 and 271 generate vibrations under the control of processors 167 and 267, and convert notification instruction information for user 10 transmitted from smartphone 100 or HMD 200 into vibrations.

[0087] The vibration of vibrator 171 can be transmitted to the hand holding smartphone 100, and the vibration of vibrator 271 can be transmitted to the head of user 10 wearing HMD 200, thereby notifying user 10 of the notification instruction information.

[0088] Device communicators 174 and 274 are communication interfaces for performing wireless communication between smartphone 100 and HMD 200 by short-range wireless communication or wireless LAN.

[0089] Each of device communicators 174 and 274 includes a communication processing circuit, an antenna, etc. corresponding to various communication interfaces, and transmits and receives captured image data, control signals, etc.

[0090] Examples of short-range wireless communication include Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), and HomeRF (Home Radio Frequency, registered trademark), and Wi-Fi (registered trademark) as wireless LAN.

[0091] Telephone network communicator 166 is a communicator for transmitting and receiving data using base station communication or long-distance wireless communication. For example, as a base station communication method, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications) can be used.

[0092] In addition, when dealing with high-definition video, etc., the amount of data used is extremely large. In this case, if a high-speed and large-capacity communication network such as 5G (5th Generation mobile communication system) or local 5G is used for wireless communication, the usability can be improved significantly.

[0093] FIG. 3 is a flowchart for explaining the basic operation of the imaging image transfer system 1 according to the first embodiment.

[0094] In FIG. 3, when the user 10 touches the camera shooting mode button 107 and enters the shooting display mode (S101), the smartphone 100 displays the captured image taken by the in-camera 102 on the display screen 101 (S102). The shooting display mode is a mode in which the captured image taken by the out-camera 103 is displayed on the display screen 101 of the smartphone 100.

[0095] At this time, it is detected whether the user 10 is looking at the display screen 101 of the smartphone 100 (S103).

[0096] When the processor 167 of the smartphone 100 determines that the user 10 is not looking at the display screen 101 (S103: No), the processor 167 of the smartphone 100 stops the display of the captured image, displays the extended shutter button 134 (in the figure, "extended shutter button" is displayed as "extended SB") on the display screen 101 (S104), and transfers the captured image to the HMD 200 (S105). The operation mode of transferring the captured image to the HMD 200 is called the shooting transfer mode.

[0097] The HMD 200 receives the transferred captured image, and the processor 267 displays the received captured image on the display screen 132 of the HMD 200 (S121).

[0098] In this state, when the extended shutter button 134 is touched (S106: Yes), the processor 167 of the smartphone 100 records the captured image in the memory 170 (S107).

[0099] After recording the captured image (S107), or if the extended shutter button 134 is not touched (S106: No), and if the shooting transfer mode of the smartphone 100 has not ended (S108: No), it is determined again whether the user 10 is looking at the display screen 101.

[0100] If it is determined that the user 10 is not looking at the display screen 101 (S109: No), the captured image continues to be transferred to the HMD 200 (S105).

[0101] On the other hand, when the processor 167 of the smartphone 100 determines that the viewpoint of the user 10 is on the display screen 101 while the captured image is being displayed on the HMD 200 (S109: Yes), the processor 167 of the smartphone 100 stops the transfer of the captured image to the HMD 200 and sends a display stop signal (S110). Then, it returns to step S102.

[0102] The HMD 200 stops receiving the transferred image, receives the display stop signal, and erases the displayed captured image (S122). The smartphone 100 returns to step S102 which is the shooting display mode.

[0103] If the user 10 is looking at the display screen 101 of the smartphone 100 (S103: Yes), while looking at the captured image displayed on the display screen 101 of the smartphone 100, the shutter button 120 (displayed as "SB" in the figure) can be touched to record the captured image (S111: Yes, S112).

[0104] If there is no instruction to end the shooting display mode of the smartphone 100 (S113: No), it returns to step S102.

[0105] If there is an instruction to end the shooting display mode of the smartphone 100 (S113: Yes), or if there is an instruction to end the shooting transfer mode of the smartphone 100 (S108: Yes), the shooting image transfer display process from the smartphone 100 to the HMD 200 is ended.

[0106] According to the first embodiment, even in a situation where the display screen 101 on which the captured image of the smartphone 100 is displayed is difficult to view, the user 10 can surely view the captured image with the HMD 200 worn on the head.

[0107] Furthermore, in a situation where the display screen 101 on which the captured image of the smartphone 100 is displayed is difficult to view, it may be difficult to perform a touch operation on the shutter button 120 displayed on the smartphone 100. In that case, since the extended shutter button 134 is displayed on the display screen 101, the shutter button operation can be performed in a region wider than the shutter button 120 on the display screen 101, and it becomes possible to easily perform the shutter button operation for camera shooting with good usability.

[0108] Also, when the display screen 101 of the smartphone 100 can be sufficiently seen from the user 10, the shutter button 120 on the display screen 101 can be operated while directly viewing the captured image displayed on the display screen 101 of the smartphone 100 without transferring and displaying the captured image to the HMD 200.

[0109] <Second Embodiment> The second embodiment is an embodiment in which the inclination of the captured image is corrected so that the captured image is in an upright state, and the corrected captured image 402 is transferred to the HMD 200. Here, the upright state means a state in which the inclination of the subject captured in the captured image is corrected in a direction that cancels out the inclination of the smartphone 100.

[0110] FIG. 4 is a diagram for schematically explaining the appearance of the captured image transfer display system 1a according to the second embodiment.

[0111] When trying to take a camera shot in a state where the display screen 101 of the smartphone 100 is not visible or is difficult to view, the smartphone 100 tends to be in an inclined state, and the subject photographed in this state is displayed as a captured image in a state inclined in the opposite direction by the amount the smartphone 100 is inclined.

[0112] As shown in Fig. 4, when the display screen 101 of the smartphone 10 is not visible to the user 10, for example, when the smartphone 10 is tilted to the right, the image of the subject taken by the camera of the smartphone 10 tilted to the right will be the captured image 401 in which the subject is tilted to the left. Even if this captured image 401 is transferred from the smartphone 10 to the HMD 200 and displayed on the HMD 200, it will be a display screen that is very uncomfortable to view because the subject is tilted for the user 10. In this embodiment, a decrease in the visibility of the captured image due to the tilt of the smartphone 10 is prevented.

[0113] Fig. 5 is a flowchart showing the processing flow of the captured image transfer display system 1a according to the second embodiment. In Fig. 5, for the sake of convenience of explanation, the parts different from the processing of the first embodiment are mainly described, and the common processing is partially omitted.

[0114] When the processor 167 of the smartphone 10 performs control to start the shooting display mode (S101), it displays the captured image on the display screen 101 (S102) and starts acquiring attitude information by the attitude sensor group (S201). The attitude sensor group is a general term for the acceleration sensor 161, the gyro sensor 162, and the geomagnetic sensor 163, and the attitude information is a general term for acceleration information, angular velocity information, and geomagnetic information.

[0115] When the processor 167 of the smartphone 10 determines that the viewpoint of the user 10 is not on the display screen 101 (S103: No), stops the display of the captured image on the display screen 101, and displays the extended shutter button 134 (S104), the processor 167 performs tilt correction on the captured image (S202).

[0116] The processor 167 detects the tilt of the smartphone 10 based on the attitude information, and according to the detected tilt amount of the smartphone 10, performs image correction so that the tilt of the subject captured in the captured image is reversed and the subject is in an upright state without tilt on the display screen 101, and generates a corrected captured image 402.

[0117] The processor 167 transfers the corrected captured image 402 to the HMD 200 (S203), and the processor 267 of the HMD 200 displays the received corrected captured image 402 on the display 265 (S204). Other processes are the same as those in the first embodiment.

[0118] According to this embodiment, since the corrected captured image 402 in which a non-inclined subject is captured is displayed on the display screen 132 of the HMD 200, it can be visually recognized straightforwardly without a sense of discomfort.

[0119] In step S111 and step S106, when a touch operation of the shutter button 120 is detected, it may be configured to be able to select whether to save the captured image in which the subject before correction is inclined or the corrected captured image 402.

[0120] In the above description, the inclination of the smartphone 10 is detected based on the attitude information. However, the method for detecting the inclination of the smartphone 10 may be detected using a well-known image recognition processing algorithm from the captured image without using the attitude information.

[0121] <Third Embodiment> The third embodiment is an embodiment for improving visibility when the hand holding the smartphone 10 covers the display screen 101 of the smartphone 10.

[0122] FIG. 6 is a diagram schematically explaining the appearance of a captured image transfer display system 1b according to the third embodiment.

[0123] As shown in FIG. 6, even when the user 10 can visually see the display screen 101 of the smartphone 10, if the hand holding the smartphone 10 covers the display screen 101 of the smartphone 10, the user 10 cannot visually recognize all of the captured images displayed on the display screen 101. In this embodiment, the visibility reduction of the captured image caused by an obstacle on the display screen 101 of the smartphone 10 is improved.

[0124] FIG. 7 is a flowchart showing the processing flow of the imaging image transfer display system 1b according to the third embodiment. In FIG. 7, for convenience of explanation, the description focuses on the parts different from the processing of the first embodiment, and some of the common processing is omitted.

[0125] The touch panel 164 formed on the display surface of the display 165 detects a touch operation on the touch panel 164 and outputs contact information indicating the touched position and area to the processor 167.

[0126] Therefore, when the processor 167 determines that the viewpoint of the user 10 is on the display screen 101 (S103: Yes), the processor 167 determines whether the touch panel 164 is performing a specific operation for determining whether the user's hand is covering the display screen 101 of the smartphone 100.

[0127] The specific operation is an operation limited to a touch operation that is determined to be covered by the hand in order to prevent the captured image from being transferred to the HMD 200 even though it is a touch operation on the shutter button 120, for example, when the touch panel 164 detects a touch operation.

[0128] For example, a touch operation on a specific area of the display screen 101 may be defined as a specific operation. The specific area may be defined as a specific operation when the touched position is a position that inhibits the visibility of the captured image, for example, not the peripheral part of the display screen 101 but near the center of the screen.

[0129] That is, when the part covered by the hand is within the specific area, the captured image is displayed on the HMD 200. However, when the part covered by the hand is outside the specific area, it is determined that the situation does not cause much hindrance to the user 10's visual recognition of the entire display screen 101, and the captured image may not be displayed on the HMD 200.

[0130] Furthermore, in order to prevent the image from being transferred to the HMD 200 as a result of accidentally touching the specific area, a case where the touch operation on the specific area continues for a predetermined time threshold or more may be defined as a specific operation.

[0131] The specific area may coincide with the area set as the extended shutter button 134.

[0132] When the processor 167 detects a specific operation (S301: Yes), it transfers the captured image to the HMD 200 (S105) and also transmits a display start signal for displaying the transferred captured image.

[0133] If the processor 167 does not detect a specific operation (S301: No), it detects a touch operation of the shutter button 120 (S111).

[0134] According to the present embodiment, when it is detected that the hand holding the smartphone 100 covers the display screen 101 of the smartphone 100, the captured image can be transferred from the smartphone 100 to the HMD 200, and the captured image 131 can be displayed on the display screen 132 of the HMD 200.

[0135] Also, as a modification, the distance between the smartphone 100 and the HMD 200 is detected using the depth sensor 104 of the smartphone 100, or the distance between the smartphone 100 and the HMD 200 is detected using the depth sensor 204 of the HMD 200, and the detected distance information is notified from the HMD 200 to the smartphone 100. In the smartphone 100, when the detected distance is greater than or equal to a distance threshold determined based on the visibility of the user 10, it may be used as a start trigger for transferring the captured image from the smartphone 100 to the HMD 200.

[0136] Thereby, although the user 10 can visually check the display screen 101 of the smartphone 100 to some extent, even when the display screen 101 of the smartphone 100 is too far away for the user 10 to view clearly, the captured image transferred from the smartphone 100 to the HMD 200 is displayed on the display screen 132 of the HMD 200, and the captured image can be easily viewed, improving the convenience.

[0137] <Fourth Embodiment> The fourth embodiment is an embodiment in which both an image that has already been captured by the camera and recorded by a shutter operation (referred to as a "captured image") and an image being captured by the camera (capturing image) are displayed on the HMD 200. Displaying both means not only a mode in which the captured image 601 and the capturing image are displayed in parallel on one screen, but also includes a case where the captured image 601 and the capturing image are alternately displayed. Below, the parallel display will be described as an example.

[0138] FIGS. 8A and 8B are diagrams schematically explaining the appearance of a capturing image transfer display system 1c according to the fourth embodiment.

[0139] Before the captured image 601 and the capturing image 131 are displayed in parallel on the HMD 200, or when they are being displayed in parallel, the captured image 601 is transferred from the smartphone 10 to the HMD 200, stored in the HMD 200, and the processor 267 of the HMD 200 switches the display range 703 and the resolution and displays them on the display 265.

[0140] FIG. 8A shows a case where when the user 10 is not looking at the display screen 101 of the smartphone 10, both the capturing image 131 being captured by the camera and transferred from the smartphone 10 to the HMD 200 and the captured image 601 are displayed on the display screen 133 of the HMD 200.

[0141] Thereby, the user 10 can visually recognize both the capturing image 131 being captured and the captured image 601. That is, when trying to confirm the captured image 601 while looking at the capturing image 131 currently being captured by the camera, there is no need to deliberately switch the display image to the captured image 601, and the usability can be significantly improved.

[0142] FIG. 8B shows a case where the user 10 is looking at the display screen 101 of the smartphone 10. In this case, since the capturing image 131 being captured by the camera is not transferred from the smartphone 10 to the HMD 200, the user 10 visually recognizes the captured image 601 displayed on the display screen 132 of the HMD 200 while directly looking at the capturing image 131 being captured by the camera on the display screen 101 of the smartphone 10.

[0143] In addition, when displaying a plurality of captured images 601, the captured images 601 may be switched by shaking the head.

[0144] Also, on the HMD200 side, the position of the display screen 101 of the smartphone 100 may be detected by the out-camera 203 of the HMD200, and the display location of the captured image 601 may be controlled so as not to overlap with the smartphone display screen.

[0145] FIG. 9 is a flowchart showing the processing flow of the captured image transfer display system 1c according to the fourth embodiment. In FIG. 9, for convenience of explanation, the parts different from the processing of the first embodiment are mainly described, and the common processing is partially omitted.

[0146] When the processor 167 of the smartphone 100 records a captured image in response to a touch operation on the extended shutter button 134 (S107), the recorded captured image 601 is transferred to the HMD200 (S401).

[0147] The processor 267 of the HMD200 acquires the captured image 601 (S402). When performing parallel display of the captured image and the captured image 601 (S403: Yes), the captured image 131 and the captured image 601 are displayed in parallel on the display screen 132 (S404). The parallel display is continued until an end operation of the parallel display is performed (S405: No), and when the parallel display is ended (S405: Yes), the process proceeds to step S121.

[0148] When the user's viewpoint is on the display screen 101 and the processor 167 of the smartphone 100 records an image in response to a touch operation on the shutter button 120 (S112), the captured image 601 is transferred (S406).

[0149] When the processor 267 of the HMD 200 acquires a captured image (S407), it performs object detection on the captured image of the outer camera 203 (S408), and identifies the position where the real image of the display screen 101 of the smartphone 100 can be seen through the display 265. Then, it identifies the area on the display 265 where the display screen 101 cannot be seen (display screen non-visible area), and displays the captured image 601 in the display screen non-visible area (S409). After that, if the display of the captured image 601 is to be terminated (S410: Yes), it proceeds to step S121, and if the display is to be continued (S410: No), it returns to step S407.

[0150] According to the present embodiment, whether the captured image 131 being currently captured by the camera is transferred to the HMD 200 (Fig. 8A) or is to be viewed on the display screen 101 of the smartphone 100 (Fig. 8B), it can be viewed simultaneously with the captured image 601.

[0151] <Fifth Embodiment> The fifth embodiment is an embodiment in which the panoramic image 702 captured by the smartphone 100 is displayed on the HMD 200. This embodiment corresponds to the case where the captured image 601 of the fourth embodiment is a panoramic image 702 obtained by panoramic shooting.

[0152] Fig. 10 is a diagram schematically explaining the appearance of the captured image transfer display system 1d according to the fifth embodiment.

[0153] As shown in Fig. 10, when the user 10 shakes their head, the acceleration sensor 261, gyro sensor 262, and geomagnetic sensor 263 mounted on the HMD 200 are used to detect the head shaking direction and panoramic angle 701 of the user 10 wearing the HMD 200. Then, according to the detected head shaking direction and panoramic angle 701 of the user 10, the display range 703 of the panoramic image 702 is moved.

[0154] Fig. 11 is a flowchart showing the processing flow of the captured image transfer display system 1d according to the fifth embodiment. In Fig. 11, for the sake of convenience of explanation, the parts different from the processing of the fourth embodiment are mainly described, and the common processing is partially omitted.

[0155] When the processor 267 of the HMD 200 receives the captured image 601 (S402, S407), it checks whether the captured image 601 is a panoramic image 702. If it is a normal image that is not the panoramic image 702 (S501: No), it proceeds to steps S403 and S408 and executes the same processing as in the fourth embodiment.

[0156] When the processor 267 determines that the captured image 601 is the panoramic image 702 (S501: Yes), it displays the vicinity of the center of the panoramic image 702 near the center of the display 265 (S502).

[0157] The processor 267 acquires the attitude information from the attitude sensor of the HMD 200 (S503), and changes the display range 703 of the panoramic image 702 according to the attitude information (S504). If the display of the panoramic image is to be continued (S505: No), it returns to step S503. If it is to end (S505: Yes), it proceeds to step S121.

[0158] According to this embodiment, the display range 703 of the already captured panoramic image 702 can be easily changed, and the display range 703 of the panoramic image 702 that the user wants to view can be conveniently selected.

[0159] <Sixth Embodiment> The sixth embodiment is a suitable embodiment for cases such as self - shooting in which the user 10 shoots himself / herself with the smartphone 100.

[0160] FIG. 12 is a diagram schematically explaining the appearance of a photographed image transfer display system 1e according to the sixth embodiment.

[0161] When the user 10 takes a self - portrait while looking at the display screen 101 of the smartphone 100, as shown in FIG. 12, the line of sight 801 of the user 10 is the display screen 101 of the smartphone 100, and there may be a deviation between the line of sight 801 of the user 10 and the position of the in - camera 102 that shoots the user 10.

[0162] In this case, in the image captured by the in-camera 102, since the line of sight of the user 10 is not directed at the in-camera 102, it becomes a captured image of the user 10 that is not the "camera line of sight".

[0163] Therefore, in this embodiment, in the HMD 200, the position of the in-camera 102 of the smartphone 100 is detected, the user 10 himself / herself and the position of his / her eyes are recognized from the captured image, and the display of the captured image transferred from the smartphone 100 is changed so that the two positions match.

[0164] That is, even when the user 10 can visually see the display screen 101 of the smartphone 100, the captured image captured by the in-camera 102 is transferred to the HMD 200, and on the display screen 132 of the HMD 200, the transferred captured image 804 is displayed so that the position 803 of the user 10 on the transferred captured image is aligned with the camera position 802 on the image that the user 10 is visually seeing.

[0165] In this state, when the user 10 turns his / her line of sight to himself / herself on the transferred captured image and takes a self-portrait, in the captured image taken by the self-portrait, it becomes the line-of-sight destination 805 facing the camera, and it is possible to obtain a captured image of the user 10 taken with the "camera line of sight".

[0166] FIG. 13 is a flowchart showing the processing flow of the captured image transfer display system 1e according to the sixth embodiment. In FIG. 5, for the sake of convenience of explanation, the parts different from the processing of the first embodiment are mainly described, and the common processing is partially omitted.

[0167] When the processor 167 of the smartphone 100 detects that it is in the shooting display mode (S101: Yes), it checks whether the in-camera 102 is activated (S601). When it is confirmed that the in-camera 102 is activated (S601: Yes), regardless of whether the line-of-sight destination of the user 10 is on the display screen 101, the captured image is transferred to the HMD 200, and an in-camera shooting notification signal is transmitted, and an in-camera shooting notification signal indicating that shooting is being performed with the in-camera 102 is transmitted (S602).

[0168] When the out-camera 203 of the HMD 200 is activated (S601: No), the captured image is displayed on the display screen 101 (S102).

[0169] When the HMD 200 receives the captured image (S603), the processor 267 performs object detection processing based on the captured image of the out-camera 203 of the HMD 200, and detects the position of the in-camera 102 of the smartphone 100 (S604).

[0170] Furthermore, the processor 267 executes face recognition processing and eye detection processing on the received captured image, and detects the eye region of the user 10 shown in the captured image (S605).

[0171] The processor 267 overlays the detected eye region from the captured image on the position on the display 265 that is on the line of sight of the user 10 when the in-camera 102 is visually recognized through the display 265, and displays the captured image on the display 265 (S606). Then, the process proceeds to step S122.

[0172] According to the present embodiment, when taking a face photo of the user 10 using the in-camera 102, it is possible to assist in taking a photo with the line of sight directed at the in-camera 102.

[0173] As a modification of the present embodiment, since the HMD 200 is captured in the self-taken photo, the processor 167 of the smartphone 100 may perform skin color detection correction processing and eye detection processing on the self-taken photo, and perform image processing to erase the frame and the display 265 of the HMD 200.

[0174] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, for a part of the configuration of each embodiment, addition, deletion, or replacement with other configurations is possible.

[0175] For example, in each embodiment, the case where various shooting operations are performed with the smartphone 100 has been described, but another method may be used. That is, various shooting operations may be performed with the HMD 200, the instruction operations of the various shooting operations performed with the HMD 200 may be detected with the HMD 200, and the detected instruction operations of the various shooting operations may be transferred to the smartphone 100 so that the HMD 200 can give an operation instruction to the smartphone 100. An operation UI (User Interface) for that purpose may be displayed with the HMD 200.

[0176] Also, in the above description, the smartphone 100 has been described as a specific example of the information processing device, and the HMD 200 has been described as a specific example of the image display device. However, the information processing device is not limited to the smartphone 100 and includes all devices with a camera shooting function and a function of displaying a camera shooting screen. The image display device is not limited to the HMD 200 and includes all devices having an image display function. Needless to say, not only shooting the surrounding field of view but also self-shooting is included as an object.

[0177] Furthermore, each of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them, for example, by means of an integrated circuit. Also, each of the above-described configurations, functions, etc. may be realized in software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, files, etc. for realizing each function may be stored in a memory, a recording device such as a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, a DVD, or may be stored in a device on a communication network. Also, the control lines and information lines show those considered necessary for explanation, and not necessarily all control lines and information lines are shown on the product. In practice, it may be considered that almost all configurations are interconnected.

Explanation of Signs

[0178] 1, 1a, 1b, 1c, 1d, 1e: Photographing Image Transfer System 10: User 100: Smartphone 101: Display Screen 102: In-Camera 103, 203: Out-Camera 104, 204: Depth Sensor 105, 205: Left Gaze Sensor 106, 206: Right Gaze Sensor 107: Camera Shooting Mode Button 110, 111, 112, 113, 114, 115: Direction 116, 143, 801, 805: Gaze Destination 120, 151: Shutter Button 130: Arrow 131, 401, 804: Shot Image 132, 133, 191: Display Screen 134: Extended Shutter Button 141, 142: Gaze 161, 261: Acceleration Sensor 162, 262: Gyro Sensor 163, 263: Geomagnetic Sensor 164: Touch panel 165, 265: Display 166: Telephone network communicator 167, 267: Processor 168, 268: Program 169, 269: Information data 170, 270: Memory 171, 271: Vibrator 172, 272: Microphone 173, 273: Speaker 174, 274: Inter-device communicator 180, 280: Bus 200: HMD 202, 402: Corrected captured image 264: Operation input interface 601: Captured image 701: Panorama angle 702: Panorama image 703: Display range 802: Camera position

Claims

1. An information processing device, Camera and a display screen for displaying an image captured by the camera; a gaze sensor that detects a user's gaze and outputs gaze information indicating the user's gaze direction; A communication device for communicating with other image display devices; A storage device for recording the captured image; a processor connected to each of the camera, the display screen, the gaze sensor, the communication device, and the storage; the display screen includes a shutter button for inputting an instruction to record an image in the storage; The processor, receiving, from another image display device via the communication device, gaze determination information indicating whether or not a gaze direction of a user is on the display screen while the camera is capturing an image; when the line of sight determination information indicates that the user's line of sight is not on the display screen, the captured image displayed on the display screen is transferred from the communication device to the other image display device, and a display start signal is transmitted to the other image display device to cause the other image display device to display the transferred captured image; while the captured image is being transferred to the other image display device, displaying the captured image and the shutter button on the display screen is stopped; 23. An information processing apparatus comprising:

2. 2. The information processing device according to claim 1, the display screen further includes an image display area for displaying the captured image, the processor performs control to display an extended shutter button for inputting an instruction to record an image in the storage in an area larger than the shutter button in the image display area while the captured image is being transferred to the other image display device; 23. An information processing apparatus comprising:

3. 2. The information processing device according to claim 1, The processor, when the line-of-sight discrimination information indicating that the user's line of sight is on the display screen is received while the captured image is being transferred to the other image display device, the transfer of the captured image to the other image display device is stopped, and a display stop signal for stopping the display of the transferred captured image is transmitted to the other image display device.

23. An information processing apparatus comprising:

4. 3. The information processing device according to claim 2, The processor, performing control to display the extended shutter button in an area inside a predetermined margin area from an edge of the image display area; 23. An information processing apparatus comprising:

5. 2. The information processing device according to claim 1, Further comprising a touch panel laminated on the display screen, The touch panel detects a touch operation on the touch panel and outputs contact information; the processor is connected to the touch panel; when the touch information is acquired while the camera is taking an image and displaying the taken image on the display screen, the touch information is transmitted to the other image display device.

23. An information processing apparatus comprising:

6. 2. The information processing device according to claim 1, A depth sensor is further provided to detect a distance from the information processing device to an object and output distance information. The processor is further connected to the depth sensor; detecting a distance between the information processing device and the other image display device based on the distance information while the camera is taking an image and the taken image is being displayed on the display screen; when it is determined that the detected distance is equal to or greater than a predetermined distance threshold, the captured image and the display start signal are transmitted to the other image display device.

23. An information processing apparatus comprising:

7. An image display device, A display screen; a gaze sensor that detects a user's gaze and outputs gaze information indicating the user's gaze direction; A communication device for communicating with other information processing devices; a processor connected to each of the display screen, the gaze sensor, and the communication device; The processor, receiving, via the communication device, image capture information indicating whether the information processing device is capturing an image; When the shooting information indicates that shooting is in progress, generating gaze determination information which is a result of determining whether or not the gaze of the user is on a display screen of the information processing device based on the gaze information, when it is determined that the gaze of the user is not on the display screen of the information processing device, the gaze determination information is transmitted to the information processing device via the communication device; when a captured image and a display start signal are received from the information processing device via the communication device, display of the captured image on the display screen is started in response to the display start signal.

1. An image display device comprising:

8. A captured image transfer and display system including an information processing device and an image display device, The information processing device includes: Camera and a first display screen for displaying an image captured by the camera; A first gaze sensor that detects a gaze of a user and outputs gaze information indicating a gaze direction of the user; A first communicator for communicating with another image display device; A storage device for recording the captured image; a first processor connected to each of the camera, the first display screen, the first gaze sensor, the first communication device, and the storage; the first display screen includes a shutter button for inputting an instruction to record an image in the storage; The first processor, receiving, via the first communication device, gaze determination information indicating whether or not a gaze point of a user is on the first display screen from another image display device while the camera is capturing an image; when the line-of-sight determination information indicates that the user's line of sight is not on the first display screen, the captured image displayed on the first display screen is transferred from the first communication device to the other image display device, and a display start signal is transmitted to the other image display device to cause the other image display device to display the transferred captured image; while the captured image is being transferred to the other image display device, displaying the captured image and the shutter button on the first display screen is stopped; The image display device includes: A second display screen; and a second gaze sensor that detects a gaze of a user and outputs gaze information indicating a gaze direction of the user; A second communication device that communicates with another information processing device; a second processor connected to each of the second display screen, the second gaze sensor, and the second communicator; The second processor, receiving, via the second communication device, imaging information indicating whether the information processing device is capturing an image; When the shooting information indicates that shooting is in progress, generating gaze determination information which is a result of determining whether or not the gaze of the user is on a second display screen of the information processing device based on the gaze information; when it is determined that the gaze of the user is not on the second display screen of the information processing device, the gaze determination information is transmitted to the information processing device via the second communication device; when a captured image and a display start signal are received from the information processing device via the second communication device, display of the captured image on the second display screen is started in response to the display start signal. A captured image transfer and display system comprising:

9. 9. The captured image transfer and display system according to claim 8, Recording the captured image in the storage as a captured image; The processor of the information processing device transfers the captured image to the other image display device, a processor of the image display device displays both the captured image and the captured image; A captured image transfer and display system comprising:

10. 10. The captured image transfer and display system according to claim 9, the captured image is a panoramic image captured by the information processing device, the image display device is a head-mounted display that is mounted on the user's head and has a display screen located in front of the user's eyes; the head mounted display includes a motion detection sensor that detects a motion of the head and outputs motion amount information indicating a direction and an amount of the motion; The processor of the head mounted display is further connected to the motion detection sensor; When the panoramic image is received, the center of the panoramic image is aligned with the center of a display screen of the head mounted display, and the image is displayed; a display range of the panoramic image is moved in accordance with the movement amount information and displayed on a display screen of the head mounted display; A captured image transfer and display system comprising:

11. 9. The captured image transfer and display system according to claim 8, the information processing device further includes an in-camera provided on the same surface as the display screen, The processor of the information processing device is further connected to the front camera, the image display device is a head-mounted display that is mounted on the user's head and has a display screen located in front of the user's eyes; The head mounted display further includes an outer camera for capturing an image of the outside world, a processor of the head mounted display connected to the rear camera; The processor of the information processing device When the in-camera is activated and starts generating a captured image, the captured image is transferred to the head mounted display regardless of whether the user's line of sight is on the display screen or not; A processor of the head mounted display executes an object detection process on the image captured by the outer camera to detect a position of the inner camera; A face recognition process and an eye detection process are performed on the captured image transferred from the information processing device, and an eye area of ​​the user shown in the captured image is detected; the captured image transferred from the information processing device is displayed on the display screen of the head mounted display by superimposing the eye area of ​​the captured image at a position where the user views the in-camera through the display screen of the head mounted display; A captured image transfer and display system comprising:

12. A captured image transfer and display method executed by communicating an information processing device equipped with a camera with an image display device that displays an image captured by the camera and received from the information processing device, comprising: displaying, on a display screen of the information processing device, a shutter button for inputting an instruction to capture an image and record the image while the camera is capturing an image; a step of determining by the image display device whether or not a line of sight of a user of the image display device is directed to a display screen provided on the information processing device while the camera is capturing an image; a step of transmitting a result of the determination, when the image display device determines that the user's line of sight is not on the display screen, to the information processing device; when the information processing device receives the determination result, transferring the captured image displayed on the display screen to another image display device and transmitting a display start signal to the other image display device to cause the other image display device to display the transferred captured image; a step of the image display device receiving the captured image and the display start signal; a step of displaying the captured image on a display provided in the image display device in response to the display start signal; stopping display of the captured image and the shutter button on the information processing device while the information processing device is transferring the captured image to the image display device; A method for transferring and displaying a captured image, comprising:

Images