Information processing device, photographed image transfer and display system, and photographed image transfer and display method

Abstract

Provided is an information processing apparatus including a camera, a display screen that displays a photographed image, a line-of-sight sensor that detects the line of sight of a user and outputs line-of-sight information, a communication device that communicates with another image display device, and a processor. The processor: determines, while the camera is shooting, whether the line of sight of the user is directed to the display screen on the basis of the line-of-sight information; transfers the photographed image displayed on the display screen from the communication device to the other image display device when it is determined that the line of sight of the user is not directed to the display screen; and sends a display start signal to cause the other image display device to display the transferred photographed image.

Description

[Technical field]

[0001] The present invention relates to an information processing device, a captured image transfer and display system, and a captured image transfer and display method, and more particularly to an information processing device, an information processing system, and an information processing method for switching a terminal on which an image is displayed. [Background technology]

[0002] Patent document 1 states that "A digital camera consists of a camera body with a built-in image sensor and a head-mounted finder that is attached to the photographer's head. When in use, the head-mounted finder is attached to the photographer's head so that the display of the head-mounted finder is located directly in front of the photographer's eyes. Then, when the photographer points the camera body, held in one hand, at a subject to take a picture, the image is displayed on the display of the head-mounted finder. (Summary excerpt)" [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-60583 A Summary of the Invention [Problem to be solved by the invention]

[0004] Information processing devices equipped with camera and display functions, particularly portable ones, are called smartphones (hereafter referred to as "smartphones"). They are roughly palm-sized and can display images captured with a camera and various information on a display screen; they are easy to use and have become widely used.

[0005] When taking a photo using a smartphone, for example, if there are many people in front of you, it is difficult to hold the smartphone in front of you and take the photo, so you end up reaching out to take the photo. Also, depending on the position of the object being photographed, you may have to take the photo in a situation where the smartphone display screen is difficult to see.

[0006] Therefore, it is conceivable to apply the technology of Patent Document 1 to a smartphone and transfer the captured image to another display device. However, there remains a problem that the captured image may be transferred to another display device despite the user's attempt to view it on the display screen of the smartphone, which is contrary to the user's intention.

[0007] The present invention has been made in consideration of the above-mentioned problems, and aims to support camera photographing operations in conditions in which it is difficult to see the display screen of an information processing device such as a smartphone. [Means for solving the problem]

[0008] As a means for solving the above-mentioned problems, the technology described in the claims is used. As an example, the present invention is an information processing device, comprising: a camera; a display screen for displaying an image captured by the camera; a gaze sensor for detecting a user's gaze and outputting gaze information indicating the gaze direction of the user; and a communicator for communicating with other image display devices. A storage device for recording the captured image; The camera, the display screen, and the line-of-sight sensor ,before Communication device and the storage a processor connected to each of the the display screen includes a shutter button for inputting an instruction to record an image in the storage; The processor determines whether or not the user's line of sight is on the display screen based on the line of sight information while the camera is taking an image, and when it determines that the user's line of sight is not on the display screen, transfers the captured image displayed on the display screen from the communication device to the other image display device, and sends a display start signal to the other image display device to cause the other image display device to display the transferred captured image. Faith , while the captured image is being transferred to the other image display device, displaying the captured image and the shutter button on the display screen is stopped; It is characterized by: Effect of the Invention

[0009] According to the present invention, it is possible to support a camera shooting operation in a situation where it is difficult to see the display screen of an information processing device such as a smartphone. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0010] [Figure 1A] FIG. 1 is a diagram illustrating a schematic external appearance of a captured image transfer system according to a first embodiment. [Figure 1B] FIG. 1 is a diagram illustrating a schematic external appearance of a captured image transfer system according to a first embodiment. [Figure 1C] FIG. 1 is a diagram illustrating a schematic external appearance of a captured image transfer system according to a first embodiment. [Figure 1D] FIG. 1 is a diagram illustrating a schematic external appearance of a captured image transfer system according to a first embodiment. [Figure 1E] FIG. 1 is a diagram illustrating a schematic external appearance of a captured image transfer system according to a first embodiment. [Figure 1F] FIG. 1 is a diagram illustrating a schematic external appearance of a captured image transfer system according to a first embodiment. [Diagram 2] Hardware configuration diagram of smartphone and HMD. [Diagram 3] 4 is a flowchart for explaining a basic operation of the captured image transfer system according to the first embodiment. [Figure 4] FIG. 13 is a diagram for explaining the external appearance of a captured image transfer and display system according to a second embodiment. [Diagram 5] 10 is a flowchart showing the flow of processing in a captured image transfer and display system according to a second embodiment. [Figure 6] FIG. 13 is a diagram for explaining the external appearance of a captured image transfer and display system according to a third embodiment. [Figure 7] 13 is a flowchart showing the flow of processing in a captured image transfer and display system according to the third embodiment. [Figure 8A] FIG. 13 is a diagram for explaining the external appearance of a captured image transfer and display system according to a fourth embodiment. [Figure 8B] FIG. 13 is a diagram for explaining the external appearance of a captured image transfer and display system according to a fourth embodiment. [Figure 9] 13 is a flowchart showing the flow of processing in a captured image transfer and display system according to a fourth embodiment. [Figure 10] FIG. 13 is a diagram for explaining the external appearance of a captured image transfer and display system according to a fifth embodiment. [Figure 11] 13 is a flowchart showing the flow of processing in a captured image transfer and display system according to the fifth embodiment. [Figure 12] FIG. 13 is a diagram for explaining the external appearance of a captured image transfer and display system according to a sixth embodiment. [Figure 13] 13 is a flowchart showing the flow of processing in a captured image transfer and display system according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and duplicated explanations will be omitted.

[0012] First Embodiment The first embodiment is a captured image transfer system 1 that links an information processing device having a camera function with an image display device configured separately from the information processing device, and transfers captured images to the image display device as necessary.

[0013] 1A to 1F are diagrams for explaining the external appearance of a captured image transfer system 1 according to a first embodiment. In the following explanation, a smartphone 100 is taken as a specific example of an information processing device, and a head mounted display (HMD, hereinafter referred to as HMD) 200 is taken as a specific example of an image display device.

[0014] In FIG. 1A, HMD 200 is an image display device that uses Augmented Reality (AR) technology to reflect and extend virtual objects in a virtual space consisting of other images or images created using CG (Computer Graphics), etc., into real space, thereby displaying virtual objects while recognizing actual objects in the real space in three dimensions.

[0015] HMD200 comes in two types: an optical see-through type that displays virtual objects on a display while allowing the user to view real objects in front of the user, and a video see-through type that displays real objects and virtual objects in real space on a display provided on the front of the head, allowing the user 10 to view the real objects and virtual objects they are gazing at up close.

[0016] A user 10 holds a smartphone 100 in his / her hand and wears an HMD 200 on his / her head. The smartphone 100 has an in-camera 102 and an out-camera 103 on the display screen 101 side and the back side, respectively, which capture the surrounding field of view.

[0017] Furthermore, the smartphone 100 is equipped on the display screen 101 side with a depth sensor 104, a left eye gaze sensor 105, and a right eye gaze sensor 106.

[0018] The display screen 101 has a camera shooting mode button 107 which, when touched, activates the camera and transitions to a shooting display mode.

[0019] The in-camera 102 captures images of people and scenery in the surroundings within a shooting range (angle of view) from a direction 110 to a direction 111 .

[0020] The outer camera 103 captures images within a capturing range extending from a direction 112 to a direction 113 .

[0021] The depth sensor 104 measures the distance and angle from the smartphone 100 to an object, and outputs distance information indicating the angle and distance at which the object is located as viewed from the depth sensor 104. Furthermore, the depth sensor 104 is a sensor that can capture the shape of an object such as a person or an object as a three-dimensional object within an observation range from a direction 114 to a direction 115. The depth sensor 104 can detect and identify the user 10 who is looking at the smartphone 100.

[0022] The left eye gaze sensor 105 and the right eye gaze sensor 106 detect the movement and direction of the left eye and the right eye of the face of the user 10 detected and identified by the depth sensor 104, respectively, and can detect the gazes 141, 142 of both the left eye and the right eye to three-dimensionally capture the gaze destination 116. The gaze direction output by the left eye gaze sensor 105 and the right eye gaze sensor 106 corresponds to gaze information, and each point in the gaze direction corresponds to the gaze destination.

[0023] The HMD 200 is worn on the head of the user 10, and a display 265 is placed in front of the user 10. An image is displayed as a virtual object on a display screen 132 (see FIG. 1C) of the display 265. If the display 265 is a see-through type, the virtual object is displayed superimposed on a real object in front of the user's eyes. If the HMD 200 is an immersive type, the virtual object is displayed superimposed on an image captured by the outer camera 203 (see FIG. 3) of the HMD 200, allowing the user to experience augmented reality.

[0024] When the smartphone 100 is powered on, the display screen in the normal menu display mode shown in Fig. 1A is displayed. In the menu display mode, the display screen 101 displays a camera shooting mode button 107 and various application icons, etc. In this state, when the user 10 touches the camera shooting mode button 107, the display screen 101 changes from the normal menu display mode to the shooting display mode.

[0025] 1B, an image captured by the rear camera 103 is displayed on the display screen 101. Also displayed on the display screen 101 is a shutter button 120 for inputting an instruction to record the captured image in the memory 170 (a form of storage) of the smartphone 100. Usually, in order to display the captured image in as wide an area as possible, the shutter button 120 is displayed in a small area that can be touched with a fingertip.

[0026] When the user 10 is looking at the display screen 101, the user 10 can touch the shutter button 120 displayed in a small area at a scene that the user wishes to record while fully viewing the displayed captured image.

[0027] On the other hand, when the display screen 101 is difficult to see or cannot be seen at all, the captured image displayed on the display screen 101 cannot be fully seen. This makes it difficult to properly touch the small area displayed shutter button 120 at the moment of the scene that you want to record.

[0028] This embodiment is intended to deal with such difficult situations, and detects the viewing state of the user 10, and when it is detected that the user 10 is not looking at the display screen 101, transfers the image captured by the outer camera 103 from the smartphone 100 to the HMD 200, as shown in Fig. 1C. In Fig. 1C, an arrow 130 indicates the transfer direction of the captured image.

[0029] The HMD 200 displays the transferred captured image 131 as a virtual object on a display screen 132 of the HMD 200. Using the HMD 200, the user 10 can visually recognize the transferred captured image 131 together with an actual object in front of the user 10 as a virtual object.

[0030] The processor 167 (see FIG. 3) of the smartphone 100 identifies the left eye and the right eye in the face of the user 10 based on the depth sensor information output by the depth sensor 104, detects the direction of the gaze destination 116 based on the left gaze information and right gaze information of the left eye gaze sensor 105 and the right eye gaze sensor 106, respectively, and determines whether the display screen 101 is on the extension of the gaze destination 116. If the display screen 101 is on the gaze destination 116, it is determined that the user 10 is looking at the display screen 101. Note that a single gaze sensor may be used.

[0031] On the other hand, when the depth sensor 104 cannot identify the left and right eyes in the face of the user 10, or when the eyes are identified but are not directed at the display screen 101, the processor 167 determines that the user 10 is not looking at the display screen 101. Also, the gaze directions of the left and right eyes in the face may be detected from the image captured by the in-camera 102 using an image processing algorithm for face recognition.

[0032] The communication between the smartphone 100 and the HMD 200 is in a connected state in advance, so that the captured image can be displayed on the HMD 200 instantly when the transfer mode is entered. Note that even if the connection is established, the HMD 200 does not need to display the captured image from the smartphone 100 at all times. In addition, the display screen 101 of the smartphone 100 displays the shutter button 120 and various operation buttons in addition to the captured image. However, since the necessity for displaying the shutter button 120 and various operation buttons is quite low under normal use conditions as the transfer image transferred from the smartphone 100 to the HMD 200, screen mirroring is not necessarily required, and only the captured image may be used. The transfer decision of the captured image is made by the processor 167 mounted on the smartphone 100. If the processor 167 determines that the gaze destination 143 is not facing the display screen 101 based on the sensor information of the depth sensor 104, the left eye gaze sensor 105, and the right eye gaze sensor 106, it transfers the captured image to the HMD 200. As another example, the processor 267 of the HMD 200 may determine that the gaze point 143 is not directed toward the display screen 101 based on the sensor information of the depth sensor 204, the left eye gaze sensor 205, and the right eye gaze sensor 206 mounted on the HMD 200, and transmit a transfer request signal for the captured image to the smartphone 100, and the smartphone 100 may transfer the captured image in response to the signal. Furthermore, the sensor information of the depth sensor 204, the left eye gaze sensor 205, and the right eye gaze sensor 206 mounted on the HMD 200 may be transmitted to the smartphone 100, and the processor 167 of the smartphone 100 may perform a transfer determination based on the received sensor information. The transfer determination in each of the following embodiments is similar.

[0033] Therefore, even when the display screen 101 of the smartphone 100 is difficult to see or the display screen 101 is not visible at all due to a shooting state in which the display screen 101 cannot be seen fully, the user 10 can look at the captured image 131 displayed on the HMD 200 and clearly see the captured image captured by the outer camera 103 of the smartphone 100.

[0034] Furthermore, the processor 167 of the smartphone 100 determines that the user 10 is not looking at the display screen 101, and while the captured image captured by the outer camera 103 is being transferred to the HMD 200, the display screen 101 is made invisible, and an extended shutter button 134 is provided in the display area of ​​the captured image on the display screen 101, which can detect touch operations and perform shutter button operations.

[0035] Therefore, even if the user 10 has difficulty seeing or cannot see the display screen 101, the user 10 can touch the shutter button 120 in a wide operation area of ​​the display screen 101, making it possible to capture a scene to be recorded reliably and conveniently. That is, when a captured image is transferred to the HMD 200, the shutter can be released even by touching an image display area of ​​the display screen 101 of the smartphone 100 other than the shutter button 120, and an effect is obtained in which the shutter button can be operated even if the user is not looking at the display screen 101 of the smartphone 100.

[0036] Furthermore, it is possible to eliminate unnecessary and wasteful display of captured images that cannot be seen by the user 10, which in turn reduces power consumption. The area that can be touched as the shutter button 120 does not have to be the entire display screen. For example, the extended shutter button 134 may be provided in an area inside a predetermined margin area from the edge of the display screen 101 to avoid the position where the smartphone 100 is held. In FIG. 1C, the extended shutter button 134 is provided only in a limited area inside, not on the entire display screen 101. In this case, it is possible to prevent the shutter button 120 from responding to the hand holding the smartphone 100.

[0037] On the other hand, when it is detected that the user 10 is looking at the display screen 101, as shown in Fig. 1D, the transfer of captured images from the smartphone 100 to the HMD 200 is stopped. The image transferred to the display screen 132 of the HMD 200 is hidden, and the user 10 can directly see the display screen 101 in front of him / her.

[0038] In addition, in the HMD 200 equipped with the depth sensor 204, the left eye gaze sensor 205, and the right eye gaze sensor 206, as shown in FIG. 1E, the depth sensor 204 identifies the display screen of the smartphone 100 in front, and the left eye gaze sensor 205 and the right eye gaze sensor 206 detect the left eye gaze 141 and the right eye gaze 142 to detect whether the gaze destination 143 is facing the display screen 101 of the smartphone 100. When it is detected that the gaze destination 143 is on the display screen 101, the captured image is not transferred from the smartphone 100 to the HMD 200, and the transferred image is not displayed on the HMD 200. That is, the same operation as that shown in FIG. 1D is performed.

[0039] On the other hand, when it is detected that the gaze point 143 is not directed toward the display screen 101, the HMD 200 transmits this detection result information to the smartphone 100 (transfer indicated by arrow 151) as shown in Fig. 1F. In response to the transmitted detection result, the smartphone 100 transmits the image captured by the outer camera 103 from the smartphone 100 to the HMD 200 (transfer indicated by arrow 130) in the same manner as in the case shown in Fig. 1C, and the HMD 200 displays the transmitted captured image 131 on the display screen 132.

[0040] In other words, the processor 267 installed in the HMD 200 determines that the gaze direction 143 is not directed toward the display screen 101 based on the sensor information of the depth sensor 204, the left eye gaze sensor 205, and the right eye gaze sensor 206, and if it determines that the gaze direction is not directed toward the display screen 101, it sends a transfer request signal for the captured image to the smartphone 100, and the smartphone 100 responds by transferring the captured image.

[0041] Alternatively, sensor information from the depth sensor 204, left eye gaze sensor 205, and right eye gaze sensor 206 mounted on the HMD 200 may be transmitted to the smartphone 100, and the processor 167 of the smartphone 100 may determine whether or not to transfer the captured image based on this sensor information.

[0042] When capturing an image for the HMD 200, as in Figures 1A to 1D, the display screen 101 of the smartphone 100 is set to a hidden state, and the shutter button can be operated by touching any area of ​​the display screen 101.

[0043] Therefore, even when detecting on the HMD 200 side whether or not the user 10 is looking at the display screen 101 of the smartphone 100, the same effect and advantages can be obtained as when detecting on the smartphone 100 side whether or not the user 10 is looking at the display screen 101 of the smartphone 100.

[0044] It should be noted that only one gaze sensor is sufficient, and if an gaze sensor is installed in the HMD 200, performance close to that of gaze detection using both eyes can be obtained even when the user 10 is relatively far away from the smartphone 100.

[0045] In the above operations, the information processing device exemplified by the smartphone 100 may instruct and control the operation of the image display device exemplified by the HMD 200, or an information processing system may be configured including the information processing device and the image display device. Thus, even in a situation where the display screen 101 of the information processing device such as the smartphone 100 is difficult for the user 10 to see, the user 10 can clearly check the image captured by the information processing device, and furthermore, the shutter button operation becomes easier, improving usability.

[0046] Next, the hardware configuration of the smartphone 100 and the HMD 200 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the hardware configuration of the smartphone 100 and the HMD 200.

[0047] The smartphone 100 includes an in-camera 102, an out-camera 103, a depth sensor 104, a left eye gaze sensor 105, a right eye gaze sensor 106, an acceleration sensor 161, a gyro sensor 162, a geomagnetic sensor 163, a touch panel 164, a display 165, a telephone network communicator 166, a processor 167 (first processor), a memory 170 in which a program 168 and information data 169 are stored, a vibrator 171, a microphone 172, a speaker 173, and an inter-device communicator 174 (first communicator), and each component is interconnected via a bus 180.

[0048] The HMD 200 includes an outer camera 203, a depth sensor 204, a left eye gaze sensor 205, a right eye gaze sensor 206, an acceleration sensor 261, a gyro sensor 262, a geomagnetic sensor 263, an operation input interface 264, a display 265, a processor 267 (second processor), a memory 270 in which a program 268 and information data 269 are stored, a vibrator 271, a microphone 272, a speaker 273, and an inter-device communicator 274 (second communicator), and each component is interconnected via a bus 280.

[0049] The in-camera 102 converts light incident from a lens into an electrical signal using an imaging element, and captures an image of a user 10 or the like looking at a display 165 .

[0050] The outer cameras 103 and 203 capture images of the surrounding field of view ahead, converting light incident from a lens into an electrical signal by an imaging element and receiving the captured image.

[0051] The rear camera 103 mounted on the smartphone 100 captures an image of the back side of the display 165 of the smartphone 100, and the user 10 can hold the smartphone 100 toward the outer periphery to capture the surrounding field of view.

[0052] The outer camera 203 mounted on the HMD 200 captures an image of the field of view seen from the HMD 200 worn by the user 10 .

[0053] The depth sensors 104 and 204 measure the distance and angle to an object and capture the shape of the object, such as a person or an object, in three dimensions. The depth sensors 104 and 204 may be LiDAR (Light Detection and Ranging), which irradiates the object with infrared or other laser light, measures the scattered light that bounces back, and detects the distance to a distant object, TOF (Time Of Flight) sensors, which measure distance by measuring the reflection time of pulsed light irradiated on the subject for each pixel, or millimeter wave radar, which emits millimeter wave radio waves, captures the reflected waves, and detects the distance to the reflecting object.

[0054] The depth sensor 104 of the smartphone 100 can detect the user 10 when the user 10 is looking at the display screen 101 and identify both eyes from the face of the detected user 10. For example, the overall shape of the user's face can be grasped based on the depth information of the depth sensor 104, and the positions of both eyes can be estimated from the overall shape.

[0055] In addition, the depth sensor 204 of the HMD 200 can identify the distance to the display screen 101 when the display screen 101 of the smartphone 100 is in front of the user 10.

[0056] The left eye gaze sensors 105, 205 and the right eye gaze sensors 106, 206 detect the gazes 141, 142 by capturing the movement and direction of the right eye and the left eye, respectively. The process of detecting the gazes 141, 142 may utilize a well-known technology commonly used as eye tracking processing. For example, as a method using corneal reflex, a technology is known in which an infrared LED (Light Emitting Diode) is irradiated onto the face and photographed with an infrared camera, and the position on the cornea of ​​the reflected light (corneal reflex) produced by the infrared LED irradiation is used as a reference point, and the gazes 141, 142 are detected based on the position of the pupil relative to the position of the corneal reflex. In addition, a method is also known in which the eyes are photographed with a visible light camera (which may also serve as the in-camera 102 in the smartphone 100), and the gazes 141, 142 are detected based on the position of the iris relative to the inner corner of the eye, with the reference point being the inner corner of the eye and the moving point being the iris. The direction of the user's 10 gaze can be identified from the gazes 141 and 142 of both eyes detected by the left eye gaze sensors 105 and 205 and the right eye gaze sensors 106 and 206 .

[0057] The acceleration sensors 161 and 261 are sensors that detect acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, and the like.

[0058] The gyro sensors 162 and 262 are sensors that detect angular velocity in the rotational direction, and can detect vertical, horizontal, and oblique posture states.

[0059] The acceleration sensor 161 and gyro sensor 162 of the smartphone 100 can be used to detect the attitude of the smartphone 100, such as its tilt and direction.

[0060] Similarly, the acceleration sensor 261 and the gyro sensor 262 of the HMD 200 can be used to detect the attitude, such as the tilt and direction, of the HMD 200 worn by the user 10.

[0061] The geomagnetic sensors 163 and 263 are sensors that detect the magnetic force of the earth and detect the direction in which the smartphone 100 and the HMD 200 are facing. By using a three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions, it is also possible to detect the movements of the smartphone 100 and the HMD 200 by capturing changes in the geomagnetism in response to the movements of the smartphone 100 and the HMD 200.

[0062] These sensors (a collective term for acceleration sensors, gyro sensors, and geomagnetic sensors) can detect the tilt (posture) of the smartphone 100 and the movement of the head of the user 10 wearing the HMD 200. Therefore, the sensor group mounted on the smartphone 100 corresponds to a posture sensor that outputs posture information indicating the tilt of the smartphone 100. Also, the sensor group mounted on the HMD 200 corresponds to a motion detection sensor that detects the movement of the head of the user 10 and outputs motion amount information.

[0063] When the display screen of the display 165 of the smartphone 100 is difficult to see, the smartphone 100 is likely to be tilted, and the detected and determined posture state can be used to return an image captured at an angle to a horizontal position, thereby correcting the image so that the subject is upright.

[0064] The processor 167, 267 is composed of a CPU, etc., and controls each component by executing programs 168, 268 such as an operating system (OS) and operation control applications stored in memory 170, 270, and performs overall operation control processing of the smartphone 100 and HMD 200.

[0065] In addition, the processor 167 of the smartphone 100 may control the operation of the HMD 200 via device-to-device communication (wireless communication).

[0066] For example, if the smartphone 100 detects that the user 10 is not looking at the display screen 191 of the smartphone 100, the processor 167 may transfer the captured image to the HMD 200 and further control the HMD 200 to display the transferred captured image.

[0067] In addition, when the smartphone 100 detects that the user 10 is looking at the display screen 101 of the smartphone 100, the processor 167 stops the transfer of the captured image and further sends a display stop signal to the HMD 200 to stop the display of the captured image that has already been transferred to the HMD 200.

[0068] The memories 170 and 270 are one form of storage and are formed, for example, from flash memory, and store programs 168 and 268 such as an operating system and an operation control application used by the processors 167 and 267 .

[0069] The information data 169, 269, such as images, sounds, and signals, handled by the smartphone 100 and the HMD 200, are also stored. The information data 169, 269 includes, for example, image data captured by the in-camera 102 and the out-camera 103, 203, and when a touch operation of the shutter button 120 or the extended shutter button 134 is detected, the captured image is stored.

[0070] The display 165 of the smartphone 100 is made of a liquid crystal or the like, and is laminated with a highly transparent touch panel 164. The display 165 displays the display content via the touch panel 164.

[0071] The display 165 displays images captured by the outer camera 103 and the inner camera 102, various operation buttons such as the camera shooting mode button 107 and the shutter button 120, icons for running applications and various status indications, notification information for the user 10, and the like.

[0072] Touch panel 164 has, for example, a pressure-sensitive sensor that detects contact pressure, and detects touch operations on display screen 101, and the position and area touched by a hand when holding display screen 101. Touch panel 164 is not limited to a pressure-sensitive sensor, and there are also types that have a capacitance sensor that detects a weak current generated when display screen 101 is touched, that is, a change in capacitance, and an ultrasonic surface acoustic wave sensor that detects the position on the screen touched by a finger or the like by the attenuation of ultrasonic surface acoustic waves, and similar operations can be obtained with any type.

[0073] The touch panel 164 corresponds to an operation input device through which the user 10 inputs information to the smartphone 100. Other examples of the operation input device may include an input device such as a keyboard or key buttons, or a voice input device for voice input from the microphone 172.

[0074] In the case of an optical see-through type HMD 200, the display 265 of the HMD 200 may be equipped with, for example, a projection unit that projects virtual objects such as captured images transmitted from the smartphone 100 and notification information to the user 10, and a transparent half mirror that forms an image of the projected virtual objects, etc., and displays them in front of the user's eyes.

[0075] This allows the user 10 to visually recognize the imaged virtual object as if it were floating together with the real objects in the field of view in front of the user's eyes.

[0076] Furthermore, when the HMD 200 is a video see-through type HMD 200, the display 265 is configured using, for example, a liquid crystal panel that displays an actual object in front of the user's eyes photographed by the outer camera 203 together with a virtual object, etc. This allows the user 10 to visually recognize the actual object in the field of view image in front of the user's eyes superimposed on the virtual object, etc.

[0077] The operation input interface 264 of the HMD 200 uses an input device such as a keyboard, key buttons, or touch keys.

[0078] The operation input interface 264 may be provided in a position and form within the HMD 200 that allows the user 10 to easily perform input operations, or may be separated from the main body of the HMD 200 and connected by wire or wirelessly.

[0079] Also, an input operation screen may be displayed within the display screen of the display 265, and input operation information may be acquired based on the positions on the input operation screen to which the lines of sight 141, 142 detected by the left eye line of sight sensor 205 and the right eye line of sight sensor 206 are directed.

[0080] Furthermore, a pointer may be displayed on the input operation screen and the pointer may be operated via the operation input interface 264 to capture input operation information.

[0081] Alternatively, the user 10 may vocalize a voice indicating an input operation, and the voice may be collected by the microphone 272 to capture the input operation information.

[0082] The microphones 172 and 272 collect sounds from the outside and the voice of the user 10 itself, and input the sounds into the smartphone 100 and the HMD 200.

[0083] When performing voice input, the user 10 can pick up voice information indicating that he or she is about to operate the shutter button while looking at the captured image displayed on the HMD 200 without looking at the display screen 101 of the smartphone 100 using microphones 172, 272, and then operate the shutter button on the smartphone 100.

[0084] The speakers 173, 273 emit various kinds of output information in the smartphone 100 and the HMD 200 to the outside as audio, and can inform the user 10 of notifications and instruction information by audio. For example, the speakers may inform the user 10 that it has been detected that the user 10 is not looking at the display screen 101 of the smartphone 100, or that the extended shutter button 134 has been set on the display screen 101. Furthermore, the speakers may inform the user of the setting position of the extended shutter button 134, for example, whether it is set on the entire display screen 101 or around the center of the display screen 101.

[0085] The speakers 173 and 273 are one aspect of an audio output device, and are used in the same manner as the speakers 173 and 273, including earphones and headphones.

[0086] The vibrators 171 and 271 generate vibrations under the control of the processors 167 and 267, and convert notification instruction information sent from the smartphone 100 and HMD 200 to the user 10 into vibrations.

[0087] Vibrator 171 transmits vibrations to the hand holding smartphone 100, and vibrator 271 transmits vibrations to the head of user 10 wearing HMD 200, thereby making it possible to notify user 10 of notification instruction information.

[0088] The inter-device communicators 174 and 274 are communication interfaces that perform wireless communication between the smartphone 100 and the HMD 200 via short-range wireless communication or wireless LAN.

[0089] Each of the inter-device communicators 174 and 274 includes a communication processing circuit, an antenna, and the like corresponding to various communication interfaces, and transmits and receives captured image data, control signals, and the like.

[0090] Examples of short-range wireless communication include Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), and HomeRF (Home Radio Frequency, registered trademark), and an example of wireless LAN is Wi-Fi (registered trademark).

[0091] The telephone network communication device 166 is a communication device for transmitting and receiving data using base station communication or long-distance wireless communication. For example, as a method of base station communication, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications) may be used.

[0092] In addition, when dealing with high-definition video, the amount of data used is dramatically larger. In such cases, using high-speed, large-capacity communication networks such as 5G (5th Generation: 5th generation mobile communication system) and local 5G for wireless communication can dramatically improve usability.

[0093] FIG. 3 is a flowchart for explaining the basic operation of the photographed image transfer system 1 according to the first embodiment.

[0094] 3, when the user 10 touches the camera shooting mode button 107 to enter the shooting display mode (S101), the smartphone 100 displays an image captured by the in-camera 102 on the display screen 101 (S102). The shooting display mode is a mode in which an image captured by the out-camera 103 is displayed on the display screen 101 of the smartphone 100.

[0095] At this time, it is detected whether the user 10 is looking at the display screen 101 of the smartphone 100 (S103).

[0096] When the processor 167 of the smartphone 100 determines that the user 10 is not looking at the display screen 101 (S103: No), the processor 167 of the smartphone 100 stops displaying the captured image, displays the extended shutter button 134 (in the figure, "extended shutter button" is displayed as "Extended SB") on the display screen 101 (S104), and transfers the captured image to the HMD 200 (S105). The operating mode in which the captured image is transferred to the HMD 200 is called a capture and transfer mode.

[0097] The HMD 200 receives the transferred captured image, and the processor 267 displays the received captured image on the display screen 132 of the HMD 200 (S121).

[0098] In this state, when the extended shutter button 134 is touched (S106: Yes), the processor 167 of the smartphone 100 records the captured image in the memory 170 (S107).

[0099] After the captured image is recorded (S107), or if the extended shutter button 134 is not touched (S106: No) and the smartphone 100's capture and transfer mode has not ended (S108: No), it is again determined whether the user 10 is looking at the display screen 101.

[0100] If it is determined that the user 10 is not looking at the display screen 101 (S109: No), the captured image continues to be transferred to the HMD 200 (S105).

[0101] On the other hand, when the captured image is displayed on the HMD 200, if the processor 167 of the smartphone 100 determines that the viewpoint of the user 10 is on the display screen 101 (S109: Yes), the processor 167 of the smartphone 100 stops the transfer of the captured image to the HMD 200 and transmits a display stop signal (S110). Then, the process returns to step S102.

[0102] The HMD 200 stops receiving the transferred image, receives the display stop signal, and erases the captured image that is being displayed (S122). The smartphone 100 returns to step S102 in the captured image display mode.

[0103] When the user 10 is looking at the display screen 101 of the smartphone 100 (S103: Yes), the user 10 can record the captured image by touching the shutter button 120 (shown as "SB" in the figure) while looking at the captured image displayed on the display screen 101 of the smartphone 100 (S111: Yes, S112).

[0104] If there is no instruction to end the shooting display mode of the smartphone 100 (S113: No), the process returns to step S102.

[0105] If there is an instruction to end the shooting display mode of the smartphone 100 (S113: Yes), or if there is an instruction to end the shooting transfer mode of the smartphone 100 (S108: Yes), the shooting image transfer and display process from the smartphone 100 to the HMD 200 is terminated.

[0106] According to the first embodiment, even in a situation where the display screen 101 on which the captured image of the smartphone 100 is displayed is difficult to see, the user 10 can reliably view the captured image on the HMD 200 worn on the head.

[0107] Furthermore, in a situation where it is difficult to see the display screen 101 on which the captured image of the smartphone 100 is displayed, it may be difficult to touch the shutter button 120 displayed on the smartphone 100. In that case, the extended shutter button 134 is displayed on the display screen 101, so that the shutter button operation can be performed in an area on the display screen 101 that is larger than the shutter button 120, making it easy to use and possible to operate the shutter button for camera photography.

[0108] In addition, if the user 10 can see the display screen 101 of the smartphone 100 sufficiently, he or she can operate the shutter button 120 on the display screen 101 while directly viewing the captured image displayed on the display screen 101 of the smartphone 100, without transferring and displaying the captured image to the HMD 200.

[0109] <Second embodiment> The second embodiment is an embodiment in which the tilt of a captured image is corrected so that the captured image is in an upright state, and the corrected captured image 402 is transferred to the HMD 200. The upright state here refers to a state in which the tilt of the subject captured in the captured image is corrected in a direction that offsets the tilt of the smartphone 100.

[0110] FIG. 4 is a diagram for explaining a schematic external appearance of a captured image transfer and display system 1a according to the second embodiment.

[0111] When attempting to take a picture with the smartphone 100 in a state where the display screen 101 is not visible or is difficult to see, the smartphone 100 is likely to become tilted, and the subject photographed with the smartphone 100 in this state is displayed in the captured image tilted in the opposite direction to the amount that the smartphone 100 was tilted.

[0112] 4, when the display screen 101 of the smartphone 100 is not visible to the user 10, for example, when the smartphone 100 is tilted to the right, an image of a subject captured by the camera of the smartphone 100 tilted to the right becomes a captured image 401 displayed with the subject tilted to the left. Even if this captured image 401 is transferred from the smartphone 100 to the HMD 200 and displayed on the HMD 200, the subject is tilted to the user 10, resulting in a display screen that is very difficult to see. In this embodiment, a decrease in visibility of the captured image caused by the tilt of the smartphone 100 is prevented.

[0113] Fig. 5 is a flowchart showing the flow of processing in the captured image transfer and display system 1a according to the second embodiment. For ease of explanation, Fig. 5 mainly describes the parts that are different from the processing in the first embodiment, and omits some of the common processing.

[0114] When the processor 167 of the smartphone 100 performs control to start the shooting display mode (S101), the captured image is displayed on the display screen 101 (S102), and the acquisition of posture information by the posture sensor group is started (S201). The posture sensor group is a general term for the acceleration sensor 161, the gyro sensor 162, and the geomagnetic sensor 163, and the posture information is a general term for the acceleration information, the angular velocity information, and the geomagnetic information.

[0115] When the processor 167 of the smartphone 100 determines that the user 10's viewpoint is not on the display screen 101 (S103: No), stops displaying the captured image on the display screen 101, and displays the extended shutter button 134 (S104), the processor 167 performs tilt correction of the captured image (S202).

[0116] The processor 167 detects the tilt of the smartphone 100 based on the posture information, and performs image correction according to the detected amount of tilt of the smartphone 100 to reverse the tilt of the subject captured in the captured image so that the subject is in an upright state without tilt on the display screen 101, and generates a corrected captured image 402.

[0117] The processor 167 transfers the corrected captured image 402 to the HMD 200 (S203), and the processor 267 of the HMD 200 displays the received corrected captured image 402 on the display 265 (S204). The other processes are the same as those in the first embodiment.

[0118] According to this embodiment, the corrected captured image 402, which is an image of a subject that is not tilted, is displayed on the display screen 132 of the HMD 200, so that the image can be viewed normally and without any sense of discomfort.

[0119] In steps S111 and S106, when a touch operation of the shutter button 120 is detected, it may be possible to select whether to save the pre-correction captured image with a tilted subject or the corrected captured image 402.

[0120] In the above explanation, the tilt of the smartphone 100 was detected based on the orientation information, but the method of detecting the tilt of the smartphone 100 may also be to detect it from the captured image using a well-known image recognition processing algorithm without using the orientation information.

[0121] <Third embodiment> The third embodiment is an embodiment that improves visibility when the hand holding the smartphone 100 covers and hides the display screen 101 of the smartphone 100.

[0122] FIG. 6 is a diagram for explaining a schematic external appearance of a captured image transfer and display system 1b according to the third embodiment.

[0123] 6, even if the user 10 can see the display screen 101 of the smartphone 100, if the hand holding the smartphone 100 covers the display screen 101 of the smartphone 100, the user 10 cannot see all of the captured image displayed on the display screen 101. In this embodiment, the decrease in visibility of the captured image caused by an obstacle on the display screen 101 of the smartphone 100 is improved.

[0124] Fig. 7 is a flowchart showing the flow of processing in the captured image transfer and display system 1b according to the third embodiment. For ease of explanation, Fig. 7 mainly describes the parts that are different from the processing in the first embodiment, and omits some of the common processing.

[0125] Touch panel 164 arranged on the display surface of display 165 detects a touch operation on touch panel 164 and outputs contact information indicating the touched position and area to processor 167.

[0126] Therefore, when the processor 167 determines that the user 10's viewpoint is on the display screen 101 (S103: Yes), it determines whether the touch panel 164 is performing a specific action to determine whether the user 10's hand is covering the display screen 101 of the smartphone 100.

[0127] A specific action is an action limited to touch operations that, when the touch panel 164 detects a touch operation, determine that the hand is covering the touch operation, for example, the shutter button 120, in order to prevent a captured image from being transferred to the HMD 200.

[0128] For example, a specific action may be defined as a touch operation on a specific area of ​​the display screen 101. The specific action may be defined as a specific area when the touched position is a position that impairs the visibility of a captured image, for example, near the center of the display screen 101 rather than the peripheral area.

[0129] In other words, if the part covered by the hand is within the specific area, the captured image is displayed on the HMD 200, but if the part covered by the hand is outside the specific area, it is determined that the situation does not significantly impede the user 10's ability to see the entire display screen 101, and the captured image does not need to be displayed on the HMD 200.

[0130] Furthermore, in order to prevent the result of accidentally touching the specific area from being transferred to the HMD 200, a case in which a touch operation on the specific area continues for a predetermined time threshold or longer may be determined as a specific action.

[0131] The specific area may coincide with the area set as the extended shutter button 134.

[0132] When the processor 167 detects a specific action (S301: Yes), it transfers the captured image to the HMD 200 (S105) and also transmits a display start signal for displaying the transferred captured image.

[0133] If the processor 167 does not detect the specific action (S301: No), it detects the touch operation of the shutter button 120 (S111).

[0134] According to this embodiment, when it is detected that the hand holding the smartphone 100 is covering the display screen 101 of the smartphone 100, the captured image is transferred from the smartphone 100 to the HMD 200, and the captured image 131 can be displayed on the display screen 132 of the HMD 200.

[0135] As a variant example, the distance between the smartphone 100 and the HMD 200 can be detected using the depth sensor 104 of the smartphone 100, or the distance between the smartphone 100 and the HMD 200 can be detected using the depth sensor 204 of the HMD 200, and the detected distance information can be notified from the HMD 200 to the smartphone 100. When the detected distance is greater than a distance threshold determined based on the visibility of the user 10, this can be used as a start trigger for transferring the captured image from the smartphone 100 to the HMD 200.

[0136] This allows the user 10 to see the display screen 101 of the smartphone 100, but even if the display screen 101 of the smartphone 100 is too far away for the user 10 to see easily, the captured image transferred from the smartphone 100 to the HMD 200 can be displayed on the display screen 132 of the HMD 200, making it possible to easily view the captured image and increasing convenience.

[0137] <Fourth embodiment> The fourth embodiment is an embodiment in which both an image that has already been captured by a camera and recorded by a shutter operation (called a "captured image") and an image being captured by the camera (captured image) are displayed together on the HMD 200. Displaying both includes a mode in which a captured image 601 and a captured image are displayed side by side on one screen, as well as a case in which the captured image 601 and the captured image are displayed in a switched manner. The following description will be given taking the side-by-side display as an example.

[0138] 8A and 8B are diagrams for explaining the external appearance of a captured image transfer and display system 1c according to the fourth embodiment.

[0139] The captured image 601 is transferred from the smartphone 100 to the HMD 200 before or when the captured image 601 and the captured image 131 are displayed side by side on the HMD 200, and is stored in the HMD 200. The processor 267 of the HMD 200 switches the display range 703 and resolution and displays the image on the display 265.

[0140] Figure 8A shows a case where, when the user 10 is not looking at the display screen 101 of the smartphone 100, both the captured image 131 captured by the camera and the captured image 601 transferred from the smartphone 100 to the HMD 200 are displayed on the display screen 133 of the HMD 200.

[0141] This allows the user 10 to simultaneously view both the image being shot 131 and the already-shot image 601. That is, when the user 10 is viewing the image being shot 131 currently being shot by the camera and wants to check the already-shot image 601, there is no need to take the trouble of switching the displayed image to the already-shot image 601, which can greatly improve usability.

[0142] Figure 8B shows the case where the user 10 is looking at the display screen 101 of the smartphone 100. In this case, the captured image 131 during shooting with the camera is not transferred from the smartphone 100 to the HMD 200, so the user 10 can directly view the captured image 131 during shooting with the camera on the display screen 101 of the smartphone 100 while also viewing the captured image 601 displayed on the display screen 132 of the HMD 200.

[0143] When a plurality of captured images 601 are displayed, the captured images 601 may be switched by shaking the head.

[0144] In addition, on the HMD 200 side, the position of the display screen 101 of the smartphone 100 may be detected by the outer camera 203 of the HMD 200, and the display location of the captured image 601 may be controlled so that it does not overlap with the smartphone display screen.

[0145] Fig. 9 is a flowchart showing the flow of processing of the captured image transfer and display system 1c according to the fourth embodiment. For ease of explanation, Fig. 9 mainly describes the parts that are different from the processing of the first embodiment, and omits some of the common processing.

[0146] When the processor 167 of the smartphone 100 records a captured image in response to a touch operation of the extended shutter button 134 (S107), the processor 167 transfers the recorded captured image 601 to the HMD 200 (S401).

[0147] The processor 267 of the HMD 200 acquires the captured image 601 (S402). If the captured image and the captured image 601 are to be displayed side by side (S403: Yes), the captured image 131 and the captured image 601 are displayed side by side on the display screen 132 (S404). The parallel display continues until an operation to end the parallel display is performed (S405: No), and when the parallel display is ended (S405: Yes), the process proceeds to step S121.

[0148] When the user's 10 viewpoint is on the display screen 101, the processor 167 of the smartphone 100 also transfers the captured image 601 (S406) when an image is recorded in response to a touch operation of the shutter button 120 (S112).

[0149] When the processor 267 of the HMD 200 acquires the captured image (S407), it performs object detection in the captured image of the outer camera 203 (S408) and identifies the position where the real image of the display screen 101 of the smartphone 100 is visible through the display 265. Then, it identifies an area in the display 265 where the display screen 101 is not visible (display screen non-visible area) and displays the captured image 601 in the display screen non-visible area (S409). After that, or if the display of the captured image 601 is to be ended (S410: Yes), the process proceeds to step S121, and if the display is to be continued (S410: No), the process returns to step S407.

[0150] According to this embodiment, when the image 131 currently being captured by the camera is transferred to the HMD 200 (FIG. 8A) or viewed on the display screen 101 of the smartphone 100 (FIG. 8B), it can be viewed simultaneously with the captured image 601.

[0151] <Fifth embodiment> The fifth embodiment is an embodiment in which a panoramic image 702 captured by the smartphone 100 is displayed on the HMD 200. This embodiment corresponds to the case in which the captured image 601 in the fourth embodiment is a panoramic image 702 captured by panoramic photography.

[0152] FIG. 10 is a diagram for explaining a schematic external appearance of a captured image transfer and display system 1d according to the fifth embodiment.

[0153] 10, when the user 10 shakes his / her head, the acceleration sensor 261, the gyro sensor 262, and the geomagnetic sensor 263 mounted on the HMD 200 are used to detect the head shaking direction and panoramic angle 701 of the user 10 wearing the HMD 200. Then, according to the detected head shaking direction and panoramic angle 701 of the user 10, a display range 703 of a panoramic image 702 is moved.

[0154] Fig. 11 is a flowchart showing the flow of processing of the captured image transfer and display system 1d according to the fifth embodiment. For the sake of convenience, Fig. 11 mainly describes the parts that are different from the processing of the fourth embodiment, and omits some of the common processing.

[0155] When the processor 267 of the HMD 200 receives the captured image 601 (S402, S407), it checks whether the captured image 601 is a panoramic image 702, and if it is a normal image and not a panoramic image 702 (S501: No), it proceeds to steps S403 and S408 and executes processing similar to that of the fourth embodiment.

[0156] When the processor 267 determines that the captured image 601 is a panoramic image 702 (S501: Yes), the center of the panoramic image 702 is aligned with the center of the display 265 and displayed (S502).

[0157] The processor 267 acquires orientation information from the orientation sensor of the HMD 200 (S503), and changes the display range 703 of the panoramic image 702 in accordance with the orientation information (S504). If the display of the panoramic image is to be continued (S505: No), the process returns to step S503, and if the display is to be ended (S505: Yes), the process proceeds to step S121.

[0158] According to this embodiment, the display range 703 of a panoramic image 702 that has already been captured can be easily changed, and the display range 703 of the panoramic image 702 that one wishes to view can be conveniently selected.

[0159] Sixth embodiment The sixth embodiment is an embodiment suitable for cases such as taking a selfie of the user 10 using the smartphone 100.

[0160] FIG. 12 is a diagram for explaining a schematic external appearance of a captured image transfer and display system 1e according to the sixth embodiment.

[0161] When a user 10 takes a selfie by photographing himself / herself with a camera while looking at the display screen 101 of a smartphone 100, as shown in FIG. 12, the direction of the user's 10 gaze 801 is the display screen 101 of the smartphone 100, and the position of the user's 10 gaze 801 may be misaligned with the position of the in-camera 102 photographing the user 10.

[0162] In this case, in the image captured by the inner camera 102, the line of sight of the user 10 is not directed toward the inner camera 102, and the captured image is of the user 10 not looking toward the camera.

[0163] Therefore, in this embodiment, the HMD 200 detects the position of the in-camera 102 of the smartphone 100, recognizes the position of the user 10 himself and his eyes from the captured image, and changes the display of the captured image transferred from the smartphone 100 so that the two positions match.

[0164] In other words, even when the user 10 can see the display screen 101 of the smartphone 100, the captured image captured by the in-camera 102 is transferred to the HMD 200, and the transferred captured image 804 is displayed on the display screen 132 of the HMD 200 so that the position 803 of the user 10 on the transferred captured image is aligned with the camera position 802 on the image that the user 10 is viewing.

[0165] In this state, when the user 10 takes a selfie by directing his / her gaze at himself / herself in the transferred photographed image, the gaze direction 805 in the photographed selfie will be directed toward the camera, and an image of the user 10 taken with the gaze of the camera can be obtained.

[0166] Fig. 13 is a flowchart showing the flow of processing of the captured image transfer and display system 1e according to the sixth embodiment. For the sake of convenience, Fig. 5 mainly describes the parts that are different from the processing of the first embodiment, and omits some of the common processing.

[0167] When the processor 167 of the smartphone 100 detects that the smartphone 100 is in the shooting display mode (S101: Yes), the processor 167 checks whether the in-camera 102 is activated (S601). When the processor 167 checks that the in-camera 102 is activated (S601: Yes), the processor 167 transfers the captured image to the HMD 200 and transmits an in-camera shooting notification signal indicating that shooting is being performed with the in-camera 102, regardless of whether the user 10 is looking at the display screen 101 or not (S602).

[0168] If the outer camera 203 is activated (S601: No), a captured image is displayed on the display screen 101 (S102).

[0169] When the HMD 200 receives the captured image (S603), the processor 267 performs object detection processing based on the image captured by the outer camera 203 of the HMD 200, and detects the position of the inner camera 102 of the smartphone 100 (S604).

[0170] Furthermore, the processor 267 executes face recognition processing and eye detection processing on the received photographed image, and detects the eye area of ​​the user 10 appearing in the photographed image (S605).

[0171] The processor 267 displays the captured image on the display 265 by superimposing the eye area detected from the captured image on the position on the display 265 that is in the line of sight of the user 10 when the user 10 views the in-camera 102 through the display 265 (S606). Then, the process proceeds to step S122.

[0172] According to this embodiment, when taking a photograph of the face of the user 10 using the in-camera 102, assistance can be provided for taking a photograph while the line of sight is directed to the in-camera 102.

[0173] As a variation of this embodiment, since the HMD 200 is captured in a selfie, the processor 167 of the smartphone 100 may perform skin color detection and correction processing and eye detection processing on the selfie, and perform image processing to erase the frame and display 265 of the HMD 200.

[0174] The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0175] For example, in each embodiment, the case where various shooting operations are performed by the smartphone 100 has been described, but another method may be used. That is, various shooting operations may be performed by the HMD 200, the instruction operations of the various shooting operations performed by the HMD 200 may be detected by the HMD 200, and the detected instruction operations of the various shooting operations may be transferred to the smartphone 100, so that the HMD 200 performs operation instructions for the smartphone 100. An operation UI (User Interface) for this purpose may be displayed on the HMD 200.

[0176] In the above description, the smartphone 100 is a specific example of an information processing device, and the HMD 200 is a specific example of an image display device, but the information processing device is not limited to the smartphone 100 but includes all devices with a camera shooting function and a camera shooting screen display function, and the image display device is not limited to the HMD 200 but includes all devices with an image display function. Needless to say, this includes not only shooting the surrounding field of view, but also selfies.

[0177] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing a part or all of them as an integrated circuit. In addition, each of the above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, DVD, or may be stored in a device on a communication network. In addition, the control lines and information lines are shown as those that are considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product. In reality, it may be considered that almost all configurations are connected to each other. [Explanation of symbols]

[0178] 1, 1a, 1b, 1c, 1d, 1e: Image capture and transfer system 10: User 100: Smartphone 101:Display screen 102: In-camera 103, 203: Rear camera 104, 204: Depth sensor 105, 205: Left eye gaze sensor 106, 206: Right eye gaze sensor 107: Camera shooting mode button 110, 111, 112, 113, 114, 115: Direction 116, 143, 801, 805: Line of sight 120, 151: Shutter button 130: Arrow 131, 401, 804: Photographed images 132, 133, 191: Display screen 134: Extended Shutter Button 141, 142: Gaze 161, 261: Acceleration sensor 162, 262: Gyro sensor 163, 263: Geomagnetic sensor 164: Touch panel 165, 265: Display 166:Telephone network communication equipment 167, 267: Processor 168, 268: Program 169, 269: Information data 170, 270: Memory 171, 271: Vibrator 172, 272: Mike 173, 273: Speaker 174, 274: Inter-device communication device 180, 280: Bus 200: HMD 202, 402: Corrected images 264: Operation input interface 601: Captured image 701: Panoramic angle 702 : Panorama image 703: Display range 802: Camera position

Claims

1. An information processing device, A camera and a display screen for displaying an image captured by the camera; a gaze sensor that detects a user's gaze and outputs gaze information indicating the user's gaze direction; A communication device for communicating with other image display devices; A storage device for recording the captured image; a processor connected to each of the camera, the display screen, the gaze sensor, the communication device, and the storage; the display screen includes a shutter button for inputting an instruction to record an image in the storage; The processor, determining whether the user's gaze is directed to the display screen based on the gaze information while the camera is capturing an image; when it is determined that the user's line of sight is not on the display screen, the captured image displayed on the display screen is transferred from the communication device to the other image display device, and a display start signal is transmitted to the other image display device to cause the other image display device to display the transferred captured image; while the captured image is being transferred to the other image display device, displaying the captured image and the shutter button on the display screen is stopped; 23. An information processing apparatus comprising:

2. 2. The information processing device according to claim 1, the display screen further includes an image display area for displaying the captured image, the processor performs control to display an extended shutter button for inputting an instruction to record an image in the storage in an area larger than the shutter button in the image display area while the captured image is being transferred to the other image display device; 23. An information processing apparatus comprising:

3. 2. The information processing device according to claim 1, The processor, when it is determined that the gaze of the user is on the display screen while the captured image is being transferred to the other image display device, the transfer of the captured image to the other image display device is stopped, and a display stop signal for stopping the display of the transferred captured image is transmitted to the other image display device.

23. An information processing apparatus comprising:

4. 3. The information processing device according to claim 2, The processor, performing control to display the extended shutter button in an area inside a predetermined margin area from an edge of the image display area; 23. An information processing apparatus comprising:

5. 2. The information processing device according to claim 1, Further comprising a touch panel laminated on the display screen, The touch panel detects a touch operation on the touch panel and outputs contact information; the processor is connected to the touch panel; when the touch information is acquired while the camera is taking an image and displaying the taken image on the display screen, the touch information is transmitted to the other image display device.

23. An information processing apparatus comprising:

6. 2. The information processing device according to claim 1, A depth sensor is further provided to detect a distance from the information processing device to an object and output distance information. The processor is further connected to the depth sensor; detecting a distance between the information processing device and the other image display device based on the distance information while the camera is taking an image and the taken image is being displayed on the display screen; when it is determined that the detected distance is equal to or greater than a predetermined distance threshold, the captured image and the display start signal are transmitted to the other image display device.

23. An information processing apparatus comprising:

7. A captured image transfer and display system configured by communicating and connecting an information processing device equipped with a camera and an image display device that displays an image captured by the camera and received from the information processing device, The information processing device includes: a display screen for displaying an image captured by the camera; A gaze sensor that outputs gaze information indicating a gaze direction of a user; A first communicator for communicating with another image display device; a first processor connected to each of the camera, the display screen, the line of sight sensor, and the first communicator; a storage for recording the captured image as a captured image; The image display device includes: A second communication device that communicates with the information processing device; a display that displays the captured image transferred from the information processing device; and a second processor connected to each of the display and the second communicator; The display screen includes: a shutter button for inputting an instruction to record an image in the storage, and an image display area for displaying the captured image, the first processor is further coupled to the storage; determining whether the user's gaze is on the display screen based on the gaze information while the information processing device is capturing an image; when it is determined that the user's line of sight is not on the display screen, the captured image displayed on the display screen is transferred from the first communication device to the other image display device, and a display start signal is transmitted to the other image display device to cause the other image display device to display the transferred captured image; The second processor When the captured image and the display start signal are received via the second communication device, the captured image is displayed on the display in response to the display start signal; The first processor transfers the captured image to the other image display device; The second processor displays both the captured image and the captured image. A captured image transfer and display system comprising:

8. The captured image transfer and display system according to claim 7, the captured image is a panoramic image captured by the information processing device, the image display device is a head-mounted display that is mounted on the user's head and has a display positioned in front of the user's eyes, the head mounted display includes a motion detection sensor that detects a motion of the head and outputs motion amount information indicating a direction and an amount of the motion; the second processor is further connected to the motion detection sensor; Upon receiving the panoramic image, displaying the panoramic image with the center of the image aligned with the center of the display; a display range of the panoramic image is moved in accordance with the movement amount information and displayed on the display; A captured image transfer and display system comprising:

9. The captured image transfer and display system according to claim 7, the information processing device further includes an in-camera provided on the same surface as the display screen, The first processor is further connected to the front camera, the image display device is a head-mounted display that is mounted on the user's head and has a display positioned in front of the user's eyes, The head mounted display further includes an outer camera for capturing an image of the outside world, the second processor is connected to the rear camera; The first processor, When the in-camera is activated and starts generating a captured image, the captured image is transferred to the head mounted display regardless of whether the user's line of sight is on the display screen or not; The second processor Executing an object detection process on an image captured by the outer camera to detect a position of the inner camera; A face recognition process and an eye detection process are performed on the captured image transferred from the information processing device, and an eye area of ​​the user shown in the captured image is detected; the captured image transferred from the information processing device is displayed on the display in such a manner that the eye area of ​​the captured image is superimposed on a position where the user views the in-camera through the display; A captured image transfer and display system comprising:

10. A method for transferring and displaying captured images, the method being executed by communicating an information processing device equipped with a camera with an image display device that displays captured images captured by the camera and received from the information processing device, comprising: displaying, on a display screen of the information processing device, a shutter button for inputting an instruction to capture an image and record the image while the camera is capturing an image; a step of determining by the information processing device whether or not a line of sight of a user of the information processing device is on a display screen provided on the information processing device while the camera is photographing; when the information processing device determines that the user's gaze is not on the display screen, the information processing device transfers the captured image displayed on the display screen to another image display device, and transmits a display start signal to the other image display device to cause the other image display device to display the transferred captured image; a step of the image display device receiving the captured image and the display start signal; a step of displaying the captured image on a display provided in the image display device in response to the display start signal; stopping display of the captured image and the shutter button on the information processing device while the information processing device is transferring the captured image to the image display device; A method for transferring and displaying a captured image, comprising:

Images