Information display device

Abstract

This information display device comprises a processor, a rear camera, and a display. When a power source of the information display device is turned on, the processor performs the following: acquiring, from the rear camera, a rear camera image obtained by imaging an external scene in the surroundings of the information display device; generating, on the basis of the rear camera image, a grip assist image for assisting gripping of a remote controller for operating the information display device; displaying the grip assist image on a display; and stopping display of the grip assist image when a predetermined display stop condition is satisfied.

Description

information display device

[0001] The present invention relates to an information display device.

[0002] Patent Document 1 discloses a head-mounted display that "provides a head-mounted display that can turn off image output and the like to ensure visibility when a specific movement motion, such as walking, of the user is detected. The head-mounted display has an image output unit that outputs images and a support unit that supports the image output unit and is worn on the user's head, and is characterized by having movement detection means that detects the movement of the user and control means that controls the image output unit in accordance with the detection result of the movement detection means (abstract excerpt)."

[0003] Japanese Patent Application Laid-Open No. 2004-233903

[0004] When a user tries to hold a remote controller while wearing a head-mounted display device such as a head-mounted display, the field of view is obstructed, making it difficult to hold the remote controller. Also, when a user tries to place the remote controller on a flat surface while wearing the head-mounted display device, the field of view is obstructed, making it difficult to place the remote controller.

[0005] Although Patent Document 1 is an invention related to a monocular information display device, a monocular information display device does not obstruct the user's field of vision while worn, allowing the user to see the outside world. Therefore, a situation where the user cannot see where the controller is placed does not occur, and therefore the above-mentioned problem cannot be found in Patent Document 1, and naturally, it cannot be solved.

[0006] The present invention has been made in consideration of the above-mentioned situation, and aims to provide an information display device that allows the user to easily hold or place a controller while wearing an immersive head-mounted display device.

[0007] In order to solve the above problems, the present invention has the configurations described in the claims.

[0008] According to the present invention, it is possible to provide an information display device that allows the user to easily hold or place a controller while wearing an immersive head-mounted display device. Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0009] 1 is a diagram illustrating a hardware configuration of an information display device according to an embodiment of the present invention. FIG. 1 is a top view showing a state in which a user is wearing the information display device. FIG. 2 is a front view showing a state in which a user is wearing the information display device. FIG. 3 is a side view showing a state in which a user is wearing the information display device. FIG. 4 is a functional block diagram of the information display device. FIG. 5 is a hardware configuration diagram of a remote controller. FIG. 6 is a front view of a remote controller. FIG. 7 is a side view of a remote controller. FIG. 8 is a flowchart showing the flow of power-on process 1. FIG. 9 is a flowchart showing the flow of power-on process 2. FIG. 10 is a flowchart showing the flow of remote controller holding status confirmation process 1. FIG. 11 is a flowchart showing the flow of remote controller holding status confirmation process 2. FIG. 12 is a flowchart showing the flow of remote controller holding status confirmation process 3. FIG. 13 is a diagram showing an example screen display in camera through display 1. FIG. 14 is a diagram showing an example screen display in camera through display 2. FIG. 15 is a diagram showing an example screen display in camera through display 3. FIG. 16 is a diagram showing an example screen display of a home screen (initial VR image). FIG. 17 is a flowchart showing the flow of power-off process 1. FIG. 18 is a flowchart showing the flow of power-off process 2. FIG. 19 is a flowchart showing the flow of control device placement status confirmation process 1. A diagram showing an example of a screen display of a camera-through display 5. A diagram showing an example of a screen display of a camera-through display 6. A flowchart showing the flow of processing 1 when an application is started. A flowchart showing the flow of processing 1 when an application is suspended. A diagram showing an example of a screen display of a camera-through display 7.

[0010] The information display device according to this embodiment is worn on the head and allows the user to easily grip and place the controller even when the user cannot see the outside world, thereby improving the convenience of the information display device. Therefore, the present invention can increase the commercial value of information display devices to which the present invention is applied, and is therefore expected to contribute to Goal 8.2 of the Sustainable Development Goals (SDGs) advocated by the United Nations (increasing economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and in labor-intensive industries).

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and duplicated explanations will be omitted.

[0012] FIG. 1 is a diagram showing the hardware configuration of an information display device 100 according to this embodiment.

[0013] The information display device 100 includes a processor 101 using a CPU, MCU, etc., RAM 103, storage 110, an operation input device 120, an image processing unit 130, an audio processing unit 140, a positioning sensor 150, a motion sensor 160, a communication interface (I / F) 170, and an expansion I / F 180, which are connected to each other by a bus 102.

[0014] The bus 102 is a path for sending and receiving commands and data.

[0015] The storage 110 is configured using a non-volatile storage medium such as a flash ROM, an EEPROM, an SSD, or an HDD.

[0016] The operation input device 120 includes operation keys 121 and a touch sensor 122 .

[0017] The operation keys 121 include, for example, switches, a power key, and a volume key.

[0018] The touch sensor 122 is configured by, for example, a touch pad, and is used to operate the information display device 100, etc. When a separate controller (not shown) is used, the touch sensor 122 may be omitted.

[0019] The image processing unit 130 includes a display 131 , an image signal processing unit 132 , an outer camera 133 , and an inner camera 134 .

[0020] The display 131 may be a transmissive display or a non-transmissive display. In this embodiment, the information display device 100 is a video see-through VR head-mounted display, and therefore a non-transmissive display is used.

[0021] The image signal processing unit 132 is configured using, for example, an image (video) signal processor.

[0022] The outer camera 133 is a camera for capturing images of the surroundings.

[0023] The in-camera 134 is a camera for detecting the line of sight.

[0024] The audio processing unit 140 includes an audio output unit 141 , an audio signal processing unit 142 , and a microphone 143 .

[0025] The audio output unit 141 includes, for example, a speaker, an audio output terminal, and the like.

[0026] The audio signal processing unit 142 is configured using, for example, an audio signal processor.

[0027] The positioning sensor 150 is a sensor that detects the current position of the information display device 100, and may be, for example, a GPS sensor, or a so-called VPS system that extracts feature points in space by analyzing the surrounding image captured by the outer camera 133 and obtains relative position information based on the feature points.

[0028] The motion sensor 160 includes a gyro sensor 161 , a geomagnetic sensor 162 , an acceleration sensor 163 , and a wearing sensor 164 .

[0029] The communication I / F 170 includes, for example, a BT I / F 171 (BT: BlueTooth (registered trademark)) and an NFC I / F 172 .

[0030] The expansion I / F 180 is, for example, a USB device connection terminal, and is used for data transmission / reception, charging, and the like.

[0031] Although not shown, a timer that holds the current time in the real world may be provided. The timer holds a time such as Coordinated Universal Time (UTC) as the current time. The timer may be configured as software or may be configured using an RTC (Real Time Clock).

[0032] 2A is a top view showing a state in which user U1 is wearing information display device 100, Fig. 2B is a front view showing a state in which user U1 is wearing information display device 100, and Fig. 2C is a side view showing a state in which user U1 is wearing information display device 100. In Figs. 2A to 2C, Xh, Yh, and Zh are coordinate axes of a coordinate system based on information display device 100, with Xh indicating the left-right direction of information display device 100, Yh indicating the up-down direction of information display device 100, and Zh indicating the front-to-back direction of information display device 100.

[0033] 2B , in the information display device 100, the left display 131L is disposed inside the housing of the information display device 100 at a position visible to the left eye of the user U1. The right display 131R is disposed inside the housing of the information display device 100 at a position visible to the right eye of the user U1. In this embodiment, the left display 131L and the right display 131R may be collectively referred to as the display 131. As another configuration example, a single display 131 may divide its display area to display an image for the left eye and an image for the right eye side by side.

[0034] Additionally, an upper left outer camera 133LU, an upper right outer camera 133RU, a lower left outer camera 133LD, and a lower right outer camera 133RD are arranged on the outside of the housing of the information display device 100. These four cameras may be collectively referred to as outer cameras 133. Images captured by these four cameras are used to photograph and measure distances to objects around the user U1, i.e., around the information display device 100. Distance measurement is performed using any two of the four outer cameras 133LU / RU / LD / RD in a stereo camera configuration. As an alternative configuration example, the outer camera 133 and a distance measurement sensor may be provided separately.

[0035] 2C, a left touch sensor 122L and a left stereo speaker 141L are provided on the left side of the housing of the information display device 100. Although the right side is not shown in FIG. 2, the right side of the information display device 100 is provided with a right stereo speaker, a monaural microphone, and a charging and wired communication terminal which is one of the expansion I / Fs 180.

[0036] A right touch sensor may be provided on the right side of the information display device 100 instead of the left touch sensor 122L. The touch sensor may be provided on either the left or right side, or on both sides. The information display device 100 of this embodiment has a housing that covers the area around the user U1's eyes to block the outside world, and the display 131 is opaque, making it easier to immerse the user in the virtual world. Furthermore, although this embodiment describes an example of a device in which the display and other components are integrated into the housing as the information display device 100, it may also be a device that is used by inserting a general-purpose smartphone or the like into a head-mounted adapter. In this case, the following processes are executed by internal functions of the general-purpose smartphone or the like.

[0037] FIG. 3 is a functional block diagram of the information display device 100.

[0038] The storage 110 of the information display device 100 includes a basic operation program 1001 including basic programs such as an OS, an application program 1002 for realizing the functions of the information display device 100 of this embodiment, and a various data storage area 1009 which serves as an area for storing various operation setting values ​​and various information (video, still images, audio, etc.).

[0039] The basic operation program 1001 and application program 1002 stored in these storages 110 are expanded (loaded) into RAM 103, and the processor 101 executes the expanded basic operation program 1001 and application program 1002 to configure the functional parts and other functional parts of the information display device of this embodiment.

[0040] The RAM 103 includes a basic operation function unit 1101 , a VR image generation unit 1102 , an image analysis unit 1111 , a control device information confirmation unit 1112 , a display control unit 1113 , and a temporary storage area 1199 .

[0041] The basic operation function unit 1101 controls the basic operations of the information display device 100 .

[0042] The VR image generation unit 1102 generates a VR image according to the operating status of the information display device 100. For example, when no app has been selected by the user U1, the VR image generation unit 1102 generates a home screen or the like under the control of the basic operation function unit 1101. When the user U1 has selected a specific app, the VR image generation unit 1102 generates an app execution screen or the like under the control of an app function unit configured by executing the selected app.

[0043] The image analysis unit 1111 analyzes image information of the periphery of the information display device 100 captured by the outer camera 133, and extracts image patterns of the remote controller 200, the hand of the user U1, etc. The image information of the periphery of the information display device 100 captured by the outer camera 133 is called an outer camera image.

[0044] The control device information confirmation unit 1112 confirms the position and posture of the remote controller 200, the holding status of the remote controller 200 by the user U1, etc. based on the analysis results by the image analysis unit 1111, various sensor data obtained from the motion sensor 160, and control device status information obtained from the remote controller 200.

[0045] The display control unit 1113 controls whether to display the VR image generated by the VR image generation unit 1102 or the out-camera image on the display 131 based on the operating status of the information display device 100, the position and posture of the remote controller 200 confirmed by the control device information confirmation unit 1112, the holding status of the remote controller 200 by the user U1, etc.

[0046] The temporary storage area 1199 is a temporary storage area for various pieces of information created or acquired by various applications.

[0047] 4 is a hardware configuration diagram of the remote controller. Duplicate explanations of components with the same names as those in the information display device 100 will be omitted.

[0048] The remote controller 200 includes a processor 201, a RAM 203, a storage 210, an operation input device 220, a sensor group 260, a communication I / F 270, an expansion I / F 280, and a timer 290 (not shown), which are connected to each other by a bus 202.

[0049] The operation input device 220 may include an operation key 221 .

[0050] The sensor group 260 includes a gyro sensor 261 , a geomagnetic sensor 262 , and an acceleration sensor 263 .

[0051] 5A is a front view of the remote controller, and FIG. 5B is a side view of the remote controller. In FIG. 5A and FIG. 5B, Xc, Yc, and Zc are coordinate axes of a coordinate system based on the remote controller 200, where Xc indicates the left-right direction of the remote controller 200, Yc indicates the up-down direction of the remote controller 200, and Zc indicates the front-back direction of the remote controller 200.

[0052] 5A and 5B , the remote controller 200 includes a housing ring 200S. The image analysis unit 1111 analyzes image information captured by the outer camera 133 of the information display device 100 to recognize the housing ring 200S, thereby enabling the information display device 100 to recognize the position of the remote controller 200 (the relative positional relationship between the information display device 100 and the remote controller 200).

[0053] Furthermore, when the position information and attitude information of the remote controller 200 are transmitted to the information display device 100 through communication between the information display device 100 and the remote controller 200, the housing ring 200S does not need to be image-recognized.

[0054] The remote controller 200 includes a first operation key 221a, a second operation key 221b, and a third operation key 221c. As other examples of the operation keys, an analog stick, a touch pad, or the like may be further included.

[0055] (Power-On Process 1) FIG. 6A is a flowchart showing the flow of power-on process 1.

[0056] The power-on process 1 is a process performed when the user U1 puts the information display device 100 on his / her head, turns the power on, and holds the remote controller 200.

[0057] The basic operation function unit 1101 checks whether the operation input device 120 has received a power-on operation (such as pressing the power key) from the user U1 (S101). If the power-on operation has not been received (S101: No), the process of S101 is repeated, and the device enters a standby state (a state waiting for a power-on operation from the user U1). In the standby state, all units are in a dormant state except for some functions (such as monitoring the power-on operation).

[0058] If a power-on operation is received (S101: Yes), the state transitions from the standby state to the normal operation state, and the basic operation function unit 1101 initializes each unit of the information processing apparatus 100 (S102).

[0059] Next, the basic operation function unit 1101 activates the outer camera 133 (S103) and starts capturing and acquiring image information around the information display device 100. The display control unit 1113 displays the outer camera image on the display 131 (camera-through or video-through display) (S104: see FIGS. 8A to 8C).

[0060] The basic operation function unit 1101 initializes a timer variable T (S105) and starts timing by the information display device 100. The value of the timer variable T corresponds to the elapsed time.

[0061] The basic operation function unit 1101 checks whether the timer variable T is greater than a predetermined value Tth1 (corresponding to a first time) (S106).

[0062] If the timer variable T is greater than the predetermined value Tth1 (S106: Yes), the process returns to S101. As a result, the information display device 100 transitions to a standby state due to a timeout of no operation after the power-on operation. As another example of processing, the information display device 100 may forcibly display the home screen without transitioning to the standby state (the transition indicated by the dashed line in FIG. 6A). In this case, for example, when switching to a gesture input mode without requiring the remote controller 200, forcibly transitioning to the home screen display is convenient.

[0063] On the other hand, if the timer variable T is equal to or less than the predetermined value (S106: No), the basic operation function unit 1101 counts up the timer variable T (S107). Then, the control device information confirmation unit 1112 confirms the state of holding the remote controller 200 by the user U1 (S108: see FIGS. 7A to 7C).

[0064] If it is not confirmed that the user U1 is holding the remote controller 200 (S109: No), the process returns to S106.

[0065] On the other hand, if it is confirmed that user U1 is holding the remote controller 200 (S109: Yes), the display control unit 1113 displays (VR display) the home screen (initial VR image) generated by the VR image generation unit 1102 on the display 131 (S110: see Figure 8D).

[0066] The basic operation function unit 1101 ends the capture and acquisition of image information around the information display device 100 by the outer camera 133 and disables the outer camera 133 (S111). Thereafter, based on the operation of the user U1, the basic operation function unit 1101 performs user login processing and processing based on the execution of various VR applications.

[0067] 6B is a flowchart showing the flow of power-on process 2. Power-on process 2 is a process performed when the action of putting on information display device 100 by user U1 is detected and the power is turned on.

[0068] The basic operation function unit 1101 acquires sensor data from various sensors of the motion sensor 160, such as the gyro sensor 161, the geomagnetic sensor 162, and the acceleration sensor 163 (S121).

[0069] The basic operation function unit 1101 analyzes the sensor data acquired in the process of S121 and checks whether the information display device 100 has been in a stationary state for a predetermined time or longer (S122).

[0070] If the information display device 100 has been stationary for a predetermined time or longer (S122: Yes), it is determined that the information display device 100 is still placed on a desk, floor, or the like, and the process returns to S121.

[0071] If the information display device 100 has not been stationary for a predetermined time or longer (S122: No), there is a possibility that the information display device 100 is being moved (attached) by the user U1. Therefore, the basic operation function unit 1101 activates the in-camera 134 (S123). The in-camera 134 starts capturing and acquiring image information of the inside of the information display device 100. The image information of the inside of the information display device 100 acquired by the in-camera 134 is called an in-camera image.

[0072] The image analysis unit 1111 analyzes the in-camera image acquired in S123 and confirms whether the information display device 100 has been worn by the user U1 (S124). If the image analysis unit 1111 determines that the eyes of the user U1 are captured in a predetermined area within the in-camera image, the basic operation function unit 1101 determines that the user U1 has finished wearing the information display device 100. If the eyes of the user U1 are not captured in the predetermined area within the in-camera image, the basic operation function unit 1101 determines that the user U1 has not yet finished wearing the information display device 100. The predetermined area within the in-camera image is a range within which, when the in-camera 134 captures and generates an in-camera image while the user U1 is wearing the information display device 100 on his or her head, there is a certain or higher probability that the eyes of the user U1 will be captured within the in-camera image.

[0073] If the user U1's eyes are not captured in the image information inside the information display device 100 (S124: No), the process returns to S121. The process loop of S121 to S124 corresponds to a standby state (a state where the device is in standby) for the user U1 to wear the device. In the standby state, all components are in a dormant state except for some functions, such as the acquisition of sensor data from various sensors of the motion sensor 160 and the acquisition of image information by the in-camera 134. However, the in-camera 134 may be in a dormant state during the processes of S121 to S122.

[0074] Instead of the processes of S121 to S124, it may be determined whether or not the user U1 has finished wearing the information display device 100 based on the detection result of the wearing sensor 164.

[0075] On the other hand, if the eyes of the user U1 are captured in the image information inside the information display device 100 (S124: Yes), the process proceeds to S102. The process from S102 onwards is the same as that in Fig. 6A, and therefore a repeated description will be omitted.

[0076] (Remote Controller Holding Status Check Process 1) FIG. 7A is a flowchart showing the flow of remote controller holding status check process 1, and is a diagram illustrating an example of details of S108 in FIGS. 6A and 6B.

[0077] As shown in FIG. 7A, the outer camera 133 captures image information of the periphery of the information display device 100 to acquire an outer camera image (S201), and the image analysis unit 1111 analyzes the outer camera image (S202).

[0078] If the basic operation function unit 1101 determines, based on the analysis results of the image analysis unit 1111, that the outer camera image contains both an image pattern that matches the appearance of the user U1's hand and an image pattern that matches the appearance of the remote controller 200, it determines that the user U1's hand is holding the remote controller 200 (S203: Yes), and the control device information confirmation unit 1112 sets the control device holding flag, which indicates that the user U1 is holding the remote controller 200, to "1" (S204), and proceeds to S109.

[0079] On the other hand, if the basic operation function unit 1101 determines, based on the analysis results of the image analysis unit 1111, that the outer camera image does not contain both an image pattern that matches the appearance of the user U1's hand and an image pattern that matches the appearance of the remote controller 200, it determines that the user U1's hand is not holding the remote controller 200 (S203: No), and the control device information confirmation unit 1112 sets the control device holding flag, which indicates that the user U1 is holding the remote controller 200, to "0" (S205), and proceeds to S109.

[0080] According to this example, it is possible to determine whether the user U1 is holding the remote controller 200 using the analysis result of the outer camera image.

[0081] (Remote Controller Holding Status Check Process 2) FIG. 7B is a flowchart showing the flow of remote controller holding status check process 2, and is a diagram illustrating another example of the details of S108 in FIGS. 6A and 6B.

[0082] When the basic operation function unit 1101 establishes a communication link between the information display device 100 and the remote controller 200 using BLUETOOTH (registered trademark) or the like (S221), the control device information confirmation unit 1112 sends a request to send control device status information to the remote controller 200 via the BT I / F 171 (S222).

[0083] When the remote controller 200 receives a request to send control device status information from the information display device 100 via the BT I / F 271, it acquires sensor data from various sensors of the sensor group 260 (gyro sensor 261, geomagnetic sensor 262, acceleration sensor 263, etc.) (S223), and the remote controller 200 transmits the sensor data acquired in the processing of S223 to the information display device 100 via the BT I / F 271 (S224).

[0084] As another example, the remote controller 200 may determine whether the remote controller 200 is in a stationary state based on the sensor data acquired in the processing of S223, and transmit information indicating the determination result to the information display device 100.

[0085] The control device information confirmation unit 1112 analyzes the sensor data received from the remote controller 200 via the BT I / F 171 and confirms whether the remote controller 200 has been stationary for a predetermined period of time or longer (S225). If it is determined that the remote controller 200 has not been stationary for a predetermined period of time or longer (S225: No), it is determined that the remote controller 200 is being moved (held) by the user U1. Therefore, the control device information confirmation unit 1112 sets a control device holding flag, which indicates that the user U1 is holding the remote controller 200, to "1" (S226), and proceeds to S109.

[0086] On the other hand, if it is determined that the remote controller 200 has been stationary for a predetermined period of time or more, or if sensor data cannot be received from the remote controller 200 (S225: Yes), it can be determined that the remote controller 200 is still placed on a desk, floor, etc. Therefore, the control device information confirmation unit 1112 sets the control device holding flag, which indicates that it has been determined that the user U1 is not holding the remote controller 200, to "0" (S227), and proceeds to S109.

[0087] According to this example, the holding state of the remote controller 200 is confirmed using sensor information that detects the movement of the remote controller 200 itself, so the holding state of the remote controller 200 can be confirmed regardless of the direction in which the outer camera 133 mounted on the information display device 100 is facing.

[0088] (Remote Controller Holding Status Check Process 3) FIG. 7C is a flowchart showing the flow of remote controller holding status check process 3, and is a diagram illustrating another example of the details of S108 in FIGS. 6A and 6B.

[0089] When the basic operation function unit 1101 establishes a communication link between the information display device 100 and the remote controller 200 using BLUETOOTH (registered trademark) or the like (S241), the remote controller 200 checks whether any operation key 221 of the operation input device 220 has been pressed (S242).

[0090] If any operation key 221 is pressed (S242: Yes), a signal indicating that the operation key 221 has been pressed is transmitted to the information display device 100 via the BT I / F 271. If any operation key 221 has not been pressed (S242: No), no processing is performed.

[0091] When the control device information confirmation unit 1112 determines that it has received a signal from the remote controller 200 via the BT I / F 171 indicating that any operation key 221 has been pressed (S243: Yes), it can determine that the any operation key 221 was pressed after the remote controller 200 was held by the user U1. Therefore, the control device information confirmation unit 1112 sets a control device holding flag indicating that the user U1 has held the remote controller 200 to "1" (S244), and proceeds to S109.

[0092] On the other hand, if the control device information confirmation unit 1112 determines that it has not received a signal from the remote controller 200 indicating that any operation key 221 has been pressed (S243: No), it can determine that the remote controller 200 is not being held by the user U1. Therefore, the control device information confirmation unit 1112 sets the control device holding flag, which indicates that the user U1 is not holding the remote controller 200, to "0" (S245), and proceeds to S109.

[0093] (Camera Through Display 1) FIG. 8A is a diagram showing an example of a screen display in Camera Through Display 1. As shown in FIG.

[0094] The processor 101 of the information display device 100 displays a camera through image 131E captured by the outer camera 133 on the entire surface of the display 131. The camera through image 131E is an example of a grip assist image displayed in the processing of S104.

[0095] The camera through image 131E includes a table image 131E1, an image 131E2M of the left-handed remote controller 200L, and an image 131E3 of the right-handed remote controller 200R. By displaying the camera through image 131E, an external image around the information display device 100 is displayed as a camera through image, making it easier to confirm the position of the remote controller 200.

[0096] (Camera Through Display 2) FIG. 8B is a diagram showing an example of a screen display of Camera Through Display 2. As shown in FIG.

[0097] The processor 101 of the information display device 100 displays the home screen (initial VR image) generated by the VR image generation unit 1102 on the display 131. Furthermore, a partial image is generated by extracting only the area around the remote controller 200 from the image acquired by the outer camera 133, and camera through images 131F2 and 131F3 made up of the partial images are superimposed on the home screen (initial VR image) to display a composite through image 131F. The composite through image 131F is another example of the grip assist image displayed in the processing of S104.

[0098] Camera through images 131F2 and 131F3 are partial camera through images obtained by cutting out the periphery of left-handed remote controller 200L and the periphery of right-handed remote controller 200R from camera through image 131E. In this example, although only the periphery of remote controller 200 is shown, the camera through display makes it easier to confirm the position of remote controller 200, and the initial VR image continues to be displayed, allowing the viewer to enjoy the lingering sensation of the VR space.

[0099] (Camera Through Display 3) FIG. 8C is a diagram showing an example of a screen display of Camera Through Display 3. As shown in FIG.

[0100] The display control unit 1113 displays the home screen (initial VR image) generated by the VR image generation unit 1102 on the display 131. Furthermore, the display control unit 1113 generates objects 131G1 and 131G2 that emphasize the outline of the remote controller 200 and displays a composite through image 131G that is superimposed on the home screen (initial VR image). The composite through image 131G is another example of the grip assist image displayed in the processing of S104.

[0101] Objects 131G2 and 131G3 are contour-enhanced objects obtained by extracting the contours of left-handed remote controller 200L and right-handed remote controller 200R from camera through image 131E. The contour-enhanced object of remote controller 200 is displayed superimposed on the home screen (initial VR image), making it easier to confirm the position of remote controller 200. When the hand pattern of user U1 is detected, a contour-enhanced object obtained by extracting the contours of user U1's hands may be generated and displayed.

[0102] (Home Screen (Initial VR Image) Display) FIG. 8D is a diagram showing an example of a home screen (initial VR image) display.

[0103] The display control unit 1113 displays the home screen (initial VR image) 131H generated by the VR image generation unit 1102 on the display 131. The home screen 131H is displayed in the processing of S110.

[0104] The home screen (initial VR image) 131H displays a main menu 131H1 and a current date and time 131H2 superimposed on a scene from the virtual world. After the remote controller 200 is picked up, the camera-through display is terminated, allowing the user to become more immersed in the virtual world.

[0105] 9A is a flowchart showing the flow of power-off process 1. Power-off process 1 is a process performed when user U1 turns off the power of information display device 100, places remote controller 200 on a desk or the like, and then removes information display device 100. In power-off process 1, camera-through or video-through display ends after a predetermined time has elapsed, regardless of whether user U1 has placed remote controller 200 on a desk or the like.

[0106] If the operation input device 120 has not received a power-off operation from the user U1 (such as pressing the power key or selecting the power-off command on the home screen) (S301: No), the basic operation function unit 1101 repeats the process of S301. Repeating the process of S301 indicates that the information display device 100 is in a normal operating state and is capable of receiving various commands.

[0107] In S301, the remote controller 200 may accept a power-off operation from the user U1 instead of the information display device 100 accepting the power-off command. In this case, the remote controller 200 may transmit a signal indicating this to the information display device 100, and the information display device 100 may receive from the remote controller 200 a signal indicating that the power-off command from the user U1 has been accepted, thereby substituting for the processing of S301.

[0108] On the other hand, when the basic operation function unit 1101 determines that the operation input device 120 has received a power-off operation (such as pressing the power key) from the user U1 (S301: Yes), it activates the outer camera 133 (S302).

[0109] When the outer camera 133 captures an image of the periphery of the information display device 100 and generates an outer camera image, the display control unit 1113 displays the outer camera image on the display 131 (camera through or video through display) (S303: see FIGS. 11A to 11C).

[0110] The basic operation function unit 1101 initializes a timer variable T (S304).

[0111] The basic operation function unit 1101 checks whether the timer variable T is greater than a predetermined value Tth2 (corresponding to a second time) (S305). The predetermined value Tth2 is set to a time (e.g., 30 seconds) sufficient for the user U1 to place the remote controller 200 on a desk or the like.

[0112] If the timer variable T is equal to or smaller than the predetermined value Tth2 (S305: No), the basic operation function unit 1101 counts up the timer variable T (S306) and returns to S305.

[0113] On the other hand, if the timer variable T is greater than the predetermined value Tth2 (S305: Yes), the process proceeds to S307. In this case, since the camera-through or video-through display has continued for the predetermined time, the camera-through or video-through display is ended (S307) and the outer camera is stopped (S308). Then, the basic operation function unit 1101 performs a process to terminate the information display device 100 (S309). After the process of S309, the information display device 100 enters a standby state.

[0114] According to this example, when the user U1 stops holding the remote controller 200 after the power-off operation and places it on a table or the like, it becomes easier for the user U1 to confirm where to place the remote controller 200.

[0115] Instead of receiving the power-off operation in S301, the above processing may be performed based on a decrease in the remaining battery power. In this case, the basic operation function unit 1101 may manage the remaining battery power, and when it detects that the remaining battery power has fallen below a predetermined value, the processing from S302 onward may be performed.

[0116] By applying the value (required remaining capacity value) required to capture and display the outer camera image as the predetermined value for the remaining battery capacity, the information display device 100 can be stopped after the user checks where the remote controller 200 is located.

[0117] 9B is a flowchart showing the flow of power-off process 2. Power-off process 2 is a process performed when user U1 turns off the power of information display device 100, places remote controller 200 on a desk or the like, and then removes information display device 100. After confirming that remote controller 200 has been placed on a desk or the like, camera-through or video-through display is terminated.

[0118] Steps S301 to S306 and steps S307 to S309 in FIG. 9B are the same as those in FIG. 9A, and therefore redundant explanations will be omitted.

[0119] After the basic operation function unit 1101 counts up the timer variable (S306), the control device information confirmation unit 1112 confirms the placement status of the remote controller 200 by the user U1 (S321: see FIGS. 10A and 10B).

[0120] If it is not confirmed that the user U1 has placed the remote controller 200 on a desk or the like (i.e., the user U1 is still holding the remote controller 200) (S322: Yes), the process returns to S305.

[0121] If it is confirmed that user U1 has placed the remote controller 200 on a desk or the like (i.e., user U1 is not holding the remote controller 200) (S322: No), or if the timer variable T is greater than the predetermined value Tth2 (S305: Yes), proceed to S307.

[0122] According to this example, the camera-through or video-through display can be terminated after continuing the camera-through or video-through display for a predetermined time, or if it is confirmed that the user U1 is not holding the remote controller 200 (i.e., the user U1 has placed the remote controller 200 on a desk, etc.) even within the predetermined time.

[0123] As a modification of the above example, the camera-through or video-through display may be continued until it is confirmed by the mounting sensor 164 or the like that the user U1 has removed the information display device 100 from his / her head.

[0124] (Controller Placement Status Confirmation Process 1) FIG. 10A is a flowchart showing the flow of controller placement status confirmation process 1, and is a diagram illustrating an example of details of S321 in FIG. 9B.

[0125] The outer camera 133 captures an image of the periphery of the information display device 100 and acquires an outer camera image (S401).

[0126] The image analysis unit 1111 analyzes the outer camera image captured in the processing of S401 (S402), and confirms whether the outer camera image captured in the processing of S401 contains both an image pattern that matches the appearance of the hand of the user U1 and an image pattern that matches the appearance of the remote controller 200, and further whether the hand of the user U1 is holding the remote controller 200 (S403).

[0127] If it cannot be confirmed that user U1 is holding the remote controller 200 in his / her hands (S403: No), it can be determined that user U1 has placed the remote controller 200 on a desk or the like, and the control device information confirmation unit 1112 sets the control device placement flag, which indicates that user U1 is not holding the remote controller 200 (i.e., user U1 has placed the remote controller 200 on a desk or the like), to "1" (S404).

[0128] In S403, if there is no pattern in the outer camera image that matches the appearance of the user U1's hand or the appearance of the remote controller 200, it is determined that the user U1's hand is holding the remote controller 200 in the blind spot of the outer camera 133.

[0129] Furthermore, if the remote controller 200 is provided with a strap or the like, even if the remote controller 200 is hanging from the wrist by the strap, if the remote controller 200 body is away from the hand of the user U1, it is determined that the hand of the user U1 is not holding the remote controller 200.

[0130] If it is confirmed that user U1 is holding the remote controller 200 in his / her hands (S403: Yes), it can be determined that user U1 has not yet placed the remote controller 200 on a desk or the like, and the control device information confirmation unit 1112 sets the control device placement flag to "0" (S405).

[0131] According to this example, it is possible to determine whether the user U1 is holding the remote controller 200 from the analysis results of the outer camera image, and to assist the user in placing the remote controller 200 on a desk or the like when they have finished using the information display device 100.

[0132] (Controller Placement Status Confirmation Process 2) FIG. 10B is a flowchart showing the flow of controller placement status confirmation process 2, and is a diagram illustrating another example of the details of S321 in FIG. 9B.

[0133] When the basic operation function unit 1101 establishes a communication link between the information display device 100 and the remote controller 200 using BLUETOOTH (registered trademark) or the like (S421), the control device information confirmation unit 1112 sends a request to send control device status information to the remote controller 200 via the BT I / F 171 (S422).

[0134] When the remote controller 200 receives a request to send control device status information from the information display device 100 via the BT I / F 271, it acquires sensor data from various sensors in the sensor group 260 (gyro sensor 261, geomagnetic sensor 262, acceleration sensor 263, etc.) (S423).

[0135] The remote controller 200 transmits the sensor data acquired in the processing of S423 to the information display device 100 via the BT I / F 271 (S424). As another example, information as to whether the remote controller 200 is in a stationary state, which is determined based on the sensor data acquired in the processing of S423, may be transmitted to the information display device 100.

[0136] The control device information confirmation unit 1112 analyzes the sensor data received from the remote controller 200 via the BT I / F 171, and confirms whether the remote controller 200 has been in a stationary state for a predetermined period of time or more (S425).

[0137] If the control device information confirmation unit 1112 determines that the remote controller 200 has been stationary for a predetermined period of time or longer (S425: Yes), the control device information confirmation unit 1112 sets the control device placement flag, which indicates that it can be determined that the user U1 has placed the remote controller 200 on a desk or the like, to "1" (S426).

[0138] On the other hand, if the control device information confirmation unit 1112 determines that the remote controller 200 has not been stationary for a predetermined time or longer (S425: No), the control device information confirmation unit 1112 determines that the remote controller 200 has not yet been placed on a desk or the like by the user U1, and sets the control device placement flag to "0" (S427). If sensor data cannot be received from the remote controller 200 in the processing of S423, the determination is also made as No in S425.

[0139] If the remote controller 200 can be charged on a dedicated or general-purpose charging stand, the processing of S424 may be configured to transmit a message to the information display device 100 indicating that the remote controller 200 has been placed on the charging stand (or that charging has started). In this case, the information display device 100 may determine that the remote controller 200 has been placed on a desk or the like when receiving a message indicating that the remote controller 200 has been placed on the charging stand (or that charging has started).

[0140] (Camera Through Display 4) FIG. 11A is a diagram showing an example of a screen display of Camera Through Display 4. As shown in FIG.

[0141] The display control unit 1113 displays a camera through image 131P on the display 131. The camera through image 131P includes a table image 131P1. The camera through image 131P is an example of a placement assist image displayed in the process of S303.

[0142] Displaying the camera through image 131P makes it easier for the user U1 to confirm the position of a flat surface (such as a desk) on which to place the remote controller 200.

[0143] (Camera Through Display 5) FIG. 11B is a diagram showing an example of a screen display of Camera Through Display 5. As shown in FIG.

[0144] The display control unit 1113 displays a camera through image 131P on the display 131. The camera through image 131P includes a table image 131P1 and a warning message 131P2. The camera through image 131P is another example of a placement assist image displayed in the process of S303.

[0145] The warning message 131P2 is a message to notify that the power-off operation has been accepted and that the camera-through or video-through display is temporarily being performed to safely place the control device. The example message in Fig. 10B is merely an example, and it may be, for example, "The power will be turned off after the control device is placed."

[0146] Displaying the camera through image 131P makes it easier for the user U1 to confirm the position of a flat surface (such as a desk) on which to place the remote controller 200.

[0147] (Camera Through Display 6) FIG. 11C is a diagram showing an example of a screen display of Camera Through Display 6. As shown in FIG.

[0148] The display control unit 1113 displays the home screen (initial VR image) 131Q generated by the VR image generation unit 1102 on the display 131, and superimposes an object 131Q1 representing a flat surface in front of the user U1 on which the remote controller 200 can be placed on the home screen 131Q. The object 131Q1 represents the flat surface of the desk in front of the user U1. The object 131Q1 may also be an object indicating the position of a charging stand. The image in which the object 131Q1 is superimposed on the home screen (initial VR image) 131Q corresponds to a placement assist image.

[0149] According to this example, an object indicating the position of the flat surface or charging stand in front of the user U1 is displayed, making it easy to place the remote controller 200.

[0150] 12A is a flowchart showing the flow of application startup process 1. In this example, when user U1 selects and starts an arbitrary application, if the started application requires the remote controller 200, an image from the outer camera is displayed.

[0151] The basic operation function unit 1101 checks whether the operation input device 120 has received an operation to select or start an application from the user U1 (S501). If the operation to select or start an application has not been received (S501: No), the process of S501 is repeated.

[0152] On the other hand, if the basic operation function unit 1101 receives an operation to select or launch an app (S501: Yes), the basic operation function unit 1101 reads information about the arbitrary app selected by user U1 in the processing of S501 (S502), and checks whether the arbitrary app selected by user U1 in the processing of S501 requires the remote controller 200 (S503).

[0153] If the application does not require the remote controller 200 (S503: No), the process proceeds to S512.

[0154] On the other hand, if the application requires the remote controller 200 (S503: Yes), the basic operation function unit 1101 activates the outer camera 133 (S504) and starts capturing outer camera images of the area around the information display device 100 using the outer camera 133.

[0155] The display control unit 1113 displays the out-camera image on the display 131 (camera through or video through display) (S505: see FIGS. 8A to 8C).

[0156] The basic operation function unit 1101 initializes a timer variable T (S506).

[0157] The basic operation function unit 1101 checks whether the timer variable T is greater than the predetermined value Tth1 (S507). If the timer variable T is greater than the predetermined value Tth1 (S507: Yes), the process proceeds to S511.

[0158] If the timer variable T is equal to or smaller than the predetermined value Tth1 (S507: No), the basic operation function unit 1101 counts up the timer variable T (S508).

[0159] The control device information confirmation unit 1112 checks the state of the user U1 holding the remote controller 200 (S509: see FIGS. 7A to 7C ). If it is confirmed that the user U1 is holding the remote controller 200 (S510: Yes), the basic operation function unit 1101 ends the capture and acquisition of an out-camera image of the area around the information display device 100 by the out-camera 133 and stops the out-camera 133 (S511). The basic operation function unit 1101 then performs a launch process for an arbitrary app selected by the user U1 in the process of S501 (S512). Thereafter, an image of the arbitrary app launched in the process of S512 is displayed on the display 131. Information about the arbitrary app selected by the user U1 in the process of S502 may be read, and the process for launching the arbitrary app may be performed, followed by the processes of S503 to S511.

[0160] On the other hand, if it is not confirmed that the user U1 is holding the remote controller 200 (S510: No), the process returns to S507.

[0161] According to this example, when an application that requires the use of the remote controller 200 is selected or started, an image from the outer camera is displayed before the application is started, making it easier to hold the remote controller 200.

[0162] (Application Suspension Process 1) FIG. 12B is a flowchart showing the flow of application suspension process 1.

[0163] The basic operation function unit 1101 checks whether the operation input device 120 has received an operation to suspend the application from the user U1 (S601). If the operation to suspend the application has not been received (S601: No), the process returns to S601, and the execution of the application continues.

[0164] When the basic operation function unit 1101 receives an operation to suspend an application (S601: Yes), the basic operation function unit 1101 activates the outer camera 133 (S602) and starts capturing and acquiring outer camera images of the periphery of the information display device 100 by the outer camera 133.

[0165] The application control unit (not shown in FIG. 3) temporarily stops the execution of the application, and the display control unit 1113 displays the image from the outer camera on the display 131 (camera through or video through display) (S603: see FIG. 13).

[0166] The basic operation function unit 1101 checks whether the operation input device 120 has received an operation to unpause the application from the user U1 (S604). If the operation to unpause the application has not been received (S604: No), the process of S604 is repeated to continue the paused state of the application and the camera-through display.

[0167] When an operation to unpause the app is received (S604: Yes), the app control unit (not shown in FIG. 3) unpauses the app and resumes execution, the display control unit 1113 displays the app's execution screen on the display 131 (S605), and the basic operation function unit 1101 stops the outer camera 133 (S606).

[0168] According to this example, since it is likely that the user will need to let go of the remote controller 200 when pausing the app, the outer camera image can be displayed during the pause to make it easier to confirm where to place the remote controller 200.

[0169] (Camera Through Display 7) FIG. 13 is a diagram showing an example of a screen display of Camera Through Display 7. As shown in FIG.

[0170] The display control unit 1113 displays a camera-through image 131S on the display 131. The camera-through image 131S is an outer camera image that is displayed when the user U1 pauses an application (application XX) that requires the remote controller 200 while the application XX is being executed. The display control unit 1113 controls the process of switching between the application XX execution image and the camera-through image.

[0171] The camera-through image 131S includes a table image 131S1 and a warning message 131S2. The warning message 131S2 is a message for informing the user that the operation of the application XX has been paused by the user U1. This message is not limited to the example shown in FIG. 13 , and may further include, for example, a guide on how to cancel the paused state.

[0172] According to this example, image information of the periphery of the information display device 100 is displayed as a camera-through image, so that it is easy to confirm the position of a surface (such as a desk) on which to temporarily place the remote controller 200 during a pause. Note that if the running app does not require the remote controller 200, the display switching process to a camera-through image does not need to be performed.

[0173] The present invention is not limited to the above-described embodiments, and it is possible to replace part of the configuration of one embodiment with another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. These all fall within the scope of the present invention, and the numerical values, messages, etc. appearing in the text and figures are merely examples, and the use of different ones does not impair the effects of the present invention.

[0174] Furthermore, some or all of the functions of the invention may be implemented in hardware, for example, by designing an integrated circuit. They may also be implemented in software by a microprocessor unit, processor, etc., interpreting and executing an operating program. Furthermore, the scope of software implementation is not limited, and both hardware and software may be used.

[0175] The embodiments include the following: (Supplementary Note 1) An information display device comprising: a processor; an outer camera; and a display, wherein when the information display device is powered on, the processor acquires an outer camera image of an external scene around the information display device from the outer camera, generates a grip assist image for assisting in gripping a remote controller for operating the information display device based on the outer camera image, and displays the image on the display, and when a predetermined display stop condition is satisfied, stops displaying the grip assist image.

[0176] (Supplementary Note 2) An information display device comprising: a processor; an outer camera; a wearing sensor that detects whether the information display device is worn by a user; and a display, wherein the processor, when determining that the user is wearing the information display device based on sensor information from the wearing sensor, acquires an outer camera image capturing an external scene around the information display device from the outer camera, generates a grip assist image that assists in gripping a remote controller that operates the information display device based on the outer camera image, and displays the image on the display, and when a predetermined display stop condition is met, stops displaying the grip assist image.

[0177] (Supplementary Note 3) An information display device comprising: a processor; an outer camera; and a display, wherein the processor, upon receiving an operation to launch an application program to be executed by the processor, determines whether a remote controller for operating the information display device is needed to execute the application program, and if it determines that a remote controller is needed, acquires an outer camera image capturing an external scene around the information display device from the outer camera, generates a grip assist image for assisting in gripping the remote controller for operating the information display device based on the outer camera image, and displays the image on the display, and stops displaying the grip assist image when a predetermined display stop condition is met.

[0178] (Supplementary Note 4) An information display device comprising: a processor; an outer camera; and a display, wherein the processor, upon receiving an operation to turn off the power of the information display device, acquires an outer camera image of an external scene around the information display device from the outer camera, generates a placement assist image based on the outer camera image to help the user know where to place a remote controller that operates the information display device, and displays the placement assist image on the display, and, upon satisfying a predetermined display stop condition, stops displaying the placement assist image.

[0179] (Supplementary Note 5) An information display device comprising: a processor; an outer camera; and a display, wherein the processor, upon receiving an operation to pause an application program being executed by the processor, acquires an outer camera image of an external scene around the information display device from the outer camera, generates a placement assist image based on the outer camera image to assist in placing a remote controller that operates the information display device, and displays the image on the display, and, upon receiving an operation to cancel the pause of the application program, stops displaying the placement assist image and resumes displaying an execution screen of the application program.

[0180] 100: Information display device 101: Processor 102: Bus 103: RAM 110: Storage 120: Operation input device 121: Operation keys 122: Touch sensor 122L: Left touch sensor 130: Image processing unit 131: Display 131E: Camera through image 131E1: Table image 131E2: Image of left-handed remote controller 200L 131E3: Image of right-handed remote controller 200R 131F: Composite through image 131F2: Camera through image 131F3: Camera through image 131G: Composite through image 131G2: Object 131G3: Object 131H: Home screen 131H1: Main menu 131H2: Current date and time 131L: Left display 131P : Camera through image 131P1 : Table image 131P2 : Warning message 131Q : Synthesized through image 131Q1 : Object 131R : Right display 131S : Camera through image 131S1 : Table image 131S2 : Warning message 132 : Image signal processing unit 133 : Outer camera 133LD : Lower left outer camera 133LU : Upper left outer camera 133RD : Lower right outer camera 133RU : Upper right outer camera 134 : Inner camera 140 : Audio processing unit 141 : Audio output unit 141L : Left stereo speaker 142 : Audio signal processing unit 143 : Microphone 150 : Positioning sensor 160 : Motion sensor 161 : Gyro sensor 162 : Geomagnetic sensor 163 : Acceleration sensor 164: Wearing sensor 170: Communication I / F 171: BT I / F 172: NFC I / F 180: Expansion I / F 200: Remote controller 200L: Left-hand remote controller 200R: Right-hand remote controller 200S: Housing ring 201: Processor 202: Bus 203: RAM 210: Storage220: Operation input device 221: Operation key 221a: First operation key 221b: Second operation key 221c: Third operation key 260: Sensor group 261: Gyro sensor 262: Geomagnetic sensor 263: Acceleration sensor 270: Communication I / F 280: Expansion I / F 290: Timer 1001: Basic operation program 1002: Application program 1009: Data storage area 1101: Basic operation function unit 1102: VR image generation unit 1111: Image analysis unit 1112: Control device information confirmation unit 1113: Display control unit 1199: Temporary storage area

Claims

1. An information display device, comprising: a processor, an out-camera, and a display; wherein when the power of the information display device is turned on, the processor acquires an out-camera image obtained by imaging an external scenery around the information display device from the out-camera, generates a gripping assist image for assisting in gripping a remote controller for operating the information display device based on the out-camera image, and displays the gripping assist image on the display, and when a predetermined display stop condition is satisfied, stops displaying the gripping assist image.

2. An information display device, comprising: a processor, an out-camera, a wearing sensor for detecting whether the information display device is worn by a user, and a display; wherein when the processor determines that the user has worn the information display device based on sensor information from the wearing sensor, the processor acquires an out-camera image obtained by imaging an external scenery around the information display device from the out-camera, generates a gripping assist image for assisting in gripping a remote controller for operating the information display device based on the out-camera image, and displays the gripping assist image on the display, and when a predetermined display stop condition is satisfied, stops displaying the gripping assist image.

3. An information display device, comprising: a processor, an out-camera, and a display; wherein when the processor receives an activation operation of an application program to be executed by the processor, the processor determines whether a remote controller for operating the information display device is required to execute the application program, and when it is determined that it is required, the processor acquires an out-camera image obtained by imaging an external scenery around the information display device from the out-camera, generates a gripping assist image for assisting in gripping a remote controller for operating the information display device based on the out-camera image, and displays the gripping assist image on the display, and when a predetermined display stop condition is satisfied, stops displaying the gripping assist image.

4. The information display device according to claim 1, 2, or 3, wherein the display stop condition is that the processor analyzes the out-camera image, and based on the result of the image analysis, determines that the user of the information display device has grasped the remote controller. Information display device.

5. The information display device according to claim 1, 2, or 3, wherein the information display device further includes a communication interface with the remote controller, and the display stop condition is that the processor determines, based on the information received from the remote controller, that the remote controller is not in a stationary state. Information display device.

6. The information display device according to claim 1, 2, or 3, wherein the information display device further includes a communication interface with the remote controller, and the display stop condition is that the processor determines that it has received from the remote controller a signal indicating that an operation key provided on the remote controller has been pressed. Information display device.

7. The information display device according to claim 1, 2, or 3, wherein the display stop condition is that the processor determines that the elapsed time since the start of the display of the grasping assist image has exceeded a predetermined first time. Information display device.

8. The information display device according to claim 1, 2, or 3, wherein the processor displays the out-camera image on the display as the grasping assist image. Information display device.

9. The information display device according to claim 1, 2, or 3, wherein the processor performs image analysis on the out-camera image, extracts a partial image of the remote controller captured in the out-camera image, and superimposes and displays the extracted partial image on the screen being displayed on the display as the grasping assist image. Information display device.

10. The information display device according to claim 1, 2, or 3, wherein the processor performs image analysis on the out-camera image, generates a contour enhancement object in which the contour of the remote controller captured in the out-camera image is enhanced, and superimposes and displays the contour enhancement object on the screen being displayed on the display as the grasping assist image. Information display device.

11. The information display device according to claim 2, further comprising an in-camera, wherein the mounting sensor is a sensor that analyzes an image of the in-camera and detects that the user's eyes are captured in the image.

12. An information display device, comprising a processor, an out-camera, and a display, wherein when the processor receives an operation to turn off the power of the information display device, the processor acquires an out-camera image obtained by imaging an external scenery around the information display device from the out-camera, generates a placement assist image for grasping a place where a remote controller for operating the information display device is placed based on the out-camera image, and displays the placement assist image on the display, and when a predetermined display stop condition is satisfied, stops displaying the placement assist image.

13. The information display device according to claim 12, wherein the display stop condition is that the processor determines that an elapsed time since the start of displaying the placement assist image exceeds a predetermined second time.

14. The information display device according to claim 12, wherein the display stop condition is that the processor analyzes the out-camera image and determines, based on a result of the image analysis, that a user of the information display device has placed the remote controller.

15. The information display device according to claim 12, wherein the information display device further comprises a communication interface with the remote controller, and the display stop condition is that the processor determines, based on information received from the remote controller, that the remote controller has become stationary.

16. An information display device, comprising: a processor, an out-camera, and a display; wherein when the processor receives a pause operation of an application program being executed by the processor, the processor acquires an out-camera image obtained by imaging an external scenery around the information display device from the out-camera, generates a placement assist image for assisting in placing a remote controller for operating the information display device based on the out-camera image, and causes the display to display the placement assist image; and when the processor receives a resume operation of the application program, the processor stops displaying the placement assist image and resumes displaying the execution screen of the application program.

17. The information display device according to claim 12 or 16, wherein the processor causes the display to display the out-camera image as the placement assist image.

18. The information display device according to claim 12, wherein the processor further superimposes and displays a warning message notifying a time until the power is turned off as the placement assist image on the screen being displayed on the display.

19. The information display device according to claim 12 or 16, wherein the processor performs image analysis on the out-camera image, extracts a flat portion in the external scenery imaged by the remote controller, and superimposes and displays the extracted flat portion on the screen being displayed on the display as the placement assist image.

Images