Information display device, information display system, and display control method for information display device

Abstract

This information display device comprises a processor, a display, a mounting sensor, and a rear camera. The processor: counts up a cumulative use time of the information display device when a power source of the information display device is turned on and the mounting sensor detects that the information display device has been mounted to a user; displays, on the display, an external world image of the real world imaged by the rear camera when the cumulative use time increases to a predetermined cumulative use limit threshold value; stops screen display of the display and counts down the cumulative use time when the mounting sensor detects that the information display device has been removed from the user; and permits the screen display of the display when the cumulative use time decreases to a predetermined restart permission threshold value.

Description

Information display device, information display system, and display control method for information display device

[0001] The present invention relates to an information display device, an information display system, and a display control method for an information display device.

[0002] Conventionally, various efforts have been made to prevent eye strain caused by viewing images using an information display device.

[0003] As one example, Patent Document 1 discloses a head-mounted display including: "a display unit that projects image light onto the eyes of an observer to display an image that corresponds to the image light; a fatigue detection unit that detects the degree of eye fatigue of the observer while the display unit is displaying an image; and a control unit that reduces the display area of ​​the image displayed by the display unit based on the detection result by the fatigue detection unit, and assigns the remaining area in the display area of ​​the display unit excluding the displayed image as a display area reduction area that urges the observer to take a break, thereby displaying a warning message (abstract excerpt)."

[0004] Furthermore, Patent Document 2 states that "The viewing determination unit of the video display device determines whether or not the observer is viewing the video provided by the display optical system based on the detection result of the line-of-sight detection sensor. When the viewing determination unit determines that the observer is viewing the video, the evaluation value calculation unit measures the time the observer has viewed the video and calculates an evaluation value according to the length of the viewing time, and accumulates the evaluation value each time the video is viewed. When the accumulated value of the evaluation values ​​obtained by the evaluation value calculation unit exceeds a threshold value, the warning control unit causes the warning unit to warn the observer (summary excerpt)."

[0005] JP 2012-212990 A JP 2018-101034 A

[0006] VR-HMDs (VR: Virtual Reality, HMD: Head Mounted Display) used for games and other applications are goggle-shaped devices that cover the area around the wearer's (user's) eyes. This allows the user wearing the VR-HMD to block their view from the outside world and display virtual reality images on a display in front of their eyes, making it easier for them to immerse themselves in the virtual world.

[0007] However, once users are immersed in a virtual world, it becomes difficult for them to check the outside world, and they tend to become so absorbed in operating the device that they lose track of time. There are concerns about excessive use of VR-HMDs, especially among minors who have low self-control, and there is a social situation in which there is a debate about the need to restrict the use of VR-HMDs by minors.

[0008] Here, it is conceivable to apply Patent Documents 1 and 2 to VR-HMDs, but since both Patent Documents 1 and 2 relate to open-type glasses-type HMDs, there is a need for usage restrictions that are appropriate for VR-HMDs, which do not allow the user to see the outside world.

[0009] The present invention has been made in view of the above circumstances, and aims to provide a technique that can appropriately restrict use of an immersive head-mounted information display device.

[0010] In order to solve the above problems, the present invention has the configurations described in the claims.

[0011] According to the present invention, it is possible to provide a technology that can appropriately restrict the use of an immersive head-mounted information display device. Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0012] 1 is a hardware configuration diagram of an information display device according to the present embodiment. A top view showing a state in which a user is wearing the information display device. A front view showing a state in which a user is wearing the information display device. A left side view showing a state in which a user is wearing the information display device. A functional block diagram of the information display device. A flowchart showing a processing flow in a normal operating state of the information display device according to the first embodiment. A conceptual diagram of operation in the normal operating state according to the first embodiment. A diagram showing an example of a screen display displayed in the normal operating state. A diagram showing an example of a screen display displayed in the normal operating state. A diagram showing a screen state when the screen display is off. A flowchart showing a processing flow in a standby state according to the first embodiment. A flowchart showing a processing flow in a normal operating state according to the second embodiment. A conceptual diagram of operation during normal operation according to the second embodiment. A diagram showing an example of a screen display displayed in the normal operating state. A system configuration diagram of an information display system according to a third embodiment. A hardware configuration diagram of a mobile terminal device according to the third embodiment. A functional block diagram of the mobile terminal device. A sequence diagram showing an operation sequence during use of the information display device according to the third embodiment. A flowchart showing details of a process for generating display control information for the information display device. A flowchart showing a processing flow in a standby state according to the third embodiment. An operation sequence diagram relating to a forced use suspension process. A diagram showing an example of an operation control screen. A diagram showing an example of an operation control screen. 10A and 10B are diagrams showing an example of a screen display during a forced disabling process;

[0013] The information display device and information display system according to the present embodiment can prevent excessive immersion in a virtual space even when the device is worn on the user's head and the user cannot see the outside world, thereby improving the convenience of the information display device. Therefore, the present invention can increase the commercial value of information display devices to which the present invention is applied, and is therefore expected to contribute to Goal 8.2 of the Sustainable Development Goals (SDGs) proposed by the United Nations (increasing economic productivity through diversification, technological improvement, and innovation, particularly in industries that increase the value of goods and services and labor-intensive industries).

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and duplicated explanations will be omitted.

[0015] First Embodiment FIG. 1 is a diagram showing the hardware configuration of an information display device 100 according to this embodiment.

[0016] The information display device 100 includes a processor 101 (corresponding to the first processor) using a CPU, MCU, etc., a RAM 103, a storage 110, an operation input device 120, an image processing unit 130, an audio processing unit 140, a location information acquisition unit 150, a group of sensors 160, a communication interface (I / F) 170 (corresponding to the first communication device), and an expansion I / F 180, which are connected to each other by a bus 102.

[0017] The bus 102 is a path for sending and receiving commands and data.

[0018] The storage 110 is configured using a non-volatile storage medium such as a flash ROM, an EEPROM, an SSD, or an HDD.

[0019] The operation input device 120 is a user operation I / F, and includes operation keys 121 and a touch sensor 122 .

[0020] The operation keys 121 include, for example, switches, a power key, and a volume key.

[0021] The touch sensor 122 is configured by, for example, a touch pad, and is used to operate the information display device 100, etc. When a separate controller (not shown) is used, the touch sensor 122 may be omitted.

[0022] The image processing unit 130 includes a display 131 (corresponding to a first display), an image signal processing unit 132 , an outer camera 133 , and an inner camera 134 .

[0023] The display 131 is a non-transmissive display because it is a video see-through VR head-mounted display. A transmissive display with variable transparency may also be used.

[0024] The image signal processing unit 132 is configured using, for example, an image (video) signal processor.

[0025] The outer camera 133 is a camera for capturing images of the surroundings.

[0026] The in-camera 134 is a camera for detecting the line of sight.

[0027] The audio processing unit 140 includes an audio output unit 141 , an audio signal processing unit 142 , and a microphone 143 .

[0028] The audio output unit 141 includes, for example, a speaker, an audio output terminal, and the like.

[0029] The audio signal processing unit 142 is configured using, for example, an audio signal processor.

[0030] The location information acquisition unit 150 is a device that detects the current location of the information display device 100, and may be, for example, a GPS sensor, or a so-called VPS system that extracts feature points in a space by analyzing a surrounding image captured by the outer camera 133 and acquires relative location information based on the feature points. Alternatively, it may be a Bluetooth (registered trademark) beacon receiver or the like.

[0031] The sensor group 160 includes a gyro sensor 161 , a geomagnetic sensor 162 , an acceleration sensor 163 , a wearing sensor 164 , and a depth sensor 165 .

[0032] The wearing sensor 164 is a sensor that detects the state in which the user is wearing the HMD.

[0033] The depth sensor 165 is a sensor that detects the distance to an object.

[0034] The communication I / F 170 includes a communication processor, a LAN I / F 171, and a BT I / F 172 (BT: BlueTooth (registered trademark)).

[0035] The expansion I / F 180 is, for example, a USB device connection terminal, and is used for data transmission / reception, charging, and the like.

[0036] Although not shown, the device is also provided with a timer that holds the current time in the real world. The timer holds a time such as Coordinated Universal Time (UTC) as the current time. The timer may be configured as software, or may be configured using an RTC (Real Time Clock).

[0037] 2A is a top view showing a state in which user U1 is wearing information display device 100, Fig. 2B is a front view showing a state in which user U1 is wearing information display device 100, and Fig. 2C is a left side view showing a state in which user U1 is wearing information display device 100. In Figs. 2A to 2C, Xh, Yh, and Zh are coordinate axes of a coordinate system based on information display device 100, with Xh indicating the left-right direction of information display device 100, Yh indicating the up-down direction of information display device 100, and Zh indicating the front-to-back direction of information display device 100.

[0038] As shown in FIG. 2A, the information display device 100 is formed in a shape that surrounds the eyes of a user U1 from above, below, left and right, and blocks the outside world from the field of view of the user U1.

[0039] 2B , in the information display device 100, the left display 131L is disposed inside the housing of the information display device 100 at a position visible to the left eye of the user U1. The right display 131R is disposed inside the housing of the information display device 100 at a position visible to the right eye of the user U1. In this embodiment, the left display 131L and the right display 131R may be collectively referred to as the display 131. As another configuration example, a single display 131 may divide its display area to display an image for the left eye and an image for the right eye side by side.

[0040] Further, on the outside of the housing of the information display device 100, a left outer camera 133L, a right outer camera 133R, a left depth sensor 165L, and a right depth sensor 165R are arranged.

[0041] 2C, a left touch sensor 122L and a left stereo speaker 141L are provided on the left side of the housing of the information display device 100. Although the right side is not shown in FIG. 2, the right side of the information display device 100 is provided with a right stereo speaker, a monaural microphone, and a charging and wired communication terminal which is one of the expansion I / Fs 180.

[0042] Instead of the left touch sensor 122L, a right touch sensor may be provided on the right side surface of the information display device 100. The touch sensor may be provided on either the left or right side, or on both sides.

[0043] In this embodiment, the information display device 100 is described as an example of a device in which a display and the like are integrated with a housing, but it may also be a device that is used by inserting a general-purpose smartphone or the like into a head-mounted adapter. In this case, the following processes are executed by the internal functions of the general-purpose smartphone or the like.

[0044] FIG. 3 is a functional block diagram of the information display device 100.

[0045] The storage 110 of the information display device 100 includes a basic operation program 1001 including basic programs such as an OS, an application program 1002 for realizing the functions of the information display device 100 of this embodiment, and a various data storage area 1009 which serves as an area for storing various operation setting values ​​and various information (video, still images, audio, etc.).

[0046] The RAM 103 includes a basic operation function unit 1101 , an attachment status confirmation unit 1111 , a timer unit 1112 , a display control unit 1113 , and a temporary storage area 1199 .

[0047] The basic operation function unit 1101 controls the basic operations of the information display device 100 .

[0048] The wearing status confirmation unit 1111 analyzes the wearing status of the information display device 100 by the user U1 based on the detection result of the wearing sensor 164 and the state of the eyes of the user U1 captured in the image captured by the in-camera 134, and outputs the result as a wearing flag H. The wearing flag H corresponds to wearing status information.

[0049] The timing unit 1112 outputs time information t indicating the time elapsed when the processor 101 performed a predetermined process by referring to a timer function (not shown) provided in the information display device 100. The time information t corresponds to an actual measurement value measured by the timing unit 112.

[0050] The display control unit 1113 controls the accumulated usage time T and the display enable flag F stored in the various data storage area 1009 based on the operating status of the information display device 100 (normal operating state, standby state, etc.), the attachment flag H, and time information t. The display control unit 1113 also controls the image information to be displayed on the display 131 based on the display enable flag F and the attachment flag H. The display control unit 1113 also controls the image information to be displayed on the display 131 based on display control information received from the mobile terminal device 200 via the LAN I / F 171.

[0051] The cumulative usage time T is (1) the cumulative value of the time during which the information display device is used within a predetermined period of time, or (2) the cumulative value of the time during which the information display device is continuously used.

[0052] In the case of (1), for example, the cumulative usage time T may be a value obtained by accumulating the time for which the information display device 100 is used between midnight and midnight. In this case, for example, three hours per day may be set as the cumulative usage limit threshold. In this way, if a child uses the information display device 100 for two hours, then temporarily stops using the information display device 100, does homework for one hour, and then resumes using the information display device 100, it is possible to limit the use of the information display device 100 to the remaining one hour for the day.

[0053] In the case of (2), the time for which the information display device 100 is continuously used is measured. As a definition of continuous use, an interruption of 10 minutes or less may be counted as a break and the measurement may continue.

[0054] The temporary storage area 1199 is a temporary storage area for various information created or acquired by an application.

[0055] The basic operation program 1001 and application program 1002 stored in the storage 110 are expanded (loaded) into the RAM 103, and the processor 101 executes them to form the various functional units of the information display device 100 of this embodiment, specifically the basic operation function unit 1101, the wearing status confirmation unit 1111, the timing unit 1112, and the display control unit 1113.

[0056] (Normal Operation State) FIG. 4A is a flowchart showing the flow of processing in the normal operation state of the information display device 100 according to the first embodiment.

[0057] The basic operation function unit 1101 checks whether the operation input device 120 has received a power-on operation (such as pressing the power key) from the user U1 (S101). If the power-on operation has been received (S101: Yes), the process proceeds to step S102, where the device transitions from the standby state to the normal operation state. On the other hand, if the power-on operation has not been received (S101: No), the process of step S101 is repeated, and the device continues to wait for a power-on operation from the user U1 (standby state).

[0058] The display control unit 1113 reads the cumulative usage time T and the display permission flag F from the various data storage area 1009 of the storage 110 (S102). In the initial state, the cumulative usage time T is set to "0" and the display permission flag F is set to "On." The display permission flag F being "On" indicates that display of VR images (virtual reality images) is permitted, and the display permission flag F being "Off" indicates that display of VR images is not permitted. The cumulative usage time T and the display permission flag F are stored in the various data storage area 1009 of the storage 110. When the information display device 100 is shared by multiple users, the cumulative usage time T and the display permission flag F for each user may be stored in association with a user ID that can identify the user. Furthermore, the cumulative usage time T and the display permission flag F may be stored in association with the user ID in a server 300 on the network 400, which will be described later. In this case, even if the same user uses different information display devices 100, it is possible to continue accumulating the usage time between the different information display devices 100.

[0059] The display control unit 1113 checks the display enable flag F (S103). If the display enable flag F is "On" (S103: On), the process proceeds to step S104, and if the display enable flag F is "Off" (S103: Off), the process proceeds to step S113.

[0060] The display control unit 1113 checks or sets the wearing flag H (S104). The wearing flag H is set to either “On” or “Off” depending on the wearing status of the information display device 100 of the user U1 detected by the wearing status check unit 1111. The “On” setting of the wearing flag H indicates that the user U1 is wearing the information display device 100, and the “Off” setting of the wearing flag H indicates that the user U1 is not wearing the information display device 100. If the wearing flag H is “On” (S104: On), the process proceeds to step S105. If the wearing flag H is “Off” (S104: Off), the process proceeds to step S109. In this step, even if the user U1 is wearing the information display device 100, if the user U1 has closed their eyes for a predetermined time or longer, the wearing flag H may be set to “Off.” Whether the user U1's eyes are open or closed may be detected by image processing of an eye image captured by the in-camera 134.

[0061] If the wearing flag H is "On", the display control unit 1113 displays the VR image on the display 131 (S105). When the display of the VR image starts, the timer unit 1112 starts measuring the cumulative usage time.

[0062] The display control unit 1113 updates the cumulative usage time T by adding the value obtained by multiplying the elapsed time t measured by the timer unit 1112 by the first coefficient α (S106). The process of adding the value obtained by multiplying the elapsed time t measured by the timer unit 1112 by the first coefficient α, as in step S106, is referred to as counting up the cumulative usage time T.

[0063] The first coefficient α may be a value according to the display content of the display 131, and the value to be used may be determined and set by the display control unit 1113 based on the type of application being executed, the brightness value of the display 131, the display magnification rate, etc. For example, the first coefficient α may be set to 1.0 for a game image with a lot of movement, 0.8 for a game image with little movement, 0.6 for a meeting image using avatars, and 0.4 for an outside world image captured through a camera. Furthermore, the first coefficient α may be set to 1.2 for an image with high brightness, 1.2 for an image including small characters, etc.

[0064] The cumulative use time T is saved while the user U1 is wearing the information display device 100, even if the information display device 100 is not being used.

[0065] The display control unit 1113 checks whether the cumulative usage time T is equal to or greater than a third predetermined threshold Th3 (cumulative usage limit threshold) (S107). If the cumulative usage time T is equal to or greater than the third predetermined threshold Th3 (S107: Yes), the process proceeds to step S108. If the cumulative usage time T is less than the third predetermined threshold Th3 (S107: No), the process proceeds to step S120. The third predetermined threshold Th3 will be described later with reference to FIG. 4B.

[0066] If the cumulative usage time T is equal to or greater than the third predetermined threshold value Th3, the display control unit 1113 updates the displayable flag F to "Off" (S108), and the process proceeds to step S120. In this step, a warning may be displayed before the displayable flag F is set to "Off." Alternatively, a grace period may be set after the displayable flag F is set to "Off" before the external image is actually displayed. This allows time to save data for a running application, such as a game.

[0067] If the attachment flag H is "Off" in step S104, the display control unit 1113 stops the image display on the display 131 or displays a black image (S109).

[0068] The display control unit 1113 updates the cumulative usage time T by subtracting the value obtained by multiplying the elapsed time t measured by the timer unit 1112 by the second coefficient β (S110). The process of subtracting the value obtained by multiplying the elapsed time t measured by the timer unit 1112 by the second coefficient β, as in step S110, is referred to as counting down the cumulative usage time T.

[0069] The display control unit 1113 may acquire biometric information from an external device while the user U1 of the information display device 100 is not wearing the information display device 100, and determine the second coefficient based on the biometric information. The external device may be, for example, a smartwatch or a wearable device that acquires biometric information, and the biometric information may include the user U1's respiratory rate, body temperature, heart rate, and activity level measured by a motion sensor (such as an acceleration sensor, gyro sensor, or geomagnetic sensor). The information display device 100 may be communicatively connected to the external device to acquire biometric information, and the cumulative usage time T may be reset (returned to 0) if the user U1 has slept for a predetermined period of time or longer based on the biometric information. Furthermore, the second coefficient β may be increased so that the subtraction value increases when the user performs a predetermined exercise.

[0070] The display control unit 1113 checks the cumulative usage time T (S111). If the cumulative usage time T is equal to or less than 0 (S111: Yes), the process proceeds to step S112. If the cumulative usage time T is greater than 0 (S111: No), the process proceeds to step S120.

[0071] If the cumulative usage time T is equal to or less than 0, the display control unit 1113 updates the value of the cumulative usage time T to 0 (S112), and the process proceeds to step S120.

[0072] On the other hand, if the displayable flag F is "Off" in step S103, the display control unit 1113 checks or sets the wearing flag H (S113). If the wearing flag H is "On" (S113: On), the process proceeds to step S114, and if the wearing flag H is "Off" (S113: Off), the process proceeds to step S116. In this step, even if the user U1 is wearing the information display device 100, if the user U1 has their eyes closed for a predetermined time or longer, the wearing flag H may be set to "Off." Whether the user U1's eyes are open or closed may be detected by image processing of the eye image captured by the in-camera 134.

[0073] When the attachment flag H is "On," the display control unit 1113 displays the external world image acquired by the outer camera 133 on the display 131 (S114). The display switching process to the external world image may be performed by setting the ratio of virtual reality image to external world image to 100:0 when the cumulative usage time T reaches the cumulative usage limit threshold (third predetermined threshold Th3), and then decreasing the ratio of virtual reality image and increasing the ratio of external world image as time passes, until the ratio finally becomes 0:100. This allows time for shutdown preparation operations such as saving data.

[0074] The display control unit 1113 does not update the value of the cumulative use time T, but keeps it as it is (S115).

[0075] If the attachment flag H is "Off" in step S113, the display control unit 1113 stops the image display on the display 131 or displays a black image (S116).

[0076] The display control unit 1113 updates the cumulative usage time T by subtracting the value obtained by multiplying the elapsed time t measured by the timer unit 1112 by the second coefficient β from the cumulative usage time T (S117).

[0077] The display control unit 1113 checks whether the cumulative usage time T is less than a first predetermined threshold Th1 (corresponding to the restart permission threshold) (S118). If the cumulative usage time T is less than the first predetermined threshold Th1 (S118: Yes), the process proceeds to step S119. If the cumulative usage time T is equal to or greater than the first predetermined threshold Th1 (S118: No), the process proceeds to step S120.

[0078] If the cumulative usage time T is less than the first predetermined threshold Th1, the display control unit 1113 updates the display enable flag F to "On" (S119).

[0079] The basic operation function unit 1101 checks whether the operation input device 120 has received a power-off operation (such as pressing the power key) from the user U1 (S120). If a power-off operation has been received (S120: Yes), the processing ends. As a result, the information display device 100 transitions from the normal operating state to the standby state. If the basic operation function unit 1101 has not received a power-off operation (S120: No), the processing returns to step S103.

[0080] 4B is a conceptual diagram of operation in a normal operating state according to the first embodiment, in which the horizontal axis indicates elapsed time t measured by the timer unit 1112, and the vertical axis indicates cumulative usage time T. In addition, Th1 in the conceptual diagram of operation indicates a first predetermined threshold, and Th3 indicates a third predetermined threshold. Below, the explanation will be divided into six time periods, areas A to F, along the elapsed time t.

[0081] Area A (0.00h to 3.00h): User U1 is wearing the information display device 100, and the display permission flag F is "On." Therefore, the display control unit 1113 allows the display of VR images on the display 131. This is indicated as "display permission" in FIG. 4B. In the flowchart of FIG. 4A, this corresponds to the process of determining No in step S107, passing through step S120, and then proceeding to step S105 again.

[0082] Region B (3.00 h to 3.50 h) When the cumulative usage time T of user U1 becomes equal to or greater than the third predetermined threshold Th3, the display enable flag F is updated to "Off" to cause user U1 to stop using the information display device 100. Although user U1 is wearing the information display device 100, because the display enable flag F is "Off," the display control unit 1113 does not allow the display of a VR image on the display 131 and instead displays an outside image acquired by the outer camera 133 on the display 131. In the flowchart of FIG. 4A, this corresponds to the process in which "Yes" is determined in step S107, the display enable flag F is updated to "Off" in step S108, the process proceeds through step S120, and then the process is again changed to "Off" in step S103 and to "On" in step S113, whereupon the outside image is displayed in step S114.

[0083] Region C (3:50 h to 6:50 h) When user U1 takes off the information display device 100 and takes a break because the VR image is no longer displayed in region B, the cumulative usage time T begins to decrease. In region C, user U1 is not wearing the information display device 100 and the display permission flag F is "Off," so the display control unit 1113 does not allow the display of VR images on the display 131, and turns off the display processing of the display 131 or displays a black image on the display 131. In the flowchart of FIG. 4A , this corresponds to the process of determining "Off" in step S103, determining "Off" in step S113, and turning off the display processing of the display 131 or displaying a black image in step S116.

[0084] Region D (6:50 h to 7:00 h) As a result of user U1 continuing to rest without the information display device 100, the cumulative usage time T becomes less than the first predetermined threshold value Th1, and the display enable flag F is updated to "On." At this time, because user U1 is not wearing the information display device 100, the display control unit 1113 turns off the display processing of the display 131 or displays a black image on the display 131. In the flowchart of FIG. 4A , this corresponds to the process of determining "Yes" in step S118, updating the display enable flag F to "On" in step S119, passing through step S120, and then determining "Off" again in step S104 and proceeding to step S109.

[0085] Region E (7:00 h to 8:75 h) When user U1 puts on the information display device 100 again after a sufficient rest, the wearing flag H is updated to "On." Since the display permission flag F is already "On" in region E, the display control unit 1113 allows the display of VR images on the display 131. In the flowchart of FIG. 4A, this corresponds to the process of determining "On" in step S103 and "On" in step S104 and proceeding to step S105.

[0086] Region F (8.75 h and above): The cumulative usage time T of user U1 again exceeds the third predetermined threshold Th3, and the display-enabled flag F is updated to "Off." As in region C, user U1 is not wearing the information display device 100, and the display-enabled flag F is "Off." Therefore, the display control unit 1113 does not allow the display of VR images on the display 131, and turns off the display on the display 131 or displays a black image. In the flowchart of FIG. 4A, this corresponds to the process of determining "Yes" in step S107, updating the display-enabled flag F to "Off" in step S108, then proceeding through step S120 to "Off" again in step S103 and to "Off" in step S113, and turning off the display or displaying a black image in step S116.

[0087] 4C and 4D are diagrams showing examples of screen displays displayed in a normal operating state.

[0088] 4C is an example of a VR image displayed on the screen. The VR image may be, for example, a game play image. In areas A and E, VR images generated by a running application are displayed, and the screen shown in FIG. 4C is displayed on the display 131 while the game is being played.

[0089] 4D is an example of a screen display of an outside image. In area B, an image of the room where the user U1 is present is displayed on the display 131 as an outside image acquired by the outer camera 133.

[0090] 4E is a diagram showing the screen state when the screen display is turned off. In areas C, D, and F, the display process on the display 131 is turned off, or a black image shown in FIG. 4E is displayed.

[0091] (Standby State) FIG. 4F is a flowchart showing the flow of processing in the standby state according to the first embodiment.

[0092] The basic operation function unit 1101 checks whether the operation input device 120 has received a power-off operation (such as pressing the power key) from the user U1 (S201). If the power-off operation has been received (S201: Yes), the normal operation state is switched to a standby state, and the display control unit 1113 stops displaying images on the display 131 (S202). If the power-off operation has not been received (S201: No), the process of step S201 is repeated, and the normal operation state is maintained.

[0093] The display control unit 1113 reads the cumulative use time T and the display enable flag F from the various data storage area 1009 of the storage 110 (S203).

[0094] The display control unit 1113 checks the display enable flag F (S204). If the display enable flag F is “On” (S204: On), the display control unit 1113 updates the cumulative usage time T by subtracting the value obtained by multiplying the elapsed time t measured by the timer unit 1112 by the second coefficient β (S205).

[0095] The display control unit 1113 checks the cumulative usage time T (S206). If the cumulative usage time T is equal to or less than 0 (S206: Yes), the display control unit 1113 sets the value of the cumulative usage time T to 0 (S207) and proceeds to the processing of step S211. If the cumulative usage time T is greater than 0 (S206: No), the display control unit 1113 proceeds to the processing of step S211.

[0096] On the other hand, the display control unit 1113 checks the displayable flag F (S204), and if the displayable flag F is "Off" (S204: Off), the display control unit 1113 updates the cumulative usage time T by subtracting the value obtained by multiplying the elapsed time t measured by the timing unit 1112 by the second coefficient β (S208).

[0097] The display control unit 1113 checks whether the cumulative usage time T is less than the first predetermined threshold value Th1 (S209). If the cumulative usage time T is less than the first predetermined threshold value Th1 (S209: Yes), the display control unit 1113 updates the displayable flag F to On (S210) and proceeds to processing of step S211.

[0098] If the cumulative usage time T is equal to or greater than the first predetermined threshold value Th1 (S209: No), the process proceeds to step S211.

[0099] The basic operation function unit 1101 checks whether the operation input device 120 has received a power-on operation (such as pressing the power key) from the user U1 (S211). If the power-on operation has been received (S211: Yes), the processing ends and the device transitions from the standby state to the normal operation state. If the power-on operation has not been received (S211: No), the processing returns to step S204.

[0100] According to this embodiment, when the cumulative usage time T of the information display device 100 reaches a third predetermined threshold Th3, an image of the outside world is displayed, and the user U1 can be brought back from virtual reality to real space, thereby preventing excessive immersion in virtual reality.

[0101] Furthermore, after the user U1 removes the information display device 100, the screen display can be turned off to reduce power consumption.

[0102] Furthermore, when use is resumed, the cumulative usage time T is subtracted, and use can be resumed when it reaches the first predetermined threshold Th1, which is a value smaller than the third predetermined threshold Th3.Therefore, user U1 can be forced to stop using the information display device 100 and take a break during the period from the third predetermined threshold Th3 to the first predetermined threshold Th1.

[0103] 4A, the above-described processes, including the measurement and storage of the cumulative usage time T, may be performed for each user identification information (hereinafter referred to as "user ID") that identifies a user. For example, the cumulative usage time T and the display enable flag F are managed for each user ID. In this case, a user login process is performed after the power-on operation is accepted (S101) to identify the user ID. In this way, when multiple people use the same information display device 100, the device can be stopped from using the device according to each user's cumulative usage time.

[0104] Furthermore, the accumulated use time T and the state of the displayable flag F are maintained even if the application being executed is changed.

[0105] The cumulative use time T may also be referred to as the fatigue level Ti.

[0106] The cumulative usage time T or the cumulative degree of fatigue Ti may be varied depending on the application being used. For example, the cumulative degree may be increased for an application in which objects move quickly, the brightness is brighter, and the text is small during execution, while the cumulative degree may be decreased for an application in which objects move slowly, the brightness is appropriate, and the text is large (easy to read). Since the user's sense of immersion in the virtual space and fatigue level differ between an application that provides a high level of immersion and stimulates gambling, such as a competitive game, and an application that plays ambient music and video, the value of the first coefficient α used in step S106 may be varied depending on the content of the application to adjust the speed at which the cumulative usage time T is counted up. The value of the second coefficient β may also be varied to adjust the speed at which the cumulative usage time T is counted down.

[0107] Furthermore, the first predetermined threshold Th1, the second predetermined threshold Th2 (corresponding to the warning start threshold described below), the third predetermined threshold Th3, the first coefficient α, the second coefficient β, and the third coefficient γ (described below) may be set by the user of the information display device 100 (for each user ID) or may be set by a user with higher authority for all users. Alternatively, the information display device 100 may set these as defaults and users may not be able to change them. Furthermore, the user U1 may set the third predetermined threshold Th3 as a self-report of usage time.

[0108] Second Embodiment: Normal Operation State 2 FIG. 5A is a flowchart showing the flow of processing in a normal operation state according to the second embodiment.

[0109] The basic operation function unit 1101 checks whether the operation input device 120 has received a power-on operation (such as pressing the power key) from the user U1 (S301). If the power-on operation has been received (S301: Yes), the process proceeds to step S302, where the device transitions from the standby state to the normal operation state. If the power-on operation has not been received (S301: No), the process of step S301 is repeated, and the device continues to wait for a power-on operation from the user U1 (standby state).

[0110] In step S302, the display control unit 1113 reads the cumulative usage time T, the display flag F, and the superimposition flag M from the various data storage area 1009 of the storage 110 (S302). In the initial state, the cumulative usage time T is set to "0", the display flag F is set to "On", and the superimposition flag M is set to "Off".

[0111] The display control unit 1113 checks the display enable flag F (S303). If the display enable flag F is "On" (S303: Yes), the display control unit 1113 checks the attachment flag H (S304). If the display enable flag F is "Off" (S303: No), the process proceeds to step S321.

[0112] If the wearing flag H is "On" (S304: Yes), the display control unit 1113 checks the superimposition flag M (S305). If the wearing flag H is "Off" (S304: No), the process proceeds to step S314. Note that in this step, even if the user U1 is wearing the information display device 100, if the user U1 has their eyes closed for a predetermined time or longer, the wearing flag H may be treated as "Off" even if it is "On." Whether the user U1's eyes are open or closed may be detected by image processing of the eye image captured by the in-camera 134.

[0113] The display control unit 1113 checks the superimposition flag M (S305). The superimposition flag M is a flag that determines whether or not to perform superimposition processing of the VR image and the outside world image. If the superimposition flag M is "Off" (S305: Off), the VR image is displayed on the display 131 (S306).

[0114] After displaying the VR image on the display 131 (S306), the display control unit 1113 adds the elapsed time t measured by the timer unit 1112 multiplied by the first coefficient α to the cumulative usage time T to update the cumulative usage time T (S307). The display control unit 1113 then checks whether the cumulative usage time T is equal to or greater than a second predetermined threshold value Th2 (S308). If the cumulative usage time T is equal to or greater than the second predetermined threshold value Th2, the display control unit 1113 updates the superimposition flag M to On (S309) and proceeds to the processing of step S328.

[0115] In step S308, if the cumulative usage time T is less than the second predetermined threshold value Th2 (S308: No), the process proceeds to step S328.

[0116] On the other hand, if the superimposition flag M is "On" in step S305 (S305: On), the display control unit 1113 displays an image on the display 131 in which the VR image and the external image acquired by the outer camera 133 are superimposed (S310). The display control unit 1113 updates the cumulative usage time T by adding a value obtained by multiplying the elapsed time t measured by the timing unit 1112 by a third coefficient γ to the cumulative usage time T (S311). The display control unit 1113 then checks whether the cumulative usage time T is equal to or greater than a third predetermined threshold Th3 (S312). If the cumulative usage time T is equal to or greater than the third predetermined threshold Th3 (S312: Yes), the display control unit 1113 updates the display enable flag F to "Off" and the superimposition flag M to "Off" (S313), and proceeds to the processing of step S328.

[0117] In step S312, if the cumulative usage time T is less than the third predetermined threshold value Th3 (S312: No), the process proceeds to step S328.

[0118] If the attachment flag H is "Off" in step S304 (S304: Off), the display control unit 1113 stops image display on the display 131 or displays a black image (S314). Then, the display control unit 1113 updates the cumulative usage time T by subtracting the elapsed time t measured by the timer unit 1112 multiplied by the second coefficient β (S315). Next, the display control unit 1113 checks the superimposition flag M (S316). If the superimposition flag M is "Off" (S316: Off), the process proceeds to step S319.

[0119] If the superimposition flag M is "On" (S316: On), the display control unit 1113 checks whether the cumulative usage time T is less than the second predetermined threshold Th2 (S317). If the cumulative usage time T is less than the second predetermined threshold Th2 (S317: Yes), the display control unit 1113 updates the superimposition flag M to "Off" (S318).

[0120] If the superimposition flag M is "Off" (S316: Off), or if the cumulative usage time T is equal to or greater than the second predetermined threshold Th2 (S317: No), or after step S318, the display control unit 1113 checks the cumulative usage time T (S319). If the cumulative usage time T is equal to or less than 0 (S319: Yes), the value of the cumulative usage time T is set to 0 (S320), and the process proceeds to step S328. If the cumulative usage time T is greater than 0 (S319: No), the process proceeds to step S328 without changing the value of the cumulative usage time T.

[0121] On the other hand, if the displayable flag F is "Off" in step S303 (S303: Off), the display control unit 1113 checks the attachment flag H (S321). If the attachment flag H is On (S321: On), the display control unit 1113 displays an external image acquired by the outer camera 133 on the display 131 (S322), maintains the value of the cumulative usage time T as is (without updating it) (S323), and proceeds to the processing of step S328.

[0122] If the attachment flag H is "Off" (S321: Off), the display control unit 1113 stops image display on the display 131 or displays a black image (S324). The display control unit 1113 updates the cumulative usage time T by subtracting the elapsed time t measured by the timer unit 1112 multiplied by the second coefficient β from the cumulative usage time T (S325). The display control unit 1113 checks whether the cumulative usage time T is less than the first predetermined threshold Th1 (S326). If the cumulative usage time T is less than the first predetermined threshold Th1 (S326: Yes), the display control unit 1113 updates the display enable flag F to "On" (S327) and proceeds to the processing of step S328. If the cumulative usage time T is equal to or greater than the first predetermined threshold Th1 (S326: No), the processing proceeds to step S328.

[0123] In step S328, the basic operation function unit 1101 checks whether the operation input device 120 has received a power-off operation (such as pressing the power key) from the user U1. If a power-off operation has been received (S328: Yes), the processing ends and the device transitions from the normal operating state to the standby state. If a power-off operation has not been received (S328: No), the processing returns to step S303.

[0124] In step S310, the degree of superimposition between the VR image and the outside world image acquired by the outer camera 133 when the superimposed image is displayed may be fixed or may be changed according to the cumulative usage time T. For example, the ratio of the outside world image may be configured to increase according to the length of the cumulative usage time T when the cumulative usage time T is equal to or greater than the second predetermined threshold Th2 and less than the third predetermined threshold Th3.

[0125] Also, while the superimposed image is being displayed, a warning message may be displayed to inform the user that the VR image will soon no longer be displayed.

[0126] 5B is a conceptual diagram of operation during normal operation according to the second embodiment, in which the horizontal axis indicates elapsed time t measured by the timer unit 1112, and the vertical axis indicates cumulative usage time T. In addition, Th1 in the conceptual diagram of operation indicates a first predetermined threshold, Th2 indicates a second predetermined threshold, and Th3 indicates a third predetermined threshold. Below, the explanation will be divided into eight time periods, region G to region O, along the elapsed time t.

[0127] Region G (0.00h to 2.50h): User U1 is wearing the information display device 100, the display permission flag F is "On," and the superimposition flag M is "Off," so the display control unit 1113 allows the display of a VR image on the display 131. In the flowchart of FIG. 5A, this corresponds to the process going through steps S306 and S307, determining "No" in S308, and then going through step S328 and proceeding to step S303 again.

[0128] Region H (2.50 h to 3.50 h): The cumulative usage time T is equal to or greater than the second predetermined threshold value Th2, and the superimposition flag M is updated to "On." In the flowchart of FIG. 5A, this corresponds to the process of determining "Yes" in step S308 and proceeding to step S309. Because user U1 is wearing the information display device 100, the display enable flag F is "On," and the superimposition flag M is updated from "Off" to "On," the display control unit 1113 displays a superimposed image of the outside world image and the VR image acquired by the outer camera 133 on the display 131. In the flowchart of FIG. 5A, this corresponds to the process of determining "On" in step S305 and proceeding to S310 and S311. The superimposition ratio of the outside world image and the VR image may be fixed or may be changed depending on the cumulative usage time T.

[0129] The display control unit 1113 may use a third coefficient γ used to count up the cumulative usage time T from the second predetermined threshold Th2 to the third predetermined threshold Th3 as a value smaller than the first coefficient α used to count up the cumulative usage time T from the start of counting up the cumulative usage time T to the second predetermined threshold Th2 (area G). This provides the user U1 with a grace period between the start of the warning and the switch to the external image, allowing them to perform preparatory operations such as saving data. The third coefficient γ may be the same as the first coefficient α.

[0130] Furthermore, the display control unit 1113 may superimpose the external world image more clearly on the display as the cumulative usage time T approaches the third predetermined threshold Th3 from the second predetermined threshold Th2. That is, in the superimposition process, the ratio of virtual reality image to external world image is 100:0 until the warning start threshold (second predetermined threshold Th2) is reached, and then the ratio of virtual reality image to external world image is decreased and increased as time passes, until the cumulative usage limit threshold (third predetermined threshold Th3) is reached and the ratio of virtual reality image to external world image is 0:100.

[0131] Region I (3.50 h to 4.00 h): The cumulative usage time T is equal to or greater than the third predetermined threshold value Th3, the display flag F is updated to "Off", and the superimposition flag M is updated to "Off". Although the user U1 is wearing the information display device 100, the display control unit 1113 does not allow the display of a VR image on the display 131, and instead displays an external image acquired by the outer camera 133 on the display 131. In the flowchart of FIG. 5A, this corresponds to the process of selecting "Yes" at step S312, proceeding through S313 and S328, selecting "Off" at S303, selecting "On" at S321, and proceeding to S322.

[0132] Region J (4:00 h to 6:00 h) Because user U1 is not wearing the information display device 100 and the display permission flag F is "Off," the display control unit 1113 does not allow the display of VR images on the display 131, and turns off the display processing of the display 131, or displays a black image on the display 131. In the flowchart of FIG. 5A, this corresponds to the processing of proceeding to "Off" in step S303, "Off" in S321, and "No" in S324, S325, and S326.

[0133] Region K (6:00 h to 6:50 h) User U1 wears the information display device 100, but because the display permission flag F is "Off," the display control unit 1113 does not allow the display of a VR image on the display 131, and instead displays an outside image acquired by the outer camera 133 on the display 131. In the flowchart of FIG. 5A, this corresponds to the process of "Off" in step S303, "On" in S321, and proceeding to S322 and S323.

[0134] Region L (6:50 h to 7:50 h): Because user U1 is not wearing the information display device 100 and the display permission flag F is "Off," the display control unit 1113 does not allow the display of VR images on the display 131, and turns off the display processing of the display 131, or displays a black image on the display 131. In the flowchart of FIG. 5A, this corresponds to the processing going through steps S324 and S325 and proceeding to "No" in S326.

[0135] Region M (7:50 h to 8:00 h): The cumulative usage time T becomes less than the first predetermined threshold value Th1, and the display enable flag F is updated to "On." Note that although the display enable flag F is "On," the user U1 is not wearing the information display device 100, so the display control unit 1113 turns off the display processing of the display 131 or displays a black image on the display 131. In the flowchart of FIG. 5A , this corresponds to the processing of "Yes" at S326, going through S327 and S328, "On" at S303, "Off" at S304, and proceeding to S314.

[0136] Region N (8:00 h to 9:25 h): User U1 is wearing the information display device 100, the display permission flag F is "On," and the superimposition flag M is "Off," so the display control unit 1113 allows the display of VR images on the display 131. In the flowchart of FIG. 5A, this corresponds to the process of "On" in S304, "Off" in S305, and proceeding to S306.

[0137] Region O (from 9.25 h): The cumulative usage time T is equal to or greater than the second predetermined threshold value Th2, and the superimposition flag M is updated to "On." Because the user U1 is wearing the information display device 100, the display enable flag F is "On," and the superimposition flag M is "On," the display control unit 1113 displays a superimposed image of the outside world image and the VR image acquired by the outer camera 133 on the display 131. In the flowchart of FIG. 5A , this corresponds to the process of selecting "Yes" in S308, passing through S309 and S328, returning to S303 again, selecting "On" in S305, and proceeding to S310. In the processing of region O, the superimposition ratio of the outside world image and the VR image may be fixed or may be changed depending on the cumulative usage time T.

[0138] Below, examples of screen displays displayed in areas G to O will be described.

[0139] In areas G and N, a screen similar to that shown in FIG. 4C is displayed, showing a VR image generated by the running application.

[0140] 5C is an example of a superimposed display of a VR image and an outside world image. In areas H and O, the screen of FIG. 5C is displayed, in which a VR image generated by a running application and an outside world image acquired by the outer camera 133 are superimposed. As the cumulative usage time T increases further, the screen transitions to an example of an outside world image similar to that of FIG. 4D.

[0141] In the areas I and K, a screen similar to that shown in FIG. 4D is displayed, which displays an outside image captured by the outer camera 133.

[0142] In the areas J, L, and M, the display process on the display 131 is turned off, or a black image is displayed as in the screen shown in FIG. 4E.

[0143] According to this embodiment, before the cumulative usage time T of the information display device 100 reaches the third predetermined threshold Th3 and the external world image is displayed, the external world image is superimposed on the VR image, so that the user U1 can be gradually brought back from the virtual reality to the real space. This allows the user U1 to mentally prepare to return from the virtual reality to the real space and perform preparatory operations such as saving the data of running applications.

[0144] Third Embodiment FIG. 6 is a system configuration diagram of an information display system according to a third embodiment.

[0145] The information display system 1 is configured by communicatively connecting an information display device 100 and a mobile terminal device 200 via a network 400. The information display device 100 may be, for example, a head-mounted display (HMD). A server 300 is connected to the network 400. A user U1 wears the information display device 100 on his / her head, runs a VR application on the information display device 100, and views VR images provided by the server 300.

[0146] The mobile terminal device 200 may be, for example, a smartphone, a smartwatch, or a personal computer. User P1 is a person with higher authority than user U1, such as a guardian. User P1 owns the mobile terminal device 200 and controls the operation of the information display device 100 worn by user U1 (a person with lower authority, such as a child).

[0147] 7 is a diagram showing the hardware configuration of a mobile terminal device 200 according to the second embodiment. Of the components of the mobile terminal device 200, those with the same names as those of the information display device 100 will not be described again.

[0148] The mobile terminal device 200 includes a processor 201 (corresponding to a second processor) using a CPU, MCU, etc., a RAM 203, a storage 210, an operation input device 220, an image processing unit 230, an audio processing unit 240, a location information acquisition unit 250, a group of sensors 260, a communication interface (I / F) 270 (corresponding to a second communication device), and an expansion I / F 280, which are connected to each other by a bus 202.

[0149] The storage 210 stores a usage information DB 211. The usage information DB 211 is a database that stores information about the usage of the information display device 100 by the user U1, and is registered in the storage 210 in advance.

[0150] The operation input device 220 is a user operation I / F, and includes operation keys 221 and a touch panel 222 .

[0151] The touch panel 222 is disposed over the display 231 (corresponding to a second display) and is used to operate a smartphone or the like, for example.

[0152] The audio output unit 241 may include, for example, a monaural speaker used during calls and a stereo speaker used during AV playback.

[0153] The microphone 243 may include a monaural microphone used during calls and a stereo microphone used during video recording.

[0154] The sensor group 260 may include a gyro sensor, a geomagnetic sensor, an acceleration sensor, an angle sensor, a distance measurement sensor, an illuminance sensor, and the like.

[0155] The communication I / F 270 includes a communication processor, and includes, for example, a LAN I / F 271, a BT I / F 272 (BT: BlueTooth (registered trademark)), and a telephone network I / F 273 (Mobile I / F).

[0156] Fig. 8 is a functional block diagram of the mobile terminal device 200. Of the components of the mobile terminal device 200 shown in Fig. 8, duplicated explanations of components with the same names as those of the information display device 100 will be omitted.

[0157] The storage 210 of the mobile terminal device 200 includes a basic operation program 2001 , an application program 2002 , a various data storage area 2009 , and a usage information DB 211 .

[0158] The RAM 203 includes a basic operation function unit 2101 , an operation status confirmation unit 2111 , a timer unit 2112 , an operation control unit 2113 , and a temporary storage area 2199 .

[0159] The basic operation function unit 2101 controls the basic operations of the mobile terminal device 200 .

[0160] The operation status check unit 2111 acquires the usage status (start and end of use) and wearing status of the information display device 100 by the user U1 from the information display device 100 via the LAN I / F 271 .

[0161] The timer unit 2112 references a timer function (not shown) provided in the mobile terminal device 200 to output information t about the time that has elapsed since the processor 201 performed a predetermined process.

[0162] The operation control unit 2113 controls the cumulative usage time T and display enable flag F for the information display device 100 of user U1 stored in the usage status information DB 211 based on the usage status, wearing status, and time information t of the information display device 100. The operation control unit 2113 also generates display control information (a VR image display command, an outside world image display command, and an image display off command) based on the above and transmits it to the information display device 100 via the LAN I / F 271. Furthermore, in response to an operation instruction from user P1, it transmits usage stop information to the information display device 100 via the LAN I / F 271.

[0163] The temporary storage area 2199 is a temporary storage area for various information created and acquired by the application program 2002 .

[0164] The basic operation program 2001 and application program 2002 stored in storage 210 are expanded (loaded) into RAM 203, and the processor 201 executes them to form the various functional units of the mobile terminal device 200 of this embodiment, specifically the basic operation function unit 2101, the operation status confirmation unit 2111, the timing unit 2112, and the operation control unit 2113.

[0165] (Operation sequence: while the information display device is in use) Fig. 9A is a sequence diagram showing an operation sequence while the information display device is in use in the second embodiment. In Fig. 9A, the information display device 100 is illustrated as an HMD, and the mobile terminal device 200 is illustrated as an S-Phone (smartphone).

[0166] The basic operation function unit 1101 of the information display device 100 checks whether the operation input device 120 has received a power-on operation (such as pressing the power key) from the user U1 (S401). If the power-on operation has been received (S401: Yes), the process proceeds to step S402, where the device transitions from the standby state to the normal operation state. If the power-on operation has not been received (S401: No), the process of step S401 is repeated, and the device continues to wait for a power-on operation from the user U1 (standby state).

[0167] When the information display device 100 transitions to the normal operation state, the basic operation function unit 1101 of the information display device 100 transmits information that use of the information display device 100 has started (start-of-use information) to the mobile terminal device 200 via the communication I / F 170 (S402), and the operation status confirmation unit 2111 of the mobile terminal device 200 receives information that use of the information display device 100 has started (start-of-use information) from the information display device 100 via the communication I / F 270 (S403). The start-of-use information includes the device ID of the information display device 100 or the user ID of the user using the information display device 100.

[0168] The operation status confirmation unit 2111 of the mobile terminal device 200 reads the cumulative usage time T and the displayable flag F associated with the device ID or user ID included in the use start information received in the process of step S403 from the usage status information DB 211 of the storage 210 (S404). In the initial state, the cumulative usage time T is set to "0" and the displayable flag F is set to "On."

[0169] The display control unit 1113 of the information display device 100 checks the wearing flag H (the wearing status of the information display device 100 of user U1 detected by the wearing status checking unit 1111) and transmits the wearing flag H to the mobile terminal device 200 via the communication I / F 170 (S405), and the operation status checking unit 2111 of the mobile terminal device 200 receives the wearing flag H from the information display device 100 via the communication I / F 270 (S406).

[0170] The operation status confirmation unit 2111 of the mobile terminal device 200 generates display control information for the information display device 100 based on the cumulative use time T, the displayable flag F, and the attachment flag H (S407). Details of this step will be described later with reference to FIG. 9B.

[0171] The operation status confirmation unit 2111 of the mobile terminal device 200 transmits the display control information generated in the processing of S407 to the information display device 100 via the communication I / F 270 (S408), and the display control unit 1113 of the information display device 100 receives the display control information from the mobile terminal device 200 via the communication I / F 170 (S409).

[0172] The display control unit 1113 of the information display device 100 checks the display control information received in the processing of S409 (S410). If the display control information is "Display VR image" (S410: VR image), the display control unit 1113 of the information display device 100 displays the VR image on the display 131 (S411). If the display control information is "Display outside world image" (S410: outside world image), the display control unit 1113 of the information display device 100 displays the outside world image acquired by the outer camera 133 on the display 131 (S412). If the display control information is "Image display off" (S410: off), the display control unit 1113 of the information display device 100 stops image display on the display 131 or displays a black image (S413).

[0173] After steps S411, S412, and S413, the basic operation function unit 1101 of the information display device 100 checks whether the operation input device 120 has received a power-off operation (such as pressing the power key) from the user U1 (S414). If the power-off operation has been received (S414: Yes), the basic operation function unit 1101 of the information display device 100 transmits end-of-use information of the information display device 100 to the mobile terminal device 200 via the communication I / F 170 (S415) and transitions to a standby state. The end-of-use information includes the device ID of the information display device 100 or the user ID of the user using the information display device 100.

[0174] The operation status check unit 2111 of the mobile terminal device 200 receives the use end information from the information display device 100 via the communication I / F 270 (S416).

[0175] On the other hand, if the basic operation function unit 1101 of the information display device 100 has not received the power-off operation (S414: No), the process returns to step S405.

[0176] The mobile terminal device 200 may perform the above process whether in a normal operating state or in a standby state.

[0177] Furthermore, if display control information cannot be received from the mobile terminal device 200 in the processing of step S409, the VR image may be displayed only for a predetermined time after the start-of-use information is sent, and thereafter the outside world image may be displayed.

[0178] Alternatively, the server 300 may perform the process on the right side instead of the mobile terminal device 200.

[0179] (Display Control Information Generation Process: Information Display Device in Use) FIG. 9B is a flowchart showing details of the process of generating display control information for the information display device 100 in step S407.

[0180] The operation status confirmation unit 2111 checks the display permission flag F (S501). If the display permission flag F is "On" (a state in which display of VR images is permitted) (S501: On), the operation status confirmation unit 2111 checks the wearing flag H (the wearing status of the information display device 100 of the user U1 detected by the wearing status confirmation unit 1111) (S502). If the display permission flag F is "Off" (a state in which display of VR images is not permitted), the process proceeds to step S511.

[0181] In step S502, if the wearing flag H is "On" (the user U1 is wearing the information display device 100) (S502: On), the operation status checking unit 2111 sets "VR image display" as the display control information (S503). Furthermore, the operation status checking unit 2111 adds the value obtained by multiplying the elapsed time t measured by the timing unit 2112 by the first coefficient α to the cumulative usage time T, thereby updating the cumulative usage time T (S504).

[0182] Next, the operation status checking unit 2111 checks whether the cumulative usage time T is equal to or greater than the third predetermined threshold value Th3 (S505). If the cumulative usage time T is equal to or greater than the third predetermined threshold value Th3 (S505: Yes), the operation status checking unit 2111 updates the displayable flag F to Off (S506). Then, the process ends and returns to the main routine. On the other hand, if the cumulative usage time T is less than the third predetermined threshold value Th3 (S505: No), the process returns to the main routine.

[0183] In step S502, if the wearing flag H is "Off" (a state in which the information display device 100 is not worn by the user U1) (S502: Off), the operation status checking unit 2111 sets "image display off" as the display control information (S507). Furthermore, the operation status checking unit 2111 updates the cumulative usage time T by subtracting a value obtained by multiplying the elapsed time t measured by the timer unit 1112 by the second coefficient β (S508).

[0184] Next, the operation status checking unit 2111 checks whether the cumulative usage time T is 0 or less (S509). If the cumulative usage time T is 0 or less (S509: Yes), the operation status checking unit 2111 sets the value of the cumulative usage time T to 0 (S510) and returns to the main routine. On the other hand, if the cumulative usage time T is greater than 0 (S509: No), returns to the main routine.

[0185] In step S501, if the display enable flag F is "Off" (S501: Off), the operation status confirmation unit 2111 checks the wearing flag H (the wearing status of the information display device 100 of the user U1 detected by the wearing status confirmation unit 1111) (S511). If the wearing flag H is "On" (the state in which the information display device 100 is worn by the user U1) (S511: On), the operation status confirmation unit 2111 sets "external world image display" as the display control information (S512). Then, the operation status confirmation unit 2111 retains the value of the cumulative usage time T as it is (without updating it) (S513) and returns to the main routine.

[0186] If the wearing flag H is "Off" in step S511 (a state in which the information display device 100 is not worn by the user U1) (S511: Off), the operation status checking unit 2111 sets "image display off" as the display control information (S514). Then, the operation status checking unit 2111 updates the cumulative usage time T by subtracting the elapsed time t measured by the timer unit 2112 multiplied by the second coefficient β (S515).

[0187] The operation status check unit 2111 checks whether the cumulative usage time T is less than the first predetermined threshold Th1 (S516). If the cumulative usage time T is less than the first predetermined threshold Th1 (S516: Yes), the display control unit 1113 updates the display enable flag F to "On" (S517) and returns to the main routine. On the other hand, if the cumulative usage time T is equal to or greater than the first predetermined threshold Th1 (S516: No), the process returns to the main routine.

[0188] In the display control information generation process, the first predetermined threshold Th1, the third predetermined threshold Th3, the first coefficient α, and the second coefficient β may be set by the user of the mobile terminal device 200.

[0189] (Standby State) FIG. 9C is a flowchart showing the flow of processing performed by the mobile terminal device 200 when the information display device 100 is in a standby state according to the second embodiment.

[0190] The operation status confirmation unit 2111 checks whether or not use end information has been received from the information display device 100 (S601). If use end information has been received (S601: Yes), the information display device 100 transitions from a use state to an unused state, and the process proceeds to step S602. If use end information has not been received (S601: No), the process of step S601 is repeated, and the information display device 100 continues to be in use. In this case, the mobile terminal device 200 performs the processes of FIGS. 9A and 9B.

[0191] In S602, the operation status checking unit 2111 reads the cumulative usage time T and the display permission flag F associated with the device ID or user ID included in the use end information received in the processing of step S601 from the usage status information DB 211 of the storage 210 (S602). Then, the operation status checking unit 2111 checks the display permission flag F (S603). If the display permission flag F is "On" (a state in which display of VR images is permitted) (S603: On), the operation status checking unit 2111 updates the cumulative usage time T by subtracting the value obtained by multiplying the elapsed time t measured by the timing unit 2112 by the second coefficient β (S604).

[0192] The operation status checking unit 2111 checks the cumulative usage time T (S605). If the cumulative usage time T is equal to or less than 0 (S605: Yes), the operation status checking unit 2111 sets the value of the cumulative usage time T to 0 (S606) and proceeds to the processing of step S610. If the cumulative usage time T is greater than 0 (S606: No), proceeds to the processing of step S610.

[0193] If the display flag F is "Off" (display of VR images is not permitted) in step S603 (S603: Off), the operation status confirmation unit 2111 updates the cumulative usage time T by subtracting the value obtained by multiplying the elapsed time t measured by the timing unit 1112 by the second coefficient β (S607).

[0194] The operation status check unit 2111 checks whether the cumulative usage time T is less than the first predetermined threshold Th1 (S608). If the cumulative usage time T is less than the first predetermined threshold Th1 (S608: Yes), the operation status check unit 2111 updates the display possible flag F to "On" (S609) and proceeds to the processing of step S610.

[0195] If the cumulative usage time T is equal to or greater than the first predetermined threshold value Th1 (S608: No), the process proceeds to step S610.

[0196] The operation status confirmation unit 2111 checks whether or not use start information has been received from the information display device 100 (S610). If use start information has been received (S610: Yes), the information display device 100 transitions from an unused state to a used state. Thereafter, the mobile terminal device 200 performs the processes of FIGS. 9A and 9B.

[0197] If the start of use information has not been received (S610: No), the process returns to step S603, and the information display device 100 continues to be in an unused state.

[0198] The mobile terminal device 200 performs the above process whether in a normal operating state or in a standby state.

[0199] (Forcible Disabling Process) FIG. 10 is an operational sequence diagram relating to the forcible disabling process.

[0200] The basic operation function unit 1101 of the information display device 100 checks whether the operation input device 120 has received a power-on operation (such as pressing the power key) from the user U1 (S701). If the power-on operation has been received (S701: Yes), the process proceeds to step S702, where the device transitions from the standby state to the normal operation state. If the power-on operation has not been received (S701: No), the process of step S701 is repeated, and the device continues to wait for a power-on operation from the user U1 (standby state).

[0201] The basic operation function unit 1101 of the information display device 100 transmits start-of-use information of the information display device 100 to the mobile terminal device 200 via the communication I / F 170 (S702), and the operation status confirmation unit 2111 of the mobile terminal device 200 receives the start-of-use information from the information display device 100 via the communication I / F 270 (S703). The start-of-use information includes the device ID of the information display device 100 or the user ID of the user using the information display device 100.

[0202] Then, the operation status check unit 2111 of the mobile terminal device 200 starts an operation control application that controls the operation of the information display device 100 (S704). The operation control application is configured by an operation control unit 2113 deployed in the RAM 203.

[0203] The basic operation function unit 1101 of the information display device 100 checks the usage status of the information display device 100 by the user U1 (such as the wearing status and app operation status) and transmits this as usage status information to the mobile terminal device 200 via the communication I / F 170 (S705), and the operation status confirmation unit 2111 of the mobile terminal device 200 receives the usage status information from the information display device 100 via the communication I / F 270 (S706).

[0204] The operation control unit 2113 of the mobile terminal device 200 displays an operation control screen including the usage status information received in the process of S706 on the display 231 (S707). Details of this step will be described later with reference to Figs. 11A and 11B.

[0205] The operation control unit 2113 of the mobile terminal device 200 checks whether the operation input device 220 has received a stop use instruction operation from the user P1 (S708). If the stop use instruction operation has been received (S708: Yes), the operation control unit 2113 of the mobile terminal device 200 transmits stop use information to the information display device 100 via the communication I / F 270 (S709). If the stop use instruction operation has not been received (S708: No), the stop use information is not transmitted.

[0206] The display control unit 1113 of the information display device 100 checks whether or not use suspension information has been received from the mobile terminal device 200 via the communication I / F 170 (S710). If use suspension information has not been received (S710: No), the display control unit 1113 of the information display device 100 displays a VR image on the display 131 (S711). On the other hand, if use suspension information has been received (S710: Yes), the display control unit 1113 of the information display device 100 displays an outside world image acquired by the outer camera 133 on the display 131 (S712).

[0207] The basic operation function unit 1101 of the information display device 100 checks whether the operation input device 120 has received a power-off operation (such as pressing the power key) from the user U1 (S713). If the power-off operation has been received (S713: Yes), the basic operation function unit 1101 of the information display device 100 transmits end-of-use information of the information display device 100 to the mobile terminal device 200 via the communication I / F 170 (S714). The end-of-use information includes the device ID of the information display device 100 or the user ID of the user using the information display device 100. The information display device 100 then transitions to a standby state.

[0208] When the operation status confirmation unit 2111 of the mobile terminal device 200 receives information indicating that use of the information display device 100 has ended from the information display device 100 via the communication I / F 270 (S715), the operation status confirmation unit 2111 ends the operation of the operation control application that controls the operation of the information display device 100 (S716). In step S716, the operation control unit 2113 (operation control application) deployed in the RAM 203 is released.

[0209] In step S713, if a power-off operation has not been received (S713: No), the process returns to step S705.

[0210] (Operation Control Screen) Figures 11A and 11B are diagrams showing an example of an operation control screen, and Figures 12A and 12B are diagrams showing an example of a screen display during a forced disable process.

[0211] The mobile terminal device 200 includes a touch screen 290 configured by overlapping a display 231 and a touch panel 222. The display 231 displays a message informing the user U1 of the usage status of the information display device 100.

[0212] 11A shows an example of a message in which a warning message 290M1 is displayed because the continuous use time of the information display device 100 by the user U1 has exceeded a predetermined value. Also, in FIG. 11B, a notification message 290M2 is displayed to notify the user U1 of the usage status of the information display device 100. Furthermore, an operation guide message 290M3 is also displayed to prompt the user P1 to select an operation based on the warning message 290M1 and the notification message 290M2.

[0213] 11A, at least one selection key is displayed on the touch screen 290 to allow the user P1 to select an operation along with the operation guide message 290M3. Specifically, a message send key 290K1, an external world display key 290K2, a force quit key 290K3, and a no action key 290K4 are displayed on the touch screen 290.

[0214] 10, when the user P1 touches the message transmission key 290K1, a predetermined message and a message display command are transmitted as usage suspension information in step S709. The predetermined message may be input by the user or may be prepared in advance. For example, as shown in FIG. 12A, the information display device 100 superimposes the predetermined message on the VR screen being displayed on the display 131.

[0215] 10, when the user P1 touches the outside world display key 290K2, an outside world image display command is sent as usage stop information in the processing of step S709. The information display device 100 switches the display on the display 131 from the VR screen to an outside world image acquired by the outer camera 133. Before the screen switching, a message indicating that the screen will be switched in n seconds may be displayed. FIG. 12B shows a screen displayed when an outside world image display command is received from the mobile terminal device 200, and a message notifying the user of the process of switching the VR image to the outside world image is superimposed.

[0216] 10, when the user P1 touches the force quit key 290K3, a standby transition command is sent as usage suspension information in the process of step S709. The information display device 100 transitions from the normal operating state to the standby state. Before transitioning to the standby state, a message indicating that transition to the standby state will occur in n seconds may be displayed.

[0217] Furthermore, if the user P1 touches the no-action key 290K4 in the process of step S708 in FIG. 10, the process of step S709 is not performed.

[0218] According to this embodiment, a user P1, who is different from the user U1 of the information display device 100, can execute a forced suspension process on the information display device 100 from the mobile terminal device 200. This allows, for example, a parent or guardian to stop their child from excessive use of the information display device 100 from the mobile terminal device 200 that they (user P1) operate. In this case, the forced suspension process can be executed by communicatively connecting the mobile terminal device 200 and the information display device 100, so that the information display device 100 can execute the forced suspension process from a position that does not interfere with the movement of user U1. This allows the forced suspension to be executed while avoiding collisions between user U1 and user P1.

[0219] The present invention is not limited to the above-described embodiments, and it is possible to replace part of the configuration of one embodiment with another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. These all fall within the scope of the present invention, and the numerical values, messages, etc. appearing in the text and figures are merely examples, and the use of different ones does not impair the effects of the present invention.

[0220] In addition, some or all of the functions of the invention may be implemented in hardware, for example, by designing an integrated circuit, a general-purpose processor, or an application-specific processor. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. It may also be implemented in software by a microprocessor unit, processor, etc., interpreting and executing an operating program. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together.

[0221] The above-described embodiments include the following inventions: (Supplementary Note 1) An information display device to be worn on a user's head, comprising: a processor, a display, a wearing sensor, and an outer camera, wherein when the information display device is powered on and the wearing sensor detects that the user is wearing the information display device, the processor counts up a cumulative usage time of the information display device, and when the cumulative usage time increases to a predetermined cumulative usage limit threshold, displays an external image of the real world captured by the outer camera on the display, and when the wearing sensor detects that the user has removed the information display device, stops screen display on the display and counts down the cumulative usage time, and when the cumulative usage time decreases to a predetermined restart permission threshold, allows screen display on the display.

[0222] (Supplementary Note 2) An information display system including an information display device and a mobile terminal device communicatively connected to the information display device, wherein the information display device comprises a first processor, a first display, a wearing sensor, and a first communication device for communicating with the mobile terminal device, and the mobile terminal device comprises a second processor and a second communication device for communicating with the information display device, wherein the first processor of the information display device: executes control to transmit, from the first communication device to the mobile terminal device, use start information indicating that use of the information display device has started and wearing status information indicating the wearing status of the information display device detected by the wearing sensor, and, upon receiving display control information of the first display from the mobile terminal device via the first communication device, executes display control of the first display based on the display control information, and the second processor of the mobile terminal device: upon receiving the use start information and the wearing status information via the second communication device, calculates a cumulative use time of the information display device based on the use start information and the wearing status information, and generates the display control information for executing an instruction to stop the first display according to the cumulative use time, an information display system that executes control for transmitting the display control information from the second communication device to the information display device.

[0223] (Supplementary Note 3) A display control method for an information display device worn on a user's head, wherein a processor executes the following steps: when the information display device is powered on, counting up a cumulative usage time of the information display device; when the cumulative usage time increases to a predetermined cumulative usage limit threshold, displaying an external image of the real world on the information display device; when the information display device is removed from the user, stopping screen display of the information display device and counting down the cumulative usage time; and when the cumulative usage time decreases to a predetermined restart permission threshold, allowing screen display of the information display device.

[0224] 1: Information display system, 100: Information display device, 101: Processor, 102: Bus, 103: RAM, 110: Storage, 112: Timekeeping unit, 120: Operation input device, 121: Operation key, 122: Touch sensor, 122L: Left touch sensor, 130: Image processing unit, 131: Display, 131L: Left display, 131R: Right display, 132: Image signal processing unit, 133: Outer camera, 133L: Left outer camera, 133R: Right outer camera, 134: Inner camera, 140: Audio processing unit, 141: Audio output unit, 141L: Left Stereo speakers, 142: audio signal processing unit, 143: microphone, 150: position information acquisition unit, 160: sensor group, 161: gyro sensor, 162: geomagnetic sensor, 163: acceleration sensor, 164: wearing sensor, 165: depth sensor, 165L: left depth sensor, 165R: right depth sensor, 170: communication I / F, 171: LAN I / F, 172: BT I / F, 180: expansion I / F, 200: mobile terminal device, 201: processor, 202: bus, 203: RAM, 210: storage, 211: usage status information DB, 220: operation input device, 221 : Operation keys, 222: Touch panel, 230: Image processing unit, 231: Display, 240: Audio processing unit, 241: Audio output unit, 243: Microphone, 250: Position information acquisition unit, 260: Sensor group, 270: Communication I / F, 271: LAN I / F, 272: BT I / F, 273: Telephone network I / F, 280: Expansion I / F, 290: Touch screen, 290K1: Message transmission key, 290K2: External world display key, 290K3: Forced termination key, 290K4: No action key, 290M1: Warning message, 290M2: Notification message, 290 M3: operation guide message, 300: server, 400: network, 1001: basic operation program, 1002: application program, 1009: data storage area, 1101: basic operation function unit, 1111: wearing status confirmation unit, 1112: timing unit, 1113: display control unit, 1199: temporary storage area, 2001: basic operation program, 2002: application program, 2009: data storage area, 2101: basic operation function unit, 2111: operation status confirmation unit, 2112: timing unit, 2113: operation control unit, 2199: temporary storage area

Claims

1. An information display device worn on the user's head, Processor and The display and The attached sensor and Rear camera and Equipped with, The aforementioned processor, When the power to the information display device is turned on and the mounting sensor detects that the information display device is mounted on the user, the cumulative usage time of the information display device is counted up, and when the cumulative usage time increases to a predetermined cumulative usage limit threshold, the display shows an image of the real world captured by the rear camera. When the attached sensor detects that the information display device has been removed from the user, it stops the display screen and counts down the accumulated usage time. When the accumulated usage time decreases to a predetermined restart permission threshold, it allows the display screen to be displayed again. To accept, Information display device.

2. An information display device according to claim 1, The cumulative usage time is the cumulative value of the time the information display device was used within a predetermined period, or the cumulative value of the time the information display device was used continuously. Information display device.

3. An information display device according to claim 1, It also has a timer, When the processor counts up the cumulative usage time, it uses a value obtained by multiplying the measured value measured by the timer by a predetermined first coefficient. Information display device.

4. An information display device according to claim 3, The processor determines the first coefficient according to the content displayed on the display. Information display device.

5. An information display device according to claim 1, It also has a timer, The processor, when counting down the cumulative usage time, uses a value obtained by multiplying the measured value measured by the timer by a predetermined second coefficient. Information display device.

6. An information display device according to claim 5, The processor acquires biometric information from an external device while the user of the information display device is removing the information display device, and determines the second coefficient according to the biometric information. Information display device.

7. An information display device according to claim 1, The aforementioned processor, When the cumulative usage time increases to a warning start threshold that is shorter than the cumulative usage limit threshold, the external image is superimposed on the virtual reality image currently displayed on the display. Information display device.

8. An information display device according to claim 7, It also has a timer, When the processor counts up the cumulative usage time, it uses a value obtained by multiplying the measured value measured by the timer by a predetermined coefficient. The value of the first coefficient, which is one of the predetermined coefficients and is used from the start of measurement of the cumulative usage time until the warning start threshold, is greater than the value of the third coefficient, which is one of the predetermined coefficients and is used from the warning start threshold until the cumulative usage limit threshold. Information display device.

9. An information display device according to claim 7, The processor, as the cumulative usage time approaches the cumulative usage limit threshold, superimposes the external image more clearly onto the display. Information display device.

10. An information display device according to claim 1, The processor measures and stores the cumulative usage time for each user identification information. Information display device.

11. An information display system including an information display device and a mobile terminal device connected to the information display device via communication, The information display device comprises a first processor, a first display, a wearable sensor, and a first communication device for communicating with a mobile terminal device. The mobile terminal device comprises a second processor and a second communication device for communicating with the information display device. The first processor of the information display device is Control is performed to transmit from the first communication device to the mobile terminal device the start of use information indicating that the use of the information display device has started, and the mounting status information indicating the mounting status of the information display device as detected by the mounting sensor. When display control information for the first display is received from the mobile terminal device via the first communication device, the display control for the first display is executed based on the display control information. The second processor of the mobile terminal device is The start of use information and the installation status information are transmitted via the second communication device. Upon receiving the above-mentioned start-up information and the above-mentioned installation status information, The cumulative usage time of the information display device is calculated, and according to the cumulative usage time The display control information is generated to execute the stop command for the first display. The second communication device executes control to transmit the display control information to the information display device. Information display system.

12. An information display system according to claim 11, The aforementioned mobile terminal device further comprises a second display, The second processor of the mobile terminal device is When the information display device receives the usage start information and the information display device receives the mounting status information indicating that it is mounted on the user, the cumulative usage time is counted up, and when the cumulative usage time increases to a predetermined cumulative usage limit threshold, at least one of the stop instructions is displayed on the second display, and when the user selects one of the stop instructions, the display control information for executing the selected stop instruction is generated. Information display system.

13. The information display system of claim 12, The aforementioned information display device further includes an outward-facing camera, The stop instruction is an instruction to display the external image captured by the out-of-camera on the first display. Information display system.

14. The information display system of claim 12, The stop instruction includes at least one of an instruction to superimpose a message on the first display and an instruction to forcibly terminate the display on the first display. Information display system.

15. A method for controlling the display of an information display device worn on a user's head, The processor, When the power to the information display device is turned on, the cumulative usage time of the information display device is counted up, When the cumulative usage time increases to a predetermined cumulative usage limit threshold, the information display device is used to display an image of the real world outside. When the information display device is removed from the user, the screen display of the information display device is stopped and the accumulated usage time is counted down. When the cumulative usage time decreases to a predetermined restart permission threshold, the following steps are performed: the screen display of the information display device is permitted. A method for controlling the display of an information display device.

Images