Head-mounted display device and method for controlling the same

Abstract

The invention provides a head-mounted display device and a control method thereof, which can reduce misoperation when the head-mounted display device is used in a shaking vehicle. The head-mounted display device includes: an image display unit; a processor that sets a virtual operation panel at a predetermined distance position within the virtual observation space region via the image display unit; and a camera that acquires an image of an operation instruction for operating the virtual operation panel, the processor analyzes the image of the operation instruction acquired by the camera, and when at least one of the virtual operation panel and the operation instruction is moved, the processor determines whether the virtual operation panel and the operation instruction are moved. And a control unit that calculates a relative and three-dimensional shake amount of the operation indicator based on the three-dimensional position of the virtual operation panel at the predetermined distance position, and controls the image display unit on the basis of the relative and three-dimensional shake amount of the operation indicator. The virtual operation panel is three-dimensionally shifted and displayed from a predetermined distance position so as to follow the operation indicator.

Description

[0001] This application is a divisional application of the invention patent application with the application date of March 14, 2018, the application number of 201880091123.7, and the title of "Portable Information Terminal". Technical Field

[0002] The present invention relates to an anti-misoperation technology for a portable information terminal having a touch screen. Background Art

[0003] The popularization of portable information terminals such as smart phones and tablet terminals is in progress. These portable information terminals have touch screens, and users can give desired operation instructions to the portable information terminals by performing touch operations on the touch screens. For example, the user can touch a region on the touch screen where an icon or the like is displayed with a finger or the like to start an application program associated with the icon at the touch position.

[0004] Generally, the display position of an object (such as an icon) on the touch screen is set to be substantially the same as the operation detection area for detecting whether the object has been touched. Thus, when the touch position deviates from the display position (operation detection area) of the object, the touch operation is not judged as an operation instruction for the above object, but becomes a misoperation.

[0005] Regarding this point, Patent Document 1 discloses a technology for reducing the possibility of misoperation when selecting a moving object on a selection screen. Patent Document 1 describes the following content: "For each display object displayed on the screen, a reaction area for accepting coordinate input of an input unit is set to be larger than the display range of the display object according to the moving speed of the display object on the screen."

[0006] Prior Art Documents

[0007] Patent Documents

[0008] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2012-217101 Summary of the Invention

[0009] Technical Problems to be Solved by the Invention

[0010] The above-mentioned Patent Document 1 addresses the case where an object displayed on the screen is moving, but there is also a possibility of accidental operation when the object on the screen is stationary. For example, when a user uses a portable information terminal while riding on a tram or a motor vehicle, etc., there is a possibility of accidental operation due to the shaking of the vehicle. Specifically, consider a case where the user holds the housing of the portable information terminal with the left hand and touches the touch screen with the finger of the right hand (or a stylus held by the right hand, etc.). As the vehicle shakes, the left and right hands of the user will shake, but the degrees of shaking of the two are not necessarily the same, and as a result, a position different from the intended position on the touch screen will be touched. That is, when operating a portable information terminal in a shaking vehicle, even if the touch target is a stationary object on the screen, there is a possibility of accidental operation. Regarding the accidental operation that occurs in such a situation, Patent Document 1 does not take any consideration.

[0011] An object of the present invention is to reduce accidental operations that occur when using a portable information terminal having a touch screen in a shaking vehicle.

[0012] Technical means for solving the problem

[0013] To solve the above problems, a representative portable information terminal of the present invention adopts the following structure, which includes a touch screen having an image display unit and a touch panel unit, and can issue an instruction by using a touch operation on the touch screen, including: a display control unit that causes the image display unit to display an object, where the object can be subjected to a touch operation; an image input unit that acquires an image of an operation indicator for performing a touch operation on the touch screen; and a shake detection unit that analyzes the image of the operation indicator acquired by the image input unit, calculates a relative shake amount of the operation indicator based on the position of the portable information terminal, and based on the relative shake amount of the operation indicator calculated by the shake detection unit, the display control unit causes the object displayed on the image display unit to shift in a manner following the relative shake amount.

[0014] Advantages of the invention

[0015] By adopting the present invention, when using a portable information terminal having a touch screen in a shaking vehicle, accidental operations can be reduced and the usability of the user can be improved. Description of the drawings

[0016] Figure 1A It is an external view showing an example of a portable information terminal.

[0017] Figure 1B It is a diagram showing an example of a main screen displayed on a portable information terminal.

[0018] Figure 2A It is a diagram showing an example of the hardware structure of a portable information terminal.

[0019] Figure 2B This is a diagram showing an example of the software structure of a portable information terminal.

[0020] Figure 3A This is a diagram for explaining the occurrence of incorrect operations on a portable information terminal.

[0021] Figure 3B This is a diagram for explaining the anti-incorrect-operation processing in Embodiment 1.

[0022] Figure 4 This is a structural diagram of the front camera of a portable information terminal.

[0023] Figure 5A This is a flowchart showing the anti-incorrect-operation processing in Embodiment 1.

[0024] Figure 5B This is a diagram for explaining the details of the processing in S106 (judgment of whether there is an operation indicator).

[0025] Figure 6 This is a diagram for explaining the anti-incorrect-operation processing in Embodiment 2.

[0026] Figure 7A This is a diagram for explaining the anti-incorrect-operation processing in Embodiment 3.

[0027] Figure 7B This is a flowchart showing the anti-incorrect-operation processing in Embodiment 3.

[0028] Figure 8A This is a diagram of the confirmation screen for anti-incorrect-operation in Embodiment 4.

[0029] Figure 8B This is a flowchart showing the anti-incorrect-operation processing in Embodiment 4.

[0030] Figure 9A This is a diagram showing the state of using a portable information terminal via an audiovisual device (Embodiment 5).

[0031] Figure 9B This is a diagram for explaining the anti-incorrect-operation processing in Embodiment 5. Detailed implementation manners

[0032] The embodiments of the present invention will be described below with reference to the accompanying drawings. The portable information terminal of this embodiment can be a mobile phone, a smart phone, a tablet terminal, etc. It can also be a head-mounted information terminal such as an HMD (Head Mounted Display). In addition, a smart phone or a tablet terminal can be fixed to an adapter to replace the head-mounted information terminal. In addition to this, as long as it is a digital device with a touch screen, it is an applicable object. For example, it can be a PDA (Personal Digital Assistants), a notebook PC (Personal Computer), an e-reader, a digital still camera, a video camera capable of shooting moving images, a portable game machine, etc. In the following, a smart phone will be mainly used as an example for description. In the following embodiments, the case where the user performs a touch operation on an icon or the like displayed on the touch screen will be taken as an example for description. However, the operation object is not limited to an icon, and can be any object such as text, a graphic, or a map (object).

[0033] Embodiment 1

[0034] [Appearance of Portable Information Terminal]

[0035] Figure 1A FIG. is an external view showing an example of the portable information terminal 100. Here, in the case where the portable information terminal 100 is a smart phone, (a) shows a front (front surface) view, and (b) shows a back (rear surface) view.

[0036] In (a), in addition to the touch screen 180, a working indicator 124, a first image input unit 133, a second image input unit 134, and a mono speaker 142 are provided on the front surface of the portable information terminal 100. The touch screen 180 is composed of a touch panel 123 and an image display unit 131 to be described later, and is a screen for the user to perform a touch operation and display objects such as icons. The working indicator 124 notifies the working state of the portable information terminal 100 by whether an LED (Light Emitting Diode) is lit / flashing. The first image input unit 133 and the second image input unit 134 are "front cameras" that input an image signal by photographing a subject on the front side. Among them, the first image input unit 133 is used to acquire an image of an "operation indicator", which is a finger or the like of a user who performs a touch operation on the portable information terminal 100. The second image input unit 134 is used to acquire a self-portrait image of the user.

[0037] In (b), a touch sensor 122, a third image input unit 135, an auxiliary light emitter / infrared range finder 136, and a stereo speaker 143 are provided on the back surface of the portable information terminal 100. The third image input unit 135 is a "rear camera" that inputs an image signal by photographing a subject on the back side. The auxiliary light emitter / infrared range finder 136 can emit auxiliary light to make up for insufficient light quantity when an image is input from the third image input unit 135. In addition, the auxiliary light emitter / infrared range finder 136 can measure the distance to an object using infrared rays.

[0038] On the top surface of the portable information terminal 100, a power button 121p, which is one of the operation buttons 121 (described later), is provided. On the bottom surface of the portable information terminal 100, a sound input unit 145 and a μ-USB input unit 170u, which is one of the expansion interface units 170 (described later), are provided.

[0039] In addition, the touch sensor 122 may be arranged not on the back surface of the portable information terminal 100 but on the side surface or the lower part of the front surface (a part that does not overlap with the touch screen 180), etc. In addition, the touch panel 123 of the touch screen 180 may also function as the touch sensor 122. In this case, the function of the touch sensor 122 (for example, the fingerprint authentication function) can be executed at any position on the touch screen 180.

[0040] [Main screen of the portable information terminal]

[0041] Figure 1B This is a diagram showing an example of the main screen (main display screen) displayed on the portable information terminal 100. The main screen 180a displayed on the touch screen 180 includes a main function icon display area 180a1, a normal icon display area 180a2, an other information display area 180a3, a control button area 180a4, and a notification area 180a5. The main screen 180a is the basic screen displayed after the portable information terminal 100 is powered on, after the sleep state is released, and when the main screen button is touched during the execution of any application.

[0042] The main function icon display area 180a1 is a display area for icons associated with the main applications frequently used in the portable information terminal 100. The normal icon display area 180a2 is a display area for icons associated with other applications. The other information display area 180a3 is an area for displaying general information such as time information and weather information. The control button area 180a4 is an area for displaying a "back button", a "main screen button", and an "application history button". The notification area 180a5 is an area for notifying information such as the signal strength status and the remaining battery level.

[0043] When the user wants to start a specified application from the main screen 180a, a start instruction can be issued by performing a touch operation on the target icon (object) displayed in the main function icon display area 180a1 or the normal icon display area 180b2.

[0044] [Hardware Structure of Portable Information Terminal]

[0045] Figure 2A This is a diagram showing an example of the hardware structure of the portable information terminal 100. The portable information terminal 100 includes a main control unit 101, a system bus 102, a ROM 103, a RAM 104, a storage unit 110, an operation input unit 120, an image processing unit 130, a sound processing unit 140, a sensor unit 150, a communication unit 160, and an expansion interface unit 170.

[0046] The main control unit 101 is a microprocessor unit that controls the entire portable information terminal 100 according to a specified operation program. The system bus 102 is a data communication path for transmitting and receiving various commands and data between the main control unit 101 and each operation module in the portable information terminal 100.

[0047] The ROM (Read Only Memory) 103 is a memory that stores basic operation programs such as the operating system and other operation programs (application programs, the same below), for example, using an erasable ROM such as an EEPROM (Electrically Erasable Programmable ROM) and a flash ROM. The RAM (Random Access Memory) 104 is a work area when the basic operation programs and other operation programs are executed. The ROM 103 and the RAM 104 can adopt a structure integrated with the main control unit 101. In addition, the ROM 103 may not be Figure 2A structured independently as shown, but instead use a part of the storage area in the storage unit 110.

[0048] The storage unit 110 stores the operation programs and operation setting values of the portable information terminal 100, personal information and authentication information of legal users of the portable information terminal 100, etc. In addition, it can store operation programs downloaded from the network and various data generated by the operation programs. It can also store contents such as moving images, still images, and sounds downloaded from the network. In addition, it can store data such as moving images and still images captured using the imaging function. Part of the area of the storage unit 110 can be used to replace the functions of all or part of the ROM 103. The storage unit 110 needs to be able to retain the stored information even when no power is supplied to the portable information terminal 100 from the outside. Thus, for example, semiconductor element memories such as flash ROM and SSD (Solid State Drive), and disk drives such as HDD (Hard Disc Drive) are used. In addition, each operation program stored in the ROM 103 and the storage unit 110 can be updated and its functions extended by downloading from each server device on the network.

[0049] The operation input unit 120 is an instruction input unit for inputting operation instructions issued to the portable information terminal 100. The operation input unit 120 includes operation keys 121 arranged by button switches, etc., a touch sensor 122 that detects the touch of a user's finger based on changes in electrostatic capacitance, and a touch panel 123 arranged overlappingly on the image display unit 131. In addition, as other operation devices, a keyboard connected to the expansion interface unit 170, other portable terminal devices connected by wired communication or wireless communication, etc. can also be used. Or the operation of the portable information terminal 100 can also be performed by voice input. In addition, the touch sensor 122 has a function of detecting the fingerprint or palm print of the finger touching the sensor unit.

[0050] The image processing unit 130 includes an image display unit 131, an image signal processing unit 132, a first image input unit 133, a second image input unit 134, and a third image input unit 135. The image display unit 131 is a display device such as a liquid crystal panel, etc., which displays the image data processed by the image signal processing unit 132 and provides it to the user of the portable information terminal 100. The image signal processing unit 132 includes a graphic RAM (not shown), and drives the image display unit 131 based on the input image data. In addition, the image signal processing unit 132 performs decoding processing of encoded video signals, format conversion processing, processing of superimposing menus and other OSD (On-Screen Display) signals, etc. as needed. The first image input unit 133, the second image input unit 134, and the third image input unit 135 are imaging units such as a front camera and a rear camera, which use electronic devices such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor to convert the light input through the lens into an electrical signal and acquire the image data of the subject.

[0051] The sound processing unit 140 includes a sound output unit 141, a sound signal processing unit 144, and a sound input unit 145. The sound output unit 141 is a speaker, which provides the sound signal processed by the sound signal processing unit 144 to the user of the portable information terminal 100. The sound output unit 141 is composed of a mono speaker and a stereo speaker. The sound signal processing unit 144 performs decoding processing of encoded sound signals, etc. as needed. The sound input unit 145 is a microphone, which converts the user's voice, etc. into sound data and inputs it.

[0052] The sensor unit 150 is a set of various sensors for detecting the state of the portable information terminal 100. The sensors include a GPS (Global Positioning System) receiver unit 151, a gyro sensor 152, a geomagnetic sensor 153, and an acceleration sensor 154, which detect the position, tilt, orientation, and movement of the portable information terminal 100. In addition to these, it may also include an illuminance sensor, a proximity sensor, a barometric pressure sensor, etc., which detect the surrounding brightness, the proximity condition of surrounding objects, etc.

[0053] The communication unit 160 includes a LAN (Local Area Network) communication unit 161, a telephone network communication unit 162, and an NFC (Near Field Communication) unit 163. The LAN communication unit 161 is connected to a network such as the Internet via an access point or the like, and transmits and receives data to and from each server device on the network. The telephone network communication unit 162 performs telephone communication (calling) and data transmission and reception through wireless communication with a base station or the like of a mobile telephone communication network. The NFC unit 163 performs wireless communication when approaching a corresponding reader / writer. Furthermore, the communication unit 160 may also include a Bluetooth (registered trademark) communication unit and an infrared communication unit.

[0054] The expansion interface unit 170 is a group of interfaces for expanding the functions of the portable information terminal 100, and is composed of an image / sound interface, a USB (Universal Serial Bus) interface, a memory interface, and the like. The image / sound interface is used for input / output of image signals / sound signals with an external image / sound device. The USB interface is connected to a PC, a keyboard, or other USB devices to transmit and receive data. The memory interface is connected to a memory card or other storage media to transmit and receive data.

[0055] Figure 2A The illustrated structural example of the portable information terminal 100 includes a plurality of structures that are not essential for the operation of this embodiment, and the effects of this embodiment are not impaired even if they are not present.

[0056] [Software Structural Example of Portable Information Terminal]

[0057] Figure 2B This is a diagram showing an example of the software structure of the portable information terminal 100. Here, the software structures in the storage unit 110 (or ROM 103, the same below) and the RAM 104 are shown. The basic operation program 1001, the state detection program 1002, the shake detection program 1003, the display control program 1004, the area control program 1005, and other operation programs 1009 are stored in the storage unit 110. The storage unit 110 also includes various information storage areas 1019 for storing other information such as terminal identification information that can identify the portable information terminal 100 and user identification information that can identify the user of the portable information terminal 100.

[0058] The basic operation program 1001 stored in the storage unit 110 is loaded into the RAM 104, and the main control unit 101 executes the loaded basic operation program to constitute the basic operation unit 1101. Similarly, the state detection program 1002, the shake detection program 1003, the display control program 1004, the area control program 1005, and the other operation programs 1009 are respectively loaded into the RAM 104, and the main control unit 101 executes the respective loaded operation programs to constitute the state detection unit 1102, the shake detection unit 1103, the display control unit 1104, the area control unit 1105, and the other operation unit 1109. In addition, the RAM 104 includes a temporary storage area 1200 for temporarily storing data generated when each operation program is executed.

[0059] The following is a description of the simplified operation. For the process in which the main control unit 101 loads the basic operation program 1001 stored in the storage unit 110 into the RAM 104 and executes it to control each operation module, it is denoted as the control of each operation module by the basic operation unit 1101. The same notation is used for the operations of other functional modules implemented by other operation programs.

[0060] The state detection unit 1102 controls the process of detecting the operation state (position information, shaking condition, etc.) of the portable information terminal 100 using the sensor group of the sensor unit 150. The shake detection unit 1103 detects the relative shake amount (jitter amount) of the operation indicator (such as the user's finger or stylus) with respect to the portable information terminal 100 based on the image obtained from the first image input unit 133. The display control unit 1104 performs display control of the image display unit 131 accordingly according to the shake amount of the operation indicator detected by the shake detection unit 1103. The area control unit 1105 controls the operation detection area of the touch panel 123 accordingly according to the shake amount of the operation indicator detected by the shake detection unit 1103.

[0061] The above-mentioned respective operation programs are pre-stored in the storage unit 110 at the time of product shipment. Alternatively, after the product is shipped, each program can be obtained from each server device on the network via the LAN communication unit 161 or the telephone network communication unit 162, etc. In addition, the above-mentioned respective operation programs stored in a memory card, an optical disc, etc. can also be obtained via the expansion interface unit 170, etc.

[0062] [Anti-misoperation processing]

[0063] The occurrence of misoperation and the anti-misoperation processing in the case of a touch operation on the touch screen 180 of the portable information terminal 100 are described below.

[0064] Figure 3AThis is a diagram illustrating the occurrence of an accidental operation on the portable information terminal 100. When the user uses the portable information terminal 100 while riding in a vehicle such as a train or a motor vehicle, an accidental operation may occur due to the shaking of the vehicle. The upper part of the diagram is a view showing the front (operation surface) of the portable information terminal 100 that the user is operating, and the lower part of the diagram is a view showing the lower side surface of the portable information terminal 100.

[0065] For example, consider a case where the user holds the portable information terminal 100 with the left hand 201 and performs a touch operation with the fingers 200 of the right hand. As shown in (a), a screen (home screen) 180a including icons and the like is displayed on the touch screen 180 (image display unit 131), and the user wants to touch the target icon 180i1 with the finger 200. At this time, if the vehicle shakes, as shown in (b), assume that the left hand holding the portable information terminal 100 shakes to the left, for example, while the fingers 200 of the right hand shake to the right, for example. As a result, as shown in (c), the user will touch not the target icon 180i1 but another adjacent icon 180i2, resulting in an accidental operation. At this time, if the direction / size of the shaking of the portable information terminal 100 coincides with the direction / size of the shaking of the fingers 200 of the right hand as the vehicle shakes, no accidental operation will occur. However, in reality, depending on the user's posture and the like, the directions / sizes of the shaking of the two do not coincide in many cases, and accidental operations cannot be avoided.

[0066] Figure 3B This is a diagram illustrating the anti-accidental operation process in the present embodiment. It describes the situation where the anti-accidental operation process is performed in the same operation state as Figure 3A As shown in (a), the first image input unit (front camera) 133 on the front side acquires an image of the finger 200 that is going to touch the target icon 180i1. Next, when the vehicle shakes as shown in (b), the shaking detection unit 1103 analyzes the image of the finger 200 acquired by the first image input unit 133 and calculates the position change of the finger 200 with respect to the housing position of the portable information terminal 100, that is, the relative shaking amount ΔX. Then, as shown in (c), the display control unit 1104 based on the relative shaking amount ΔX of the finger 200 calculated by the shaking detection unit 1103 causes the screen including icons on the touch screen 180 to be offset by a specified amount ΔX' in a manner following the shaking amount, and displays it as 180b. The offset amount ΔX' at this time is equal to the shaking amount ΔX or a value obtained by multiplying by a specified coefficient. Of course, in association with the movement of the display screen, the operation detection areas corresponding to the respective icons and the like on the display screen on the touch panel 123 also move. As a result, even when the vehicle in which the user is riding shakes, the user can accurately touch the target icon 180i1 with the fingers 200 of the right hand.

[0067] Figure 3A and Figure 3BThe description is about the situation where the left hand of the user holding the portable information terminal 100 and the right hand of the user performing the touch operation shake in the left - right direction. However, the same applies when the left and right hands shake in the front - back direction and the tilt direction.

[0068] [Structure of the front - facing camera]

[0069] Figure 4 This is a diagram showing the structure of the two front - facing cameras (the first image input unit 133 and the second image input unit 134) that the portable information terminal 100 has. It is a view observed from the left side of the portable information terminal 100 (that is, the operation surface is at the top in the figure).

[0070] (a) represents the second image input unit 134, which is used as a front - facing camera for self - shooting. The viewing angle of the camera is about 90 degrees, the same as that of a normal smartphone.

[0071] (b) represents the first image input unit 133, which is used as a front - facing camera for shooting an operation indicator (such as the user's finger 200, etc.) for the above - mentioned anti - misoperation processing. In this case, in order to be able to shoot the finger even when the finger 200 is close to the touch screen 180, it is set to have a wider viewing angle (for example, a viewing angle close to 180 degrees) than the second image input unit 134.

[0072] (c) represents another structure of the first image input unit 133. Its viewing angle is about the same as that of the second image input unit 134 (about 90 degrees), but it is installed so that the shooting direction is tilted towards the touch screen 180 side. Thus, it is possible to shoot the finger 200 close to the touch screen 180.

[0073] In addition, the portable information terminal 100 of this embodiment adopts a structure with two front - facing cameras (the first image input unit 133 and the second image input unit 134), but it is also possible to share one camera. In this case, only the Figure 4 first image input unit 133 with a wide - angle viewing angle shown in (b) above is used for both anti - misoperation and self - shooting. However, when used for self - shooting, appropriate image processing such as cropping is to be performed on the obtained captured image.

[0074] [Action flow of misoperation mode processing]

[0075] Figure 5A This is a flowchart showing the anti - misoperation processing in this embodiment. The following processing is mainly controlled by the state detection unit 1102, the shake detection unit 1103, and the display control unit 1104. In addition, whether to perform the following anti - misoperation processing can be selected through user settings, etc.

[0076] First, the state detection unit 1102 obtains the position information, tilt information, motion information, etc. of the portable information terminal 100 from the GPS receiving unit 151, gyro sensor 152, acceleration sensor 154, etc. of the sensor unit 150 (S101). Next, the state detection unit 1102 determines whether the user holding the portable information terminal 100 is riding in a vehicle such as a tram or a motor vehicle and the vehicle is shaking based on the respective information obtained through the processing of S101 (S102). Here, the so-called shaking of the vehicle does not refer to the state where the vehicle simply moves, but means that the speed or acceleration detected by the sensor changes by more than a specified amount over time, resulting in a state where misoperations are likely to occur.

[0077] In the process of S102, if it is determined that the user is riding in the vehicle and the vehicle is shaking (S102: Yes), the process proceeds to the process of S103. If it is determined that the user is not riding in the vehicle or even if the user is riding in the vehicle, the vehicle is not shaking (S102: No), the process proceeds to the process of S109. In this case, instead of performing the anti-misoperation processing of S103 to S108 shown below, normal touch operation processing is performed.

[0078] In the process of S103, the shaking detection unit 1103 activates the first image input unit (operation indicator camera) 133 for photographing the operation indicator (such as the user's finger or stylus) (S103), and obtains an image on the front side of the touch screen 180 (S104). Next, the shaking detection unit 1103 analyzes the captured image obtained from the first image input unit 133 to identify the operation indicator (S105). Among them, in order to improve the recognition process, it is preferable to pre-register (register) the operation indicator to be used. In the process of S106, it is determined whether there is an operation indicator in the captured image. In this determination, even if the operation indicator exists in the captured image, the case where the user does not perform a touch operation needs to be excluded. Specifically, when the operation indicator is separated from the touch screen 180 by more than a specified amount, it is determined that there is no operation indicator. This will be described Figure 5B later.

[0079] According to the determination of S106, if there is an operation indicator (successful recognition) (S106: Yes), the process proceeds to the process of S107. If it is determined in S106 that there is no operation indicator (failure to recognize) (S106: No), the process returns to the process of S104 to obtain the captured image again. In addition, if the state where there is no operation indicator (failure to recognize) continues for more than a specified time (S106: timeout), the process returns to the process of S101. Alternatively, it can also be determined that the user has no intention of operating the portable information terminal 100, and the anti-misoperation process ends.

[0080] In the process of S107, the shake detection unit 1103 analyzes the captured image obtained from the first image input unit 133 and calculates the relative shake amount of the operation indicator based on the position of the housing of the portable information terminal 100 (S107). The shake detection unit 1103 also sends the relative shake amount ΔX of the operation indicator calculated above to the display control unit 1104. Based on the relative shake amount ΔX of the operation indicator received from the shake detection unit 1103, the display control unit 1104 performs a process of offsetting the display screen displayed on the touch screen 180 (image display unit 131) in a manner that follows the shake amount (S108). At this time, the area control unit 1105 and the display control unit 1104 work in conjunction to offset the operation detection area on the touch panel 123 corresponding to each icon on the display screen.

[0081] The offset ΔX' of the display screen can be an amount that is roughly equal to the relative shake amount ΔX of the operation indicator calculated in the process of S107, but can also be an amount obtained by multiplying the above relative shake amount ΔX by a specified coefficient k (≤1) (ΔX'=kΔX). For example, the coefficient k=0.5 can be set to cause the display screen to offset by about 50% of the relative shake amount ΔX. This is because the operation detection area corresponding to each icon on the touch panel 123 has a specified area, so the touch position allows for a slight misalignment. In addition, by making the coefficient k<1, the offset processing of the display screen that occurs too sensitively is eliminated, which has the effect of realizing a screen that is easy for the user to operate.

[0082] The basic operation unit 1101 performs the above-mentioned processing of S101 to S108, and at the same time determines whether the operation indicator has touched the touch screen 180 (touch panel 123) (S109). If the operation indicator has touched the touch screen (S109: Yes), various processes corresponding to the touch operation are performed (S110). For example, if the icon of a specified application is touched, the corresponding application is started. If S109 determines that no touch operation has occurred, the process branches into two cases. If S102 determines that the user is riding in a vehicle (and shaking occurs) (S109: No / Riding), the process returns to S104. If S102 determines that the user is not riding in a vehicle (or the vehicle is not shaking) (S109: No / Other than Riding), the process returns to S101.

[0083] Figure 5BThis is a diagram illustrating the details of the process in S106 (determination of the presence of an operation indicator). A side view of the portable information terminal 100 and the operation indicator (the user's finger 200) is shown here. The captured image obtained by the first image input unit 133 includes the user's finger 200. If the distance between the finger 200 and the touch screen 180 is L, then when the user performs a touch operation, the distance L will necessarily be 0. However, when the distance L is large, it is in a state of preparing for the operation before the touch operation (for example, a state where the target icon for the touch operation has not been determined, or a state of searching for the target icon within the screen). If the above anti-misoperation process is executed in such a state of large distance L during preparation, the display screen (icon) will move following the movement of the finger 200, resulting in the inconvenience that the user cannot select the target icon. Therefore, control is required such that the above anti-misoperation process is not performed in the operation preparation state, in other words, it is determined that there is no operation indicator in the process of S106.

[0084] Specifically, a threshold value L1 is defined for the distance L between the operation indicator and the touch screen 180. When L ≤ L1, it is determined that there is an operation indicator, and when L > L1, it is determined that there is no operation indicator. Additionally, since the operation methods of each user (whether the starting position of the touch operation is close to or far from the touch screen 180) are different, it is preferable that the threshold value L1 can be appropriately set for each user. Thus, by invalidating the anti-misoperation process during the preparation period of the touch operation, no inconvenience will be caused to the user during the touch operation.

[0085] As described above, in the first embodiment, even when a user in a shaking vehicle uses the portable information terminal 100, since the position of the display screen is offset in a manner following the relative shaking amount of the operation indicator, misoperations caused by the displacement of the touch operation position can be prevented.

[0086] Second Embodiment

[0087] In the second embodiment, in order to implement the anti-misoperation process, a process of keeping the position of the display screen unchanged but offsetting the detection area of the touch operation is performed. The basic structure of the portable information terminal 100 is the same as that of the first embodiment. Therefore, the following mainly describes the differences from the first embodiment, and the repeated description of the common parts is omitted.

[0088] Figure 6 This is a diagram illustrating the anti-misoperation process in this embodiment. (a) to (c) correspond to (a) to (c) of the first embodiment. Figure 3B Here, the screen displayed on the touch screen 180 is 180a, and the operation detection areas corresponding to the respective icons on the touch panel 123 are represented by the dashed frames 123a and 123b.

[0089] As shown in (a), the first image input unit (camera for photographing an operation indicator) 133 acquires an image of the finger 200 that is to touch the target icon 180i1. In this state, the operation detection area 123a of each icon coincides with the display position of each icon on the display screen 180a. Next, as shown in (b), when the vehicle shakes, the shake detection unit 1103 analyzes the image of the finger 200 acquired by the first image input unit 133, and calculates the position change of the finger 200, i.e., the relative shake amount ΔX, with respect to the housing position of the portable information terminal 100 as a reference.

[0090] In (c), the screen 180a displayed on the touch screen 180 is kept as it is, and the area control unit 1105 sets the operation detection area corresponding to each of the displayed icons on the touch panel 123 to be offset by a specified amount ΔX' as shown in 123b so as to follow the relative shake amount ΔX of the finger 200 calculated by the shake detection unit 1103.

[0091] For example, the detection area corresponding to the icon 180i1 is set to a position shifted to the right compared to the display position of the icon 180i1 (indicated by diagonal lines). As a result, even when the finger 200 touches a position shifted to the right compared to the display area of the target icon 180i1 due to relative shaking, since the detection area is set to follow the position, it is possible to accurately detect that a touch operation has been performed on the icon 180i1.

[0092] The operation flow of the second embodiment changes Figure 5A a part of the flowchart shown. That is, in the process of S108, the area control unit 1105 performs a process of shifting the operation detection area corresponding to each icon etc. on the display screen on the touch panel 123 by ΔX' based on the relative shake amount ΔX of the operation indicator received from the shake detection unit 1103. At this time, the relationship between the relative shake amount ΔX of the operation indicator and the shift amount ΔX' of the detection area can be set as ΔX' = kΔX (k ≤ 1) using a coefficient k in the same manner as in the case of the first embodiment. Other processes are the same as Figure 5A the flowchart shown.

[0093] As described above, by adopting the second embodiment, even when a user uses the portable information terminal 100 in a shaking vehicle, since the position of the operation detection area is shifted according to the relative shake amount of the operation indicator, it is possible to prevent an erroneous operation caused by a misalignment of the touch operation position.

[0094] Embodiment 3

[0095] In Embodiment 3, in order to implement the anti-misoperation process, the display intervals of the respective icons are enlarged on the display screen. The basic structure of the portable information terminal 100 is the same as that of Embodiment 1, and the following mainly describes the differences from Embodiment 1.

[0096] Figure 7A FIG. is a diagram for explaining the anti-misoperation process in the present embodiment. (a) shows an example of the display screen 180a when the user is not in the vehicle or when the vehicle is not shaking even if the user is in the vehicle. The display control unit 1104 sets the display intervals of the icons in the display screen 180a to the normal intervals (d1 in the horizontal direction of the screen and d2 in the vertical direction). On the other hand, (b) shows an example of the display screen 180c when the user is in the vehicle and the vehicle is shaking. The display control unit 1104 sets the display intervals of the icons in the display screen 180c to wider intervals (d1' in the horizontal direction of the screen and d2' in the vertical direction) (d1' > d1, d2' > d2). Thus, since the icon intervals are enlarged when the vehicle is shaking, even if the position of the user's hand shakes, the misoperation of the touch operation can be reduced.

[0097] Here, in order to implement the anti-misoperation process, the intervals of the icons are enlarged, and in association therewith, the area control unit 1105 performs a process of enlarging the operation detection areas corresponding to the respective icons on the display screen on the touch panel 123, and the effect of further reducing misoperations can be obtained.

[0098] Figure 7B FIG. is a flowchart showing the anti-misoperation process in the present embodiment. The following processes are mainly controlled by the state detection unit 1102 and the display control unit 1104.

[0099] First, the state detection unit 1102 acquires the position information, inclination information, motion information, etc. of the portable information terminal 100 from the sensor unit 150 (S301). Next, the state detection unit 1102 determines whether the user holding the portable information terminal 100 is in the vehicle and the vehicle is shaking based on the respective information acquired in the process of S301 (S302).

[0100] In the process of S302, when it is determined that the user is not in the vehicle or the vehicle is not shaking even if the user is in the vehicle (S302: No), the process proceeds to S303. When it is determined that the user is in the vehicle and the vehicle is shaking (S302: Yes), the process proceeds to S304.

[0101] In the process of S303, the display control unit 1104 displays the screen with the intervals between the icons displayed on the touch screen 180 set to normal intervals (d1, d2). On the other hand, in the process of S304, the screen is displayed with the intervals between the icons displayed on the touch screen 180 set to wider intervals (d1', d2'). In this case, the area control unit 1105 can also expand the operation detection areas on the touch panel 123 corresponding to the respective icons on the display screen.

[0102] The basic operation unit 1101 executes the above processes of S301 to S304 and simultaneously confirms whether the operation indicator has touched the touch screen 180 (touch panel 123) (S305). When the operation indicator has performed a touch operation (S305: Yes), various processes corresponding to the touch operation are executed (S306). When the operation indicator has not performed a touch operation (S305: No), the process returns to S301.

[0103] In the above description, there is only one type of interval (d1', d2') of the icons set when the vehicle shakes, but multiple intervals can also be set according to the magnitude of the vehicle shake.

[0104] As described above, in the case of the third embodiment, when a user riding in a shaking vehicle uses the portable information terminal 100, since the display intervals of the respective icons are enlarged on the display screen, it is possible to prevent misoperations caused by the deviation (misalignment) of the touch operation position.

[0105] Embodiment 4

[0106] In Embodiment 4, in order to implement the anti-misoperation process, when the user performs a touch operation, a confirmation screen for confirming whether the touch operation is correct is displayed. This process can be performed independently or in combination with the processes of the above Embodiments 1 to 3.

[0107] Figure 8A This is a diagram of the confirmation screen for anti-misoperation in this embodiment. When the operation indicator (the user's finger 200) touches the touch screen 180, the display control unit 1104 displays an operation confirmation message 180m and temporarily interrupts the operation to wait for the user's response. The operation confirmation message 180m displays an "OK" button and a "Cancel" button. Among them, the "OK" button is for the user to respond when the operation just performed is correct (normal operation), and the "Cancel" button is for the user to respond when the operation just performed is incorrect (misoperation). The user judges the correctness of the operation based on the position just touched and selects one of the buttons.

[0108] When the user selects the "OK" button, the touch operation just performed is made valid, and various processes corresponding to the touch operation are executed. When the user selects the "Cancel" button, the touch operation just performed is made invalid, and the user's touch operation is accepted again.

[0109] Figure 8B It is a flowchart showing the anti-misoperation process in this embodiment. The following processes are mainly controlled by the state detection unit 1102 and the display control unit 1104.

[0110] First, the state detection unit 1102 acquires the position information, tilt information, motion information, etc. of the portable information terminal 100 from the sensor unit 150 (S401). Next, based on the various information acquired in the process of S401, the state detection unit 1102 determines whether the user holding the portable information terminal 100 is in the vehicle and the vehicle is shaking (S402).

[0111] In the process of S402, when it is determined that the user is not in the vehicle or the vehicle is not shaking even if the user is in the vehicle (S402: No), it proceeds to S403. When it is determined that the user is in the vehicle and the vehicle is shaking (S402: Yes), it proceeds to S405.

[0112] In the process of S403, the basic operation unit 1101 confirms whether the operation indicator has touched the touch screen 180 (touch panel 123). When the operation indicator has touched the touch screen (S403: Yes), various processes corresponding to the touch operation are executed (S404). When the operation indicator has not touched the touch screen (S403: No), it returns to the process of S401.

[0113] In the process of S405, the basic operation unit 1101 confirms whether the operation indicator has touched the touch screen 180. When the operation indicator has touched the touch screen (S405: Yes), a message 180m for operation confirmation ("OK" / "Cancel" button) is displayed to confirm whether the above touch operation is correct (S406). When the operation indicator has not touched the touch screen (S405: No), it returns to the process of S401.

[0114] In the process of S407, the response of the user to the above operation confirmation message 180m is accepted. When the user selects the "OK" button (the above touch operation is a normal operation) (S407: "OK"), various processes corresponding to the above touch operation are executed (S408). When the user selects the "Cancel" button (the above touch operation is a misoperation) (S407: "Cancel"), it returns to the process of S401.

[0115] As described above, in the case where the user uses the portable information terminal 100 while riding in a jolting vehicle, since a confirmation screen for confirming whether the touch operation performed by the user is correct is displayed, in the case of a misoperation, it is possible to immediately shift to a correction operation, improving the usability for the user.

[0116] In addition, in the above-described embodiment, the operation confirmation message 180m is displayed to prevent misoperations. However, instead, the correctness of the touch operation may be determined based on the length of the duration of the touch state. For example, if the duration of the touch state is equal to or longer than a specified time, it is determined to be valid, and if it is less than the specified time, it is determined to be invalid. Alternatively, the user may set the definition of valid / invalid to be the opposite. Further, the above-described operation confirmation message may be displayed only when the touch state continues for a specified time or longer.

[0117] Furthermore, when the user notices that the wrong icon has been touched, before the operation is confirmed, the user can keep the finger in contact with the touch screen surface, slide the finger to the position of the target icon, and perform an operation to reselect the target icon. Then, the above-described operation confirmation message is displayed in this state. Thus, the correction action and the confirmation action can be performed continuously. Among them, when the finger is slid while keeping contact in a state where the touch operation is pending confirmation, the display position of the icon is fixed. When the finger is slid while keeping contact in a state other than the state where the touch operation is pending confirmation, the display position of the icon is moved. By using such a method of distinguishing use, the convenience can be further improved.

[0118] Embodiment 5

[0119] In Embodiment 5, as a usage mode of the portable information terminal 100, an audiovisual device that can be worn on the user's head for VR (Virtual Reality) or the like display will be described. In this case, the anti-misoperation process is also equally effective.

[0120] Figure 9A FIG. is a diagram showing a state in which the portable information terminal 100 is used via the audiovisual device 10. The portable information terminal 100 is inserted into the audiovisual device (adapter) 10 and used by the user 1 wearing it. (a) is a top view of the user 1 wearing the audiovisual device 10 as viewed from directly above, and (b) is a front view of the user 1 wearing the audiovisual device 10 as viewed from the front. By using such an audiovisual device 10, the same functions as those of an HMD (Head Mounted Display) can be achieved. Alternatively, instead of the audiovisual device 10, the user 1 may hold the portable information terminal 100 with both hands or one hand and place the portable information terminal 100 in front of the eyes.

[0121] The audiovisual device 10 is structured to support a portable information terminal 100 such as a smartphone or a tablet terminal by inserting it into the gap portion 20. The user 1 places the audiovisual device 10 with the portable information terminal 100 inserted in front of the eyes and fixes it to the head using a headband or the like. The display screen (image display unit 131) of the portable information terminal 100 can superimpose an artificial object generated using CG (Computer Graphics) or the like on the image data (camera image) input from the rear camera, i.e., the third image input unit 135, to perform VR (Virtual Reality) display, MR (Mixed Reality) display, or the like. Additionally, in the case of an HMD with a transmissive display, AR (Augmented Reality) display or the like that superimposes an artificial object generated using CG or the like on the surrounding scenery can be performed.

[0122] The image displayed on the display screen of the portable information terminal 100 enters the eyes of the user 1 through the lens unit 11 disposed inside the audiovisual device 10. Additionally, the partition wall portion 12 prevents the mixing of images passing through the left and right lens units 11. The above-mentioned gap portion 20, lens unit 11, partition wall portion 12, etc. are internal structures of the audiovisual device 10 and are not usually exposed to the outside. Additionally, preferably, a part of the outer shell of the audiovisual device 10 covers the portable information terminal 100 like a lid, forming a structure that can prevent the portable information terminal 100 from falling and being misaligned even when the head of the user 1 moves up, down, left, or right. Additionally, a window 13 is provided on the outer shell of the audiovisual device 10 to expose the third image input unit 135 on the back of the portable information terminal 100 to the outside.

[0123] Figure 9B This is a diagram for explaining the anti-misoperation processing in this embodiment, showing the state where the user 1 operates the portable information terminal 100. The user 1 views the image data input from the third image input unit 135 of the portable information terminal 100 via the image display unit 131 and can obtain a virtual viewing range 180s. Within the virtual viewing range 180s, a virtual operation panel 180t is arranged at a position in front at a specified distance. The user 1 can issue an operation instruction to the portable information terminal 100 by operating an icon or the like within the virtual operation panel 180t with a physical finger 200.

[0124] For example, consider the case where the user 1 in a vehicle touches an object icon on the virtual operation panel 180t with the finger 200 of the right hand. Due to the shaking of the vehicle, for example, the user's head may shake to the left, and as a result, the virtual operation panel 180t shakes to the left while the finger 200 of the right hand shakes to the right. As a result, the user touches another icon on the virtual operation panel 180t instead of the target icon, causing a misoperation.

[0125] In this case, the same anti-misoperation processing as that in the above-described Embodiment 1 can be applied. The third image input unit (rear camera) 135 of the portable information terminal 100 acquires an image of the finger 200 that is to perform a touch operation on the virtual operation panel 180t. The shake detection unit 1103 analyzes the image of the finger 200 acquired by the third image input unit 135, and calculates the relative shake amount of the finger 200 based on the display position of the virtual operation panel 180t (or the housing position of the portable information terminal 100). Furthermore, the display control unit 1104 performs control based on the calculated relative shake amount ΔX of the finger 200, so as to shift the display position of the virtual operation panel 180t within the virtual viewing range 180s by ΔX'.

[0126] In this case, it is also possible to introduce Figure 5B the determination of whether the operation indicator (finger 200) described in. That is, by measuring the distance from the portable information terminal 100 to the finger 200 using the auxiliary light emitter / infrared rangefinder 136, the distance L from the virtual operation panel 180t to the finger 200 can be calculated. Then, the anti-misoperation processing can be made effective only when the distance L is below the threshold. In addition, in the case of the virtual operation panel 180t of the present embodiment, the display position may be three-dimensionally shifted including the line-of-sight direction. As a result, even when the user uses the portable information terminal 100 via the audiovisual device 10, misoperations caused by vehicle shaking can be prevented.

[0127] The embodiments of the present invention have been described above using Embodiments 1 to 5, but the structure of the technology for implementing the present invention is of course not limited to the above embodiments, and various modification examples can be considered. For example, the case where the icons (objects) displayed on the touch screen are stationary has been described, but the anti-misoperation processing of the present invention can also be applied when the icons are moving, and the same effect can be obtained. In addition, the object is not limited to an icon, and it is also effective in the case where an image such as a map is displayed and the user designates a certain position on the map by a touch operation.

[0128] Between the embodiments, a part of the structure of one embodiment can be replaced with the structure of another embodiment, and the structure of another embodiment can also be added to the structure of one embodiment. All of these fall within the scope of the present invention. In addition, the numerical values, messages, etc. shown in the text and the drawings are only examples, and even if they are different, the effects of the present invention are not impaired.

[0129] Functions of the present invention described above, etc. can be implemented in part or in whole by hardware through, for example, integrated circuit design. Additionally, they can be implemented by software by a microprocessor unit, etc. interpreting and executing programs for implementing respective functions, etc. It is also possible to use both hardware and software simultaneously. The above software can be pre-stored in the ROM 103 and the storage unit 110, etc. of the portable information terminal 100 at the time of product shipment. It can also be obtained from various server devices, etc. on the Internet after the product is shipped. Additionally, the above software provided on a memory card, an optical disc, etc. can be obtained.

[0130] In addition, the control lines and information lines shown in the figure only represent the parts necessary for explanation and do not necessarily represent all the control lines and information lines on the product. In fact, it can be considered that almost all the structures are interconnected.

[0131] Description of Reference Numerals

[0132] 10... Audio-visual device, 100... Portable information terminal, 123... Touch panel, 123a, b... Operation detection areas, 131... Image display unit, 133... First image input unit (camera for photographing an operation indicator), 134... Second image input unit (front camera), 135... Third image input unit (rear camera), 150... Sensor unit, 180... Touch screen, 180a, b, c... Display screens, 180i... Icons, 180m... Messages for operation confirmation, 180t... Virtual operation panel, 200... User's finger (operation indicator), 1101... Basic operation unit, 1102... State detection unit, 1103... Shake detection unit, 1104... Display control unit, 1105... Area control unit.

Claims

1. A head-mounted display device, characterized in that, Comprising: An image display unit; A processor that sets a virtual operation panel at a specified distance position within a virtual viewing space region via the image display unit; And A camera that acquires an image of an operation indicator for operating the virtual operation panel, wherein, in the processor, Analyzes the image of the operation indicator acquired by the camera, When at least one of the virtual operation panel and the operation indicator has moved, calculates a relative and three-dimensional amount of wobbling of the operation indicator based on the three-dimensional position of the virtual operation panel at the specified distance position, Based on the relative and three-dimensional amount of wobbling of the operation indicator, controls the image display unit to three-dimensionally offset and display the virtual operation panel from the specified distance position in a manner that follows the operation indicator.

2. The head-mounted display device according to claim 1, wherein: In the processor, Measures the distance between the operation indicator and the virtual operation panel according to the image of the operation indicator, When the distance is equal to or greater than a threshold value, controls to display without causing the virtual operation panel to three-dimensionally shift.

3. The head-mounted display device according to claim 1, wherein: Includes a sensor for detecting the state of the head-mounted display device, In the processor, when an operation on the virtual operation panel is performed while the sensor determines that the head-mounted display device is shaking, controls the image display unit to display a confirmation screen for confirming whether the operation is correct.

4. The head-mounted display device according to claim 3, wherein: The confirmation screen is superimposed and displayed.

5. The head-mounted display device according to claim 1, wherein: The relative and three-dimensional amount of wobbling of the operation indicator with respect to the three-dimensional position of the virtual operation panel includes the amount of wobbling of the virtual operation panel in the line-of-sight direction.

6. The head-mounted display device according to claim 1, wherein: Includes a sensor for detecting the state of the head-mounted display device, In the processor, when the sensor determines that the state of the head-mounted display device is not in a vehicle or, even if in a vehicle, there is no shaking, controls to display without causing the virtual operation panel to three-dimensionally shift.

7. A control method for a head-mounted display device including an image display unit, characterized in that, Comprising: Sets a virtual operation panel at a specified distance position within a virtual viewing space region via the image display unit; Acquires an image of an operation indicator for operating the virtual operation panel; Analyzes the image of the operation indicator, and when at least one of the virtual operation panel and the operation indicator has moved, calculates a relative and three-dimensional amount of wobbling of the operation indicator based on the three-dimensional position of the virtual operation panel at the specified distance position; And Based on the relative and three-dimensional amount of wobbling of the operation indicator, controls the image display unit to three-dimensionally shift and display the virtual operation panel from the specified distance position in a manner that follows the operation indicator.

8. The control method according to claim 7, wherein: measuring the distance between the operation indicator and the virtual operation panel according to the image of the operation indicator, When the distance is equal to or greater than a threshold value, control is performed so as to display the virtual operation panel without causing three-dimensional deviation.

9. The control method according to claim 7, wherein: detecting a status of the head mounted display device, When an operation on the virtual operation panel is performed while it is determined that the head mounted display device is shaking, the image display unit is controlled to display a confirmation screen for confirming whether the operation is correct.

10. The control method according to claim 9, wherein: The confirmation screen is displayed in a superimposed manner.

11. The control method according to claim 7, wherein: The relative three-dimensional shaking amount of the operation indicator with respect to the three-dimensional position of the virtual operation panel includes the shaking amount of the virtual operation panel in the line of sight direction.

12. The control method according to claim 7, wherein: detecting a status of the head mounted display device, When it is determined that the head mounted display device is not in a vehicle or is not shaking even if the user is in a vehicle, control is performed so as to display the virtual operation panel without causing three-dimensional deviation.

Images