Portable information terminal

Abstract

The present invention provides a portable information terminal, including a touch screen having an image display unit and a touch panel unit, which can issue instructions by using a touch operation on the touch screen, and includes: a display control unit that causes the image display unit to display an object, wherein the object can be subjected to a touch operation; an image input unit that obtains an image of an operation indicator that performs a touch operation on the touch screen; and a shake detection unit that analyzes the image of the operation indicator obtained by the image input unit and calculates the relative shake amount of the operation indicator based on the position of the portable information terminal. Based on the relative shake amount of the operation indicator calculated by the shake detection unit, the display control unit causes the object displayed on the image display unit to shift in a manner that follows the relative shake amount. In this way, it is possible to reduce erroneous operations caused by misalignment of the touch position when using the portable information terminal in a shaking vehicle.

Description

Technical Field

[0001] The invention relates to a misoperation prevention technology for a portable information terminal with a touch screen. Background Art

[0002] The popularity of portable information terminals such as smart phones and tablet terminals is advancing. These portable information terminals have touch screens, and users can issue desired operation instructions to the portable information terminals by touching the touch screen. For example, the user can touch an area on the touch screen where icons are displayed with a finger to start an application associated with the icon at the touched position.

[0003] Generally speaking, the display position of an object such as an icon on a touch screen is set to be roughly consistent with an operation detection area that detects whether a touch operation has been performed on the object. Therefore, if the touch position deviates from the display position of the object (operation detection area), the touch operation will not be judged as an operation instruction for the object, but will become an erroneous operation.

[0004] In this regard, Patent Document 1 discloses a technology for reducing the possibility of erroneous operation when selecting a moving object on the screen. The above Patent Document 1 records the following content: "For each display object displayed on the screen, according to the moving speed of the display object on the screen, the reaction area for receiving the coordinate input of the input unit is set to be larger than the display range of the display object."

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2012-217101 Summary of the invention

[0008] Technical problem to be solved by the invention

[0009] The above-mentioned patent document 1 is aimed at the situation where the object displayed on the screen is moving, but there is also the possibility of misoperation when the object on the screen is stationary. For example, when a user uses a portable information terminal while riding on a train or a motor vehicle, there is the possibility of misoperation caused by the shaking of the vehicle. Specifically, imagine a situation where the user holds the housing of the portable information terminal with his left hand and touches the touch screen with the fingers of his right hand (or a stylus held in his right hand, etc.). As the vehicle shakes, the user's left and right hands will shake, but the degree of shaking of the two is not necessarily the same, resulting in touching a position on the touch screen that is different from the intended position. That is, when operating a portable information terminal in a shaking vehicle, there is the possibility of misoperation even if the touch target is a stationary object on the screen. The above-mentioned patent document 1 does not take into account any misoperation that occurs under such circumstances.

[0010] An object of the present invention is to reduce erroneous operations that occur when a portable information terminal having a touch screen is used in a shaking vehicle.

[0011] Technical means of solving problems

[0012] To solve the above-mentioned problem, a representative portable information terminal of the present invention adopts the following structure, which includes a touch screen having an image display unit and a touch panel unit, and can issue instructions by using a touch operation on the touch screen, including: a display control unit that causes the image display unit to display an object, wherein the object can be touch-operated; an image input unit that obtains an image of an operation indicator that performs a touch operation on the touch screen; and a shake detection unit, which analyzes the image of the operation indicator obtained by the image input unit, calculates the relative shake amount of the operation indicator based on the position of the portable information terminal, and based on the relative shake amount of the operation indicator calculated by the shake detection unit, the display control unit causes the object displayed on the image display unit to shift in a manner that follows the relative shake amount.

[0013] Effects of the Invention

[0014] By adopting the present invention, when a portable information terminal with a touch screen is used in a shaking vehicle, erroneous operations can be reduced and the usability for users can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A This is a diagram showing the appearance of an example of a portable information terminal.

[0016] Figure 1B This is a diagram showing an example of a home screen displayed on a portable information terminal.

[0017] Figure 2A This is a diagram showing an example of the hardware configuration of a portable information terminal.

[0018] Figure 2B This is a diagram showing an example of the software structure of a portable information terminal.

[0019] Figure 3A This is a diagram for explaining the occurrence of erroneous operation on a portable information terminal.

[0020] Figure 3B This is a diagram for explaining the erroneous operation prevention process in the first embodiment.

[0021] Figure 4 It is a structural diagram of a front camera of a portable information terminal.

[0022] Figure 5A 1 is a flowchart showing the erroneous operation prevention process in the first embodiment.

[0023] Figure 5B This is a diagram for explaining the details of the process of S106 (determination of the presence or absence of the operation indicator).

[0024] Figure 6 This is a diagram for explaining the erroneous operation prevention process in the second embodiment.

[0025] Fig. 7A This is a diagram for explaining the erroneous operation prevention process in the third embodiment.

[0026] Figure 7B This is a flowchart showing the erroneous operation prevention process in the third embodiment.

[0027] Fig. 8A This is a diagram of a confirmation screen for preventing erroneous operation in Example 4.

[0028] Figure 8B It is a flowchart showing the error operation prevention process in the fourth embodiment.

[0029] Fig.9A This is a diagram showing a state in which a portable information terminal is used via an audio-visual device (Example 5).

[0030] Fig. 9B This is a diagram for explaining the erroneous operation prevention process in the fifth embodiment. DETAILED DESCRIPTION

[0031] The following is an explanation of the embodiments of the present invention using the accompanying drawings. The portable information terminal of this embodiment may be a mobile phone, a smart phone, a tablet terminal, etc. It may also be a head-mounted information terminal such as an HMD (Head Mounted Display). Moreover, the head-mounted information terminal may be replaced by fixing a smart phone or a tablet terminal on an adapter. In addition, any digital device with a touch screen is applicable, such as a PDA (Personal Digital Assistants), a notebook PC (Personal Computer), an e-book reader, a digital still camera, a video camera capable of shooting dynamic images, a portable game console, etc. The following mainly uses a smart phone as an example for explanation. In the following embodiments, the user performs a touch operation on an icon displayed on the touch screen as an example for explanation, but the operation object is not limited to an icon, and may be any object such as text, graphics, or a map.

[0032] Example 1

[0033] [Appearance of portable information terminal]

[0034] Figure 1A 1 and 2 are external views showing an example of the portable information terminal 100. Here, when the portable information terminal 100 is a smart phone, (a) shows a front view, and (b) shows a back view.

[0035] In (a), in addition to the touch screen 180, the front of the portable information terminal 100 is also provided with an operation indicator 124, a first image input unit 133, a second image input unit 134 and a monaural speaker 142. The touch screen 180 is composed of a touch panel 123 and an image display unit 131 described later, and is a screen for the user to perform touch operations and display objects such as icons. The operation indicator 124 notifies the operating status of the portable information terminal 100 by whether the LED (Light Emitting Diode) is lit / flashing. The first image input unit 133 and the second image input unit 134 are "front cameras" that input image signals by photographing the subject on the front side. Among them, the first image input unit 133 is used to obtain an image of an "operation indicator", which is a finger of a user who performs a touch operation on the portable information terminal 100. The second image input unit 134 is used to obtain a selfie image of the user.

[0036] In (b), a touch sensor 122, a third image input unit 135, an auxiliary light emission / infrared rangefinder 136, and a stereo speaker 143 are provided on the back of the portable information terminal 100. The third image input unit 135 is a "rear camera" that inputs an image signal by photographing a subject on the back side. The auxiliary light emission / infrared rangefinder 136 can emit auxiliary light to compensate for insufficient light when an image is input from the third image input unit 135. In addition, the auxiliary light emission / infrared rangefinder 136 can measure the distance to the object using infrared rays.

[0037] A power button 121p as one of the operation buttons 121 (described later) is provided on the top surface of the portable information terminal 100. A sound input unit 145 and a μ-USB input unit 170u as one of the expansion interface units 170 (described later) are provided on the bottom surface of the portable information terminal 100.

[0038] In addition, the touch sensor 122 may not be arranged on the back of the portable information terminal 100, but may be arranged on the side or the lower part of the front (a part that does not overlap with the touch screen 180), etc. In addition, the touch panel 123 of the touch screen 180 may also have the function of the touch sensor 122. In this case, the function of the touch sensor 122 (for example, the fingerprint authentication function) can be executed at any position on the touch screen 180.

[0039] [Home screen of portable information terminal]

[0040] Figure 1B 1 is a diagram showing an example of a home screen (main display screen) displayed on the portable information terminal 100. The home screen 180a displayed on the touch screen 180 includes a main function icon display area 180a1, a normal icon display area 180a2, an other information display area 180a3, a control button area 180a4, and a notification area 180a5. The home screen 180a is a basic screen that is displayed when the portable information terminal 100 is powered on, the sleep state is released, or when the home button is touched during the execution of any application.

[0041] The main function icon display area 180a1 is a display area for icons associated with the main applications frequently used in the portable information terminal 100. The general icon display area 180a2 is a display area for icons associated with other applications. The other information display area 180a3 is an area for displaying general information such as time information and weather information. The control button area 180a4 is an area for displaying a "back button", a "home screen button", and an "application history button". The notification area 180a5 is an area for notifying information such as the signal strength status and the remaining battery amount.

[0042] When the user wishes to activate a predetermined application from the home screen 180 a , the user can issue an activation instruction by touching a target icon (object) displayed in the main function icon display area 180 a 1 or the normal icon display area 180 b 2 .

[0043] [Hardware structure of portable information terminal]

[0044] Figure 2A 1 is a diagram showing an example of the hardware configuration of the portable information terminal 100. The portable information terminal 100 includes a main control unit 101, a system bus 102, a ROM 103, a RAM 104, a storage unit 110, an operation input unit 120, an image processing unit 130, a sound processing unit 140, a sensor unit 150, a communication unit 160, and an expansion interface unit 170.

[0045] The main control unit 101 is a microprocessor unit that controls the entire portable information terminal 100 according to a predetermined operation program. The system bus 102 is a data communication path for transmitting and receiving various commands and data between the main control unit 101 and each operation module in the portable information terminal 100.

[0046] ROM (Read Only Memory) 103 is a memory that stores basic operating programs such as the operating system and other operating programs (application programs, the same below). For example, an erasable ROM such as EEPROM (Electrically Erasable Programmable ROM) and flash ROM is used. RAM (Random Access Memory) 104 is a work area when the basic operating programs and other operating programs are executed. ROM 103 and RAM 104 can be formed integrally with the main control unit 101. In addition, ROM 103 may not be Figure 2A Instead of adopting an independent structure as shown, a part of the storage area in the storage unit 110 is used.

[0047] The storage unit 110 stores the action program and action setting value of the portable information terminal 100, the personal information and authentication information of the legal user of the portable information terminal 100, etc. In addition, it is possible to store the action program downloaded from the network and various data generated by the action program, etc. It is also possible to store content such as dynamic images, still images and sounds downloaded from the network. In addition, it is possible to store data such as dynamic images and still images taken using the camera function. The function of all or part of ROM103 can be replaced by a part of the area of ​​the storage unit 110. The storage unit 110 needs to be able to maintain the stored information even when the portable information terminal 100 is not powered from the outside. Thus, for example, semiconductor element memories such as flash ROM and SSD (Solid State Drive), disk drives such as HDD (Hard Disc Drive) and other devices are used. In addition, the action programs stored in ROM103 and the storage unit 110 can be updated and expanded by downloading from various server devices on the network.

[0048] The operation input unit 120 is an instruction input unit for inputting operation instructions issued to the portable information terminal 100. The operation input unit 120 includes an operation key 121 arranged by a push button switch, etc., a touch sensor 122 that detects the touch of a user's finger based on a change in electrostatic capacitance, and a touch panel 123 arranged in an overlapping manner on the image display unit 131. In addition, as other operating devices, a keyboard connected to the expansion interface unit 170, other portable terminal devices connected by wired communication or wireless communication, etc. can also be used. Alternatively, the portable information terminal 100 can also be operated by voice input. In addition, the touch sensor 122 has a function of detecting a fingerprint or palm print of a finger touching the sensor unit.

[0049] The image processing unit 130 includes an image display unit 131, an image signal processing unit 132, a first image input unit 133, a second image input unit 134, and a third image input unit 135. The image display unit 131 is a display device such as a liquid crystal panel, which displays the image data processed by the image signal processing unit 132 and provides it to the user of the portable information terminal 100. The image signal processing unit 132 includes a graphic RAM (not shown) and drives the image display unit 131 based on the input image data. In addition, the image signal processing unit 132 performs decoding processing of the encoded image signal, format conversion processing, superimposition menu and other OSD (OnScreen Display) signal processing, etc. as needed. The first image input unit 133, the second image input unit 134, and the third image input unit 135 are shooting units such as a front camera and a rear camera, and use electronic devices such as CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensors to convert light input through the lens into electrical signals to obtain image data of the subject.

[0050] The sound processing unit 140 includes a sound output unit 141, a sound signal processing unit 144, and a sound input unit 145. The sound output unit 141 is a speaker, and provides the user of the portable information terminal 100 with the sound signal processed by the sound signal processing unit 144. The sound output unit 141 is composed of a monaural speaker and a stereo speaker. The sound signal processing unit 144 performs decoding processing of the encoded sound signal as needed. The sound input unit 145 is a microphone, and converts the user's voice into sound data for input.

[0051] The sensor unit 150 is a group of various sensors for detecting the state of the portable information terminal 100. The sensors include a GPS (Global Positioning System) receiving unit 151, a gyro sensor 152, a geomagnetic sensor 153, and an acceleration sensor 154, which detect the position, tilt, orientation, and movement of the portable information terminal 100. In addition, an illumination sensor, a proximity sensor, an air pressure sensor, etc. may also be included to detect the surrounding brightness, the proximity of surrounding objects, etc.

[0052] The communication unit 160 includes a LAN (Local Area Network) communication unit 161, a telephone network communication unit 162, and an NFC (Near Field Communication) unit 163. The LAN communication unit 161 is connected to a network such as the Internet via an access point, etc., and transmits and receives data with each server device on the network. The telephone network communication unit 162 performs telephone communication (calls) and data transmission and reception through wireless communication with a base station of a mobile phone communication network, etc. The NFC unit 163 performs wireless communication when it is close to a corresponding reader / writer. Furthermore, the communication unit 160 may also include a Bluetooth (registered trademark) communication unit and an infrared communication unit.

[0053] The expansion interface unit 170 is an interface group for expanding the functions of the portable information terminal 100, and is composed of an image / audio interface, a USB (Universal Serial Bus) interface, a memory interface, etc. The image / audio interface is used to input / output image signals / audio signals with external image / audio devices. The USB interface is connected to a PC or a keyboard or other USB devices to send and receive data. The memory interface is connected to a memory card or other storage media to send and receive data.

[0054] Figure 2A The illustrated configuration example of the portable information terminal 100 includes a plurality of configurations that are not essential for the operation of the present embodiment, and the effects of the present embodiment are not impaired even if these configurations are not included.

[0055] [Software configuration example of a portable information terminal]

[0056] Figure 2B 1 is a diagram showing an example of the software structure of the portable information terminal 100. The structure of the software in the storage unit 110 (or ROM 103, the same below) and the RAM 104 is shown here. The storage unit 110 stores a basic action program 1001, a state detection program 1002, a shake detection program 1003, a display control program 1004, an area control program 1005, and other action programs 1009. The storage unit 110 also includes a various information storage area 1019 for storing other information such as terminal identification information that can identify the portable information terminal 100, user identification information that can identify the user of the portable information terminal 100, etc.

[0057] The basic action program 1001 stored in the storage unit 110 is loaded into the RAM 104, and the main control unit 101 executes the loaded basic action program to form the basic action unit 1101. Similarly, the state detection program 1002, the shake detection program 1003, the display control program 1004, the area control program 1005, and the other action program 1009 are loaded into the RAM 104, respectively, and the main control unit 101 executes the loaded action programs to form the state detection unit 1102, the shake detection unit 1103, the display control unit 1104, the area control unit 1105, and the other action unit 1109. In addition, the RAM 104 includes a temporary storage area 1200 for temporarily storing data generated when each action program is executed.

[0058] In order to simplify the description of the operation, the main control unit 101 loads the basic operation program 1001 stored in the storage unit 110 into the RAM 104 and executes the process of controlling each operation module, which is described as the basic operation unit 1101 controlling each operation module. The operation of other functional modules implemented by other operation programs is described in the same way.

[0059] The state detection unit 1102 controls the process of detecting the action state (position information, shaking condition, etc.) of the portable information terminal 100 using the sensor group of the sensor unit 150. The shake detection unit 1103 detects the relative shake amount (shake amount) of the operation indicator (the user's finger or stylus, etc.) with respect to the portable information terminal 100 based on the image obtained from the first image input unit 133. The display control unit 1104 controls the display of the image display unit 131 according to the shake amount of the operation indicator detected by the shake detection unit 1103. The area control unit 1105 controls the operation detection area of ​​the touch panel 123 according to the shake amount of the operation indicator detected by the shake detection unit 1103.

[0060] The above-mentioned operation programs are stored in the storage unit 110 before the product is shipped. Alternatively, the above-mentioned operation programs can be obtained from the server devices on the network through the LAN communication unit 161 or the telephone network communication unit 162 after the product is shipped. In addition, the above-mentioned operation programs stored in a memory card or an optical disk can be obtained through the expansion interface unit 170.

[0061] [Preventing misoperation]

[0062] Next, the occurrence of erroneous operation and erroneous operation prevention processing when a touch operation is performed on the touch screen 180 of the portable information terminal 100 will be described.

[0063] Figure 3A1 is a diagram for explaining the occurrence of erroneous operation on the portable information terminal 100. When the user uses the portable information terminal 100 while riding in a vehicle such as a train or a motor vehicle, erroneous operation may occur due to the shaking of the vehicle. The upper part of the figure is a diagram showing the front (operation surface) of the portable information terminal 100 being operated by the user, and the lower part of the figure is a diagram showing the lower side of the portable information terminal 100.

[0064] For example, it is assumed that the user holds the portable information terminal 100 with the left hand 201 and performs a touch operation with the finger 200 of the right hand. As shown in (a), the screen (home screen) 180a including icons and the like is displayed on the touch screen 180 (image display unit 131), and the user wants to touch the target icon 180i1 with the finger 200. At this time, if the vehicle shakes, as shown in (b), it is assumed that the left hand holding the portable information terminal 100 shakes, for example, to the left, and the finger 200 of the right hand shakes, for example, to the right. As a result, as shown in (c), the user touches not the target icon 180i1, but other icons 180i2 adjacent to it, and an erroneous operation occurs. At this time, as the vehicle shakes, if the direction / size of the shaking of the portable information terminal 100 is consistent with the direction / size of the shaking of the finger 200 of the right hand, no erroneous operation will occur. However, in reality, depending on the user's posture, etc., there are many cases where the direction / size of the shaking of the two is inconsistent, and erroneous operation cannot be avoided.

[0065] Figure 3B FIG. 1 is a diagram illustrating the error-prevention process in this embodiment. Figure 3A The case where the anti-misoperation processing is performed under the same operating state. As shown in (a), the first image input unit (front camera) 133 located on the front side obtains the image of the finger 200 that is about to touch the target icon 180i1. Next, when the vehicle shakes as shown in (b), the shaking detection unit 1103 analyzes the image of the finger 200 obtained by the first image input unit 133, and calculates the position change of the finger 200 based on the position of the shell of the portable information terminal 100, that is, the relative shaking amount ΔX. Then, as shown in (c), the display control unit 1104 offsets the screen containing the icon class displayed on the touch screen 180 by a specified amount ΔX' based on the relative shaking amount ΔX of the finger 200 calculated by the shaking detection unit 1103 in a manner that follows the shaking amount, and displays it as 180b. The offset amount ΔX' at this time is equal to the shaking amount ΔX or an amount obtained by multiplying it by a specified coefficient. Of course, in conjunction with the movement of the display screen, the operation detection area corresponding to each icon on the display screen is also moved on the touch panel 123. As a result, the user can accurately touch the target icon 180i1 with the finger 200 of the right hand even when the vehicle is shaking.

[0066] Figure 3A and Figure 3BThe description is given of the case where the left hand of the user holding the portable information terminal 100 and the right hand of the user performing the touch operation are shaken in the left-right direction, but the same is true when the left and right hands are shaken in the front-back direction and the oblique direction.

[0067] [Structure of the front camera]

[0068] Figure 4 The diagram is a diagram for explaining the structure of two front cameras (first image input unit 133 and second image input unit 134) included in the portable information terminal 100, and is a diagram viewed from the left side of the portable information terminal 100 (ie, the operation surface is the upper side in the diagram).

[0069] (a) shows the second image input unit 134, which is used as a front camera for self-portraits. The viewing angle of the camera is about 90 degrees, which is the same as that of a normal smartphone.

[0070] (b) shows the first image input unit 133, which is used as a front camera to capture the operation indicator (the user's finger 200, etc.) for the above-mentioned error operation prevention process. In this case, in order to capture the finger even when the finger 200 is close to the touch screen 180, it is set to have a wider viewing angle (e.g., a viewing angle of nearly 180 degrees) than the second image input unit 134.

[0071] (c) shows another structure of the first image input unit 133. Its viewing angle is the same as that of the second image input unit 134 (about 90 degrees), but it is installed so that the shooting direction is inclined toward the touch screen 180. In this way, the finger 200 approaching the touch screen 180 can be photographed.

[0072] In addition, the portable information terminal 100 of this embodiment has a structure with two front cameras (the first image input unit 133 and the second image input unit 134), but it can also share one camera. In this case, only one camera is used. Figure 4 The first image input unit 133 with a wide-angle view shown in (b) is used for two purposes: preventing erroneous operation and taking selfies. However, when used for taking selfies, the obtained captured image needs to be appropriately processed by cropping or other image processing.

[0073] [Processing flow of incorrect operation]

[0074] Figure 5A 1104. The following processing is mainly controlled by the state detection unit 1102, the shaking detection unit 1103 and the display control unit 1104. In addition, whether to execute the following wrong operation prevention processing can be selected by the user setting or the like.

[0075] First, the state detection unit 1102 obtains the position information, tilt information, motion information, etc. of the portable information terminal 100 from the GPS receiving unit 151, gyro sensor 152, accelerometer sensor 154, etc. of the sensor unit 150 (S101). Next, the state detection unit 1102 determines whether the user holding the portable information terminal 100 is riding in a vehicle such as a train or a motor vehicle, and whether the vehicle is shaking (S102) based on the information obtained through the processing of S101. The shaking of the vehicle here does not refer to the state of the vehicle simply moving, but refers to the state in which the speed or acceleration detected by the sensor changes by more than a specified amount over time, resulting in a state that is prone to erroneous operation.

[0076] In the process of S102, if it is determined that the user is in the vehicle and the vehicle is shaking (S102: Yes), the process proceeds to S103. If it is determined that the user is not in the vehicle, or even if the user is in the vehicle, the vehicle is not shaking (S102: No), the process proceeds to S109. In this case, the error operation prevention process of S103 to S108 shown below is not performed, and the normal touch operation process is performed.

[0077] In the process of S103, the shake detection unit 1103 starts the first image input unit (operation indicator shooting camera) 133 (S103) for shooting the operation indicator (user's finger or stylus, etc.), and obtains the image of the front side of the touch screen 180 (S104). Then, the shake detection unit 1103 analyzes the captured image obtained from the first image input unit 133 and identifies the operation indicator (S105). In order to improve the recognition process, it is preferred to pre-register (register) the operation indicator used. In the process of S106, it is determined whether there is an operation indicator in the captured image. In this judgment, even if the operation indicator exists in the captured image, it is necessary to exclude the situation where the user does not perform a touch operation. Specifically, when the operation indicator leaves the touch screen 180 by more than a specified amount, it is determined that there is no operation indicator. This will be used Figure 5B To be described later.

[0078] According to the judgment of S106, if there is an operation indicator (successful recognition) (S106: Yes), proceed to the processing of S107. If S106 judges that there is no operation indicator (failed to recognize) (S106: No), return to the processing of S104 and obtain the captured image again. In addition, if the state of no operation indicator (failed to recognize) continues for more than a predetermined time (S106: Timeout), return to the processing of S101. Alternatively, it can also be determined that the user has no intention to operate the portable information terminal 100, and the error-prevention processing is terminated.

[0079] In the process of S107, the shake detection unit 1103 analyzes the captured image obtained from the first image input unit 133, and calculates the relative shake amount of the operating indicator based on the position of the housing of the portable information terminal 100 (S107). The shake detection unit 1103 also sends the relative shake amount ΔX of the operating indicator calculated above to the display control unit 1104. Based on the relative shake amount ΔX of the operating indicator received from the shake detection unit 1103, the display control unit 1104 performs a process of offsetting the display screen displayed on the touch screen 180 (image display unit 131) in a manner that follows the shake amount (S108). At this time, the area control unit 1105 works in conjunction with the display control unit 1104 to offset the operation detection area on the touch panel 123 corresponding to each icon on the display screen.

[0080] The offset ΔX' of the display screen can be an amount that is roughly equal to the relative shake amount ΔX of the operation indicator calculated in the process of S107, but it can also be an amount obtained by multiplying the above relative shake amount ΔX by a specified coefficient k (≤1) (ΔX'=kΔX). For example, the coefficient k=0.5 can be set to cause the display screen to shift by about 50% of the relative shake amount ΔX. This is because the operation detection area corresponding to each icon on the touch panel 123 has a specified area, so a slight misalignment of the touch position is allowed. In addition, by making the coefficient k<1 to eliminate the offset processing of the display screen that occurs too sensitively, it has the effect of being able to realize a screen that is easy for the user to operate.

[0081] The basic operation unit 1101 performs the above-mentioned processing of S101 to S108, and at the same time determines whether the operation indicator has performed a touch operation on the touch screen 180 (touch panel 123) (S109). In the case where the operation indicator has performed a touch operation (S109: Yes), various processes corresponding to the touch operation are performed (S110). For example, in the case where the icon of a specified application is touched, the corresponding application is started. In the case where S109 determines that no touch operation has occurred, it branches into two cases. In the case where S102 determines that the user is riding in a vehicle (and shaking occurs) (S109: No / riding), the process returns to S104. In the case where S102 determines that the user is not riding in a vehicle (or the vehicle is not shaking) (S109: No / other than riding), the process returns to S101.

[0082] Figure 5B: is a diagram for explaining the details of the processing of S106 (judgment of whether the operation indicator is present). A side view of the portable information terminal 100 and the operation indicator (the user's finger 200) is shown here, and the user's finger 200 is included in the captured image obtained by the first image input unit 133. If the distance between the finger 200 and the touch screen 180 is L, then when the user performs a touch operation, the distance L will inevitably become 0, but when the distance L is large, it is in an operation preparation state before the touch operation (for example, a state where the target icon of the touch operation has not yet been determined, or a state where the target icon is being searched for on the screen). If the above-mentioned anti-misoperation processing is performed in such a preparation state where the distance L is large, the display screen (icon) will move following the movement of the finger 200, resulting in the inconvenience that the user cannot select the target icon. For this reason, it is necessary to control so that the above-mentioned anti-misoperation processing is not performed in the operation preparation state, in other words, it is judged that there is no operation indicator in the processing of S106.

[0083] Specifically, a threshold value L1 is defined for the distance L between the operation indicator and the touch screen 180. When L≤L1, it is determined that the operation indicator exists, and when L>L1, it is determined that the operation indicator does not exist. In addition, since each user's operation method (whether the touch operation start position is close or far from the touch screen 180) is different, it is preferred that the threshold value L1 can be appropriately set for each user. Thus, by disabling the error operation prevention process during the preparation period of the touch operation, no inconvenience is caused to the user in performing the touch operation.

[0084] As described above, by using Example 1, even when a user riding in a shaking vehicle uses the portable information terminal 100, the position of the display screen is shifted in accordance with the relative shaking amount of the operation indication unit in a manner that follows the shaking amount, thereby preventing erroneous operations caused by misalignment of the touch operation position.

[0085] Example 2

[0086] In Example 2, in order to implement the misoperation prevention process, the position of the display screen is kept unchanged but the detection area of ​​the touch operation is shifted. The basic structure of the portable information terminal 100 is the same as that of Example 1, so the following mainly focuses on the differences from Example 1, and the repeated description of the common parts is omitted.

[0087] Figure 6 The figures are for explaining the error-operation prevention process in this embodiment. (a) to (c) correspond to the figures of the embodiment 1. Figure 3B Here, the screen displayed on the touch panel 180 is assumed to be 180a, and the operation detection areas corresponding to the icons on the touch panel 123 are indicated by dotted-line frames 123a and 123b.

[0088] As shown in (a), the first image input unit (operation indicator shooting camera) 133 obtains an image of the finger 200 that is about to touch the target icon 180i1. In this state, the operation detection area 123a of each icon is consistent with the display position of each icon on the display screen 180a. Next, as shown in (b), when the vehicle shakes, the shaking detection unit 1103 analyzes the image of the finger 200 obtained by the first image input unit 133, and calculates the position change of the finger 200 based on the housing position of the portable information terminal 100, that is, the relative shaking amount ΔX.

[0089] In (c), the screen 180a displayed on the touch screen 180 is kept intact, and the area control unit 1105 sets the operation detection area corresponding to each displayed icon on the touch panel 123 to be offset by a specified amount ΔX' as shown in 123b in a manner that follows the relative shake amount ΔX of the finger 200 calculated by the shake detection unit 1103.

[0090] For example, the detection area corresponding to the icon 180i1 is set to a position shifted to the right compared to the display position of the icon 180i1 (indicated by oblique lines). As a result, even if the finger 200 touches a position shifted to the right compared to the display area of ​​the target icon 180i1 due to relative shaking, the detection area is set to follow the position, so that it can be detected that the icon 180i1 is accurately touched.

[0091] The action flow of Example 2 has changed Figure 5A That is, in the process of S108, the area control unit 1105 performs a process of offsetting the operation detection area corresponding to each icon on the display screen on the touch panel 123 by ΔX' based on the relative shake amount ΔX of the operation indicator received from the shake detection unit 1103. At this time, the relationship between the relative shake amount ΔX of the operation indicator and the offset amount ΔX' of the detection area can be set to ΔX'=kΔX using the coefficient k (≤1) as in the case of Example 1. Other processes and Figure 5A The flowchart shown is the same.

[0092] As described above, according to the second embodiment, even when a user riding in a shaking vehicle uses the portable information terminal 100, the position of the operation detection area is shifted according to the relative shaking amount of the operation instruction unit, thereby preventing erroneous operations caused by misalignment of the touch operation position.

[0093] Example 3

[0094] In the third embodiment, the intervals between icons are enlarged on the display screen to prevent misoperation. The basic structure of the portable information terminal 100 is the same as that of the first embodiment, and the differences from the first embodiment will be mainly described below.

[0095] Fig. 7A 11 is a diagram illustrating the anti-misoperation processing in the present embodiment. (a) shows an example of a display screen 180a in which the user is not in a vehicle, or even if the user is in a vehicle, the vehicle is not shaking. The display control unit 1104 sets the display interval of the icons in the display screen 180a to a normal interval (the horizontal direction of the screen is d1, and the vertical direction is d2). On the other hand, (b) shows an example of a display screen 180c in which the user is in a vehicle and the vehicle is shaking. The display control unit 1104 sets the display interval of the icons in the display screen 180c to a wider interval (the horizontal direction of the screen is d1', and the vertical direction is d2') (d1'>d1, d2'>d2). In this way, since the icon interval is enlarged when the vehicle is shaking, even if the position of the user's hand shakes, it is possible to reduce misoperation of touch operations.

[0096] Here, in order to implement the error operation prevention processing, the interval between icons is enlarged. By linking with this, the area control unit 1105 expands the operation detection area corresponding to each icon on the display screen on the touch panel 123, which can further reduce the effect of error operation.

[0097] Figure 7B The following processing is mainly controlled by the state detection unit 1102 and the display control unit 1104.

[0098] First, the state detection unit 1102 obtains the position information, tilt information, motion information, etc. of the portable information terminal 100 from the sensor unit 150 (S301). Next, based on the information obtained in the process of S301, the state detection unit 1102 determines whether the user holding the portable information terminal 100 is riding in a vehicle and the vehicle is shaking (S302).

[0099] In the process of S302, if it is determined that the user is not in the vehicle, or even if the user is in the vehicle, the vehicle is not shaking (S302: No), the process proceeds to S303. If it is determined that the user is in the vehicle and the vehicle is shaking (S302: Yes), the process proceeds to S304.

[0100] In the process of S303, the display control unit 1104 sets the intervals of icons displayed on the touch screen 180 to the normal intervals (d1, d2) for screen display. On the other hand, in the process of S304, the intervals of icons displayed on the touch screen 180 are set to the wider intervals (d1', d2') for screen display. In this case, the area control unit 1105 may also expand the operation detection area corresponding to each icon on the display screen on the touch panel 123.

[0101] The basic operation unit 1101 performs the above-mentioned processing of S301 to S304, and at the same time confirms whether the operation indicator has performed a touch operation on the touch screen 180 (touch panel 123) (S305). In the case where the operation indicator has performed a touch operation (S305: Yes), various processes corresponding to the touch operation are performed (S306). In the case where the operation indicator has not performed a touch operation (S305: No), the process returns to S301.

[0102] In the above description, only one type of interval (d1', d2') of icons is set when the vehicle shake occurs, but a plurality of types of intervals may be set according to the magnitude of the vehicle shake.

[0103] As described above, according to the third embodiment, when a user riding in a shaking vehicle uses the portable information terminal 100, since the display intervals of the icons on the display screen are enlarged, erroneous operations due to displacement (misalignment) of the touch operation position can be prevented.

[0104] Example 4

[0105] In the fourth embodiment, in order to implement the error operation prevention process, when the user performs a touch operation, a confirmation screen is displayed for confirming whether the touch operation is correct. This process can be performed alone or in combination with the processes of the first to third embodiments.

[0106] Fig. 8A 1 is a diagram of a confirmation screen for preventing misoperation in this embodiment. When the operation indicator (the user's finger 200) performs a touch operation on the touch screen 180, the display control unit 1104 displays an operation confirmation message 180m, temporarily interrupts the operation, and waits for the user's response. The operation confirmation message 180m displays an "OK" button and a "Cancel" button, wherein the "OK" button is for the user to respond when the operation just performed is correct (normal operation), and the "Cancel" button is for the user to respond when the operation just performed is incorrect (misoperation). The user determines the correctness of the operation based on the position just touched and selects a button.

[0107] When the user selects the "OK" button, the touch operation just performed is validated and various processes corresponding to the touch operation are executed. When the user selects the "Cancel" button, the touch operation just performed is invalidated and the user's touch operation is accepted again.

[0108] Figure 8B The following processing is mainly controlled by the state detection unit 1102 and the display control unit 1104.

[0109] First, the state detection unit 1102 obtains the position information, tilt information, motion information, etc. of the portable information terminal 100 from the sensor unit 150 (S401). Next, the state detection unit 1102 determines whether the user holding the portable information terminal 100 is riding in a vehicle and the vehicle is shaking based on the information obtained in the process of S401 (S402).

[0110] In the process of S402, if it is determined that the user is not in the vehicle, or even if the user is in the vehicle, the vehicle is not shaking (S402: No), the process proceeds to S403. If it is determined that the user is in the vehicle and the vehicle is shaking (S402: Yes), the process proceeds to S405.

[0111] In the process of S403, the basic operation unit 1101 confirms whether the operation indicator has performed a touch operation on the touch screen 180 (touch panel 123). If the operation indicator has performed a touch operation (S403: Yes), various processes corresponding to the touch operation are performed (S404). If the operation indicator has not performed a touch operation (S403: No), the process returns to S401.

[0112] In the process of S405, the basic operation unit 1101 confirms whether the operation indicator has performed a touch operation on the touch screen 180. When the operation indicator has performed a touch operation (S405: Yes), an operation confirmation message 180m ("OK" / "Cancel" button) is displayed to confirm whether the above touch operation is correct (S406). When the operation indicator has not performed a touch operation (S405: No), the process returns to S401.

[0113] In the process of S407, the user's response to the operation confirmation message 180m is received. When the user selects the "OK" button (the above touch operation is a normal operation) (S407: "OK"), various processes corresponding to the above touch operation are performed (S408). When the user selects the "Cancel" button (the above touch operation is an erroneous operation) (S407: "Cancel"), the process returns to S401.

[0114] As described above, with Example 4, when a user riding in a shaking vehicle uses the portable information terminal 100, a confirmation screen is displayed to confirm whether the touch operation performed by the user is correct. Therefore, if an erroneous operation is made, the user can immediately switch to a correction operation, thereby improving user usability.

[0115] In addition, in the above embodiment, the operation confirmation message 180m is displayed to prevent erroneous operation, but instead, the correctness of the touch operation can be judged according to the duration of the touch state. For example, if the duration of the touch state is longer than the prescribed time, it is judged to be valid, and if it is less than the prescribed time, it is judged to be invalid. Alternatively, the definition of valid / invalid can be set to the opposite by the user. Furthermore, the above operation confirmation message can also be displayed only when the touch state lasts for more than the prescribed time.

[0116] Furthermore, when the user realizes that the wrong icon has been touched, before the operation is confirmed, the user can keep the finger in contact with the touch screen surface and slide the finger to the target icon position to reselect the target icon. Then, the above-mentioned operation confirmation message is displayed in this state. In this way, correction actions and confirmation actions can be performed continuously. Among them, when the finger is kept in contact and slid in a state where the touch operation is to be confirmed, the display position of the icon is fixed. When the finger is kept in contact and slid in a state other than the state where the touch operation is to be confirmed, the display position of the icon is moved. By distinguishing the use in this way, convenience can be further improved.

[0117] Example 5

[0118] In the fifth embodiment, an audio-visual device that can be worn on the user's head and displays VR (Virtual Reality) etc. is described as a usage mode of the portable information terminal 100. In this case, the erroneous operation prevention process is also effective.

[0119] Fig.9A The figures show the state of using the portable information terminal 100 via the audio-visual device 10. The portable information terminal 100 is inserted into the audio-visual device (adapter) 10, and the user 1 wears it for use. (a) is a top view of the user 1 wearing the audio-visual device 10 as viewed from directly above, and (b) is a front view of the user 1 wearing the audio-visual device 10 as viewed from the front. By using such an audio-visual device 10, the same function as that of an HMD (Head Mounted Display) can be achieved. Alternatively, instead of the audio-visual device 10, the user 1 can hold the portable information terminal 100 with both hands or one hand to place the portable information terminal 100 in front of the eyes.

[0120] The audio-visual device 10 is a structure in which a portable information terminal 100 such as a smartphone or a tablet terminal is inserted into the gap portion 20 for support. The user 1 places the audio-visual device 10 with the portable information terminal 100 inserted in front of the eyes and fixes it on the head using a headband or the like. The display screen (image display unit 131) of the portable information terminal 100 can superimpose artificial objects generated using CG (Computer Graphics) or the like on the image data (camera image) input from the rear camera, i.e., the third image input unit 135, to perform VR (Virtual Reality) display or MR (Mixed Reality) display, etc. In addition, in the case of using an HMD with a transmissive display, it is possible to perform AR (Augmented Reality) display, etc., in which artificial objects generated using CG or the like are superimposed on the surrounding scenery.

[0121] The image displayed on the display screen of the portable information terminal 100 is incident on the eyes of the user 1 via the lens portion 11 disposed inside the audio-visual device 10. In addition, the partition wall portion 12 prevents the mixing of the images passing through the left and right lens portions 11. The above-mentioned slit portion 20, lens portion 11, partition wall portion 12, etc. are internal structures of the audio-visual device 10 and are not normally exposed to the outside. In addition, preferably, a part of the outer shell of the audio-visual device 10 normally covers the portable information terminal 100 like a cover, forming a structure that can prevent the portable information terminal 100 from falling and being misplaced even when the head of the user 1 moves up and down or left and right. In addition, a window 13 is provided on the outer shell of the audio-visual device 10 to expose the third image input unit 135 on the back of the portable information terminal 100 to the outside.

[0122] Fig. 9B 1 is a diagram for explaining the misoperation prevention process in this embodiment, and shows the state of the user operating the portable information terminal 100. The user 1 can obtain a virtual observation range 180s by viewing the image data input from the third image input unit 135 of the portable information terminal 100 via the image display unit 131. In the virtual observation range 180s, a virtual operation panel 180t is arranged at a predetermined distance in front. The user 1 can issue an operation instruction to the portable information terminal 100 by operating the icons in the virtual operation panel 180t with a real finger 200.

[0123] For example, assume that the user 1 in a vehicle touches a target icon on the virtual operation panel 180t with the finger 200 of the right hand. The shaking of the vehicle may cause the user's head to shake to the left, and the virtual operation panel 180t to shake to the left, and the finger 200 of the right hand to shake to the right. As a result, another icon on the virtual operation panel 180t is touched instead of the target icon, and an erroneous operation occurs.

[0124] In this regard, the same misoperation prevention process as in the first embodiment can be applied. The third image input unit (rear camera) 135 of the portable information terminal 100 obtains an image of the finger 200 that is to perform a touch operation on the virtual operation panel 180t. The shake detection unit 1103 analyzes the image of the finger 200 obtained by the third image input unit 135, and calculates the relative shake amount of the finger 200 based on the display position of the virtual operation panel 180t (or the housing position of the portable information terminal 100). Furthermore, the display control unit 1104 controls based on the calculated relative shake amount ΔX of the finger 200 so as to shift the display position of the virtual operation panel 180t within the virtual observation range 180s by ΔX'.

[0125] In this case, it is also possible to introduce Figure 5B It is determined whether the operation indicator (finger 200) described in the figure is present. That is, by measuring the distance from the portable information terminal 100 to the finger 200 using the auxiliary light emission / infrared rangefinder 136, the distance L from the virtual operation panel 180t to the finger 200 can be calculated. Then, the anti-misoperation processing can be enabled only when the distance L is below the threshold. In addition, in the case of the virtual operation panel 180t of the present embodiment, the display position can also be shifted three-dimensionally including the line of sight direction. As a result, when the user uses the portable information terminal 100 via the audio-visual device 10, misoperation caused by vehicle shaking can also be prevented.

[0126] The above uses Examples 1 to 5 to illustrate the implementation of the present invention, but the structure of the technology implementing the present invention is of course not limited to the above examples, and various modifications can be considered. For example, the case where the icon (object) displayed on the touch screen is stationary is described, but the anti-misoperation processing of the present invention can also be applied when the icon is moving, and the same effect can be obtained. In addition, the object is not limited to the icon, and it is also effective when an image such as a map is displayed and the user specifies a position on the map through a touch operation.

[0127] Between the various embodiments, a part of the structure of a certain embodiment can be replaced with the structure of another embodiment, and the structure of another embodiment can be added to the structure of a certain embodiment. All of these belong to the scope of the present invention. In addition, the numerical values ​​and messages appearing in the text and the figures are only examples, and even if they are different, they do not impair the effects of the present invention.

[0128] The functions of the present invention described above can be implemented in part or in whole by hardware, for example, by designing an integrated circuit. Alternatively, the functions of the present invention can be implemented by software by interpreting and executing a program for implementing the functions by a microprocessor unit. Hardware and software can also be used simultaneously. The software can be stored in the ROM 103 and the storage unit 110 of the portable information terminal 100 before the product leaves the factory. The software can also be obtained from various server devices on the Internet after the product leaves the factory. Alternatively, the software provided by a memory card or a CD can be obtained.

[0129] In addition, the control lines and information lines shown in the figure only represent the parts necessary for explanation, and do not necessarily represent all the control lines and information lines on the product. In fact, it can be considered that almost all structures are connected to each other.

[0130] Description of Reference Numerals

[0131] 10...audio-visual device, 100...portable information terminal, 123...touch panel, 123a, b...operation detection area, 131...image display unit, 133...first image input unit (camera for shooting operation indicator), 134...second image input unit (front camera), 135...third image input unit (rear camera), 150...sensor unit, 180...touch screen, 180a, b, c...display screen, 180i...icon, 180m...message for operation confirmation, 180t...virtual operation panel, 200...user's finger (operation indicator), 1101...basic action unit, 1102...state detection unit, 1103...shake detection unit, 1104...display control unit, 1105...area control unit.

Claims

1. A portable information terminal that can be installed in an audio-visual adapter and used as a head-mounted display, comprising a touch screen having an image display portion and a touch panel portion, and capable of issuing instructions by touching the touch screen, characterized in that: include: a display control unit that sets and displays a virtual operation panel at a predetermined position in a virtual observation space region realized via the image display unit; an image input unit, which is arranged on the back side of the surface provided with the touch screen and can obtain an image of an operation indicator for performing a touch operation on the virtual operation panel; and a shake detection unit that analyzes the image of the operation indicator obtained by the image input unit and calculates a relative and three-dimensional shake amount of the operation indicator based on a three-dimensional position of the virtual operation panel including a position in a line of sight direction of a user of the portable information terminal, Based on the relative and three-dimensional shake amount of the operation indicator calculated by the shake detection unit, the display control unit controls the image display unit to follow the relative and three-dimensional shake amount so that the virtual operation panel is offset and displayed in a three-dimensional direction including the user's line of sight.

2. The portable information terminal according to claim 1, wherein: comprising a state detection unit for detecting the state of the portable information terminal, When the state detection unit determines that the portable information terminal is not in a vehicle or is not shaken even if in a vehicle, the display control unit does not shift the object displayed on the image display unit.

3. The portable information terminal according to claim 2, characterized in that: The shake detection unit measures the distance between the operation indicator and the touch screen based on the image of the operation indicator obtained by the image input unit, and determines that the operation indicator does not exist when the distance is greater than a threshold value. The display control unit does not offset the object displayed on the image display unit.

4. The portable information terminal according to claim 2, characterized in that: When the state detection unit determines that the portable information terminal is in a vehicle and is shaking, and when a touch operation is performed on the touch screen, the display control unit displays a confirmation screen on the image display unit for confirming whether the touch operation is correct.

Images