Portable information terminal device and program thereof
The integration of a rear-facing camera in smartphones to detect hand motions simplifies content navigation, enhancing operability and visibility by allowing users to switch or scroll content without obstructing the display.
Patent Information
- Application Number
- JP2024158838
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2041-05-27
AI Technical Summary
Conventional mobile information terminal devices, such as smartphones, require complex and repetitive operations to navigate through content, leading to reduced visibility and operability due to the need for frequent swiping and tapping on the display, which can obscure content and lead to frustration.
Equipping smartphones with a rear-facing camera to detect hand motions, allowing users to switch or scroll content by moving their hand behind the device, thereby simplifying navigation and maintaining visibility.
Enables seamless content switching and scrolling without obstructing the display, improving operability and reducing the complexity of operations by using hand gestures captured by the rear camera.
Smart Images

Figure 2024163311000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a portable information terminal device having a camera on the back side of a surface having a display, and a program for use therewith, and is particularly applicable to smartphones and the like. [Background technology]
[0002] Recent mobile information terminal devices are equipped with touch panels. A touch panel is a transparent film-type pointing device placed on a display, and may also be called a touch screen, touch display, contact screen, or the like. In such a mobile information terminal device equipped with a touch panel, as disclosed in Patent Document 1, for example, a finger or a stylus is brought into contact with a display (more precisely, a touch panel on a display), and then the finger or the like is moved in a sliding manner in a predetermined direction, thereby moving content or the like displayed on the display in that direction or causing the device to perform a predetermined operation. Such an operation is called a swipe or slide gesture.
[0003] Incidentally, a smartphone is a typical example of a modern portable information terminal device, and in addition to its original telephone function, it also has the function of using the Internet and taking pictures of scenery with an attached camera, making it a highly useful multi-function device. In particular, when using the Internet, it is possible to view the contents of a searched website, and if the website is an EC (Electronic Commerce) site, it is also possible to purchase products that are on sale. Therefore, although it is desirable for more information to be displayed on the display of a smartphone, there is a limit to the size of the display when portability is taken into consideration.
[0004] Therefore, in smartphones and the like, by using technology such as that disclosed in Patent Document 1 below, it is possible to move some of multiple contents that cannot be displayed on the display at once due to the display size in the swipe direction and move them out of the display range, thereby making it possible to display other contents. As a result, for example, by swiping the display left and right or up and down, a user of the smartphone can visually check the information on a large number of contents or products displayed as reduced images on the display like thumbnails by scrolling left and right or up and down in order. In addition, for contents or products about which the user wishes to know more, detailed information can be displayed on the display by tapping on the reduced image. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] Special Publication No. 2010-515978 Summary of the Invention [Problem to be solved by the invention]
[0006] In this way, a mobile information terminal device such as a smartphone allows a user to swipe or tap the display with a finger or the like, making it possible to display a large amount of information even on a small display in which portability is prioritized over display size, but there are problems with operation, such as the following: For example, when a thumbnail image of a certain piece of content is selected from thumbnail images of many pieces of content and detailed information is displayed, if the detailed information differs from what was expected, it becomes necessary to tap the display with a finger or the like again to perform a "back operation" that displays many thumbnail images, or to perform a "selection operation" that taps a thumbnail image of another piece of content and displays detailed information.
[0007] In other words, with conventional mobile information terminal devices, users had to repeatedly swipe or tap the display to perform selection and back operations until they found the desired content, product, or other information, which could cause the user to feel that the operations required to reach the desired information were complicated and troublesome.
[0008] In addition, because the touch panel is disposed on the display, when the display is swiped or tapped with a finger or a stylus, part of the content or information displayed on the display may be hidden by the hand doing the swiping, making it difficult for the user to see, which causes a problem in that the visibility of the display may be impaired.
[0009] The present invention has been made to solve the above-mentioned problems, and has an object to provide a portable information terminal device and a program therefor that can improve operability while ensuring visibility. [Means for solving the problem]
[0010] In order to achieve the above object, a mobile information terminal device according to a first aspect of the present invention comprises: A mobile information terminal device having a camera on the back side of a surface having a display, a motion determination means for determining whether or not a motion of a human hand or finger (hereinafter referred to as "hand, etc.") within a predetermined period of time is a predetermined motion based on a series of image information captured by the camera when the series of image information includes an image of the human hand or finger (hereinafter referred to as "hand, etc."); a content switching means for switching the content currently being displayed on the display to another content when the motion determining means determines that the motion of the hand or the like is the predetermined motion, When a predetermined search target is input and a search result is obtained, and when a specific content selected from the plurality of contents is displayed on the display, if the motion determining means determines that the motion of the hand or the like is the predetermined motion, the content switching means switches the content currently displayed on the display to another content. It is characterized by:
[0011] In addition, a program according to a second aspect of the present invention for achieving the above object is as follows: A computer that controls a mobile information terminal device having a camera on the back side of a surface having a display, a motion determination means for determining, when a series of image information captured by the camera within a predetermined period includes an image of a human hand or finger (hereinafter referred to as "hand, etc."), whether or not a movement of the hand, etc. within the predetermined period is a predetermined motion based on the series of image information; a program that functions as a content switching means that switches a content currently being displayed on the display to another content when the motion determination means determines that the motion of the hand or the like is the predetermined motion, In a case where there are a plurality of the other contents, and the content switching means determines a plurality of times in succession that the movement of the hand or the like is the predetermined movement, and the display of the plurality of the other contents is switched in sequence for each determination, When a next display content to be displayed next on the display among the plurality of other contents has the same content as a content already displayed on the display, the content switching means switches to the other content to be displayed next to the next display content instead of the next display content and displays the same on the display. It is characterized by:
[0012] In addition, when the series of image information captured by the camera within a specified period includes an image of a human's (typically a user of the portable information terminal device) hand or finger (hereinafter referred to as "hand, etc." in the "Means for solving the problem" and "Effects of the invention" sections) and the movement determination means determines based on the series of image information that the movement of the hand, etc. within the specified period is a predetermined movement, the content switching means switches the content currently being displayed on the display to the display of other content.
[0013] This allows the user of the portable information terminal device to switch the content currently displayed on the display to another content by moving the hand captured by the camera to perform a predetermined motion, instead of swiping or tapping on the display. For example, when the user holds the portable information terminal device in the left hand, the user can hold the right hand over the back side of the portable information terminal device and perform a predetermined motion to switch the content without the content displayed on the display being hidden by the hand. Note that the "camera" in this specification mainly refers to a digital camera that forms an image of a subject on a solid-state imaging element.
[0014] Furthermore, when there are a plurality of other contents and the content switching means judges multiple times in succession that the movement of the hand or the like is a predetermined movement and switches the display of the plurality of other contents in sequence for each judgment, when the next display content to be displayed next among the plurality of other contents has the same content as the displayed content already displayed on the display, the content switching means switches to the other content to be displayed after the next display content instead of the next display content and displays it on the display. This prevents content with the same content as the already displayed content from being displayed on the display, making it possible to prevent a situation in which the user of the mobile information terminal device feels disappointed or feels that time has been wasted by showing the content that has already been viewed again.
[0015] The display device may also be configured to include an image determination means for determining whether or not a series of image information captured by the rear camera within a predetermined period includes an image (image information, specific image) that satisfies a control condition. When the image determination means determines that an image of a human hand or finger (typically, a user of the portable information terminal device) is included in a series of image information captured by the camera within a predetermined period, and the motion determination means determines based on the series of image information that the motion of the hand or finger within the predetermined period is a predetermined motion, the content switching means may be configured to switch the content currently being displayed on the display to another content.
[0016] Here, the "image satisfying the control conditions (specific image)" refers to an image (series of image information) that the camera on the back side of the mobile information terminal device recognizes as an appropriate image within the imaging area set for executing the display control of the content of the present invention. Furthermore, if the image determination means or the motion determination means recognizes an image within a predetermined imaging area, it can recognize it as a target image even if it is not an image of a human hand or finger (for example, even if it is an image of a stylus or other object), and can execute the display control of the content to be displayed on the display. However, it is preferable that at least an image of a human hand or finger can be recognized as a target image. In addition to this configuration, by adding certain conditions (for example, shape conditions, temperature conditions, etc.) to the image determination means, the display control of the content to be displayed on the display can be performed only when a human hand or finger, or something similar thereto (i.e., something that satisfies certain conditions common to a human hand or finger) is recognized.
[0017] In addition, the present invention assumes that the camera on the back side of the mobile information terminal captures (detects) an image of a human hand or finger and controls the display of the content to be displayed on the display. Therefore, the imaging area in which the captured image can be recognized as a proper image is set to a distance (depth distance) from the camera on the back side of the mobile information terminal between about 0 cm and about 25 cm. More preferably, the imaging area may be set between about 1 cm and about 15 cm.
[0018] This imaging area may be appropriately set according to the performance of the camera on the back side of the mobile information terminal device, and the present invention is not limited to such a specific range (about 0 cm to about 25 cm). However, if the camera is too close, it becomes difficult to recognize (distinguish) various movements such as hand movements, and if the camera is too far away, the user may recognize other image information that is not intended by the user (for example, movements of other people other than the user), so it is preferable to limit the imaging area to the specific range (about 0 cm to about 25 cm) as described above. Effect of the Invention
[0019] According to the mobile information terminal device of the present invention and the computer program for controlling it, the user can, for example, control the display mode by switching the currently displayed content to another content, scrolling the display in the vertical or horizontal directions, or enlarging or reducing the display, by moving the hand or the like captured by the camera so as to perform a predetermined motion or another predetermined motion, instead of swiping or tapping the display. For example, when the user holds the mobile information terminal device with one hand (e.g., the left hand), the user can hold the other hand (e.g., the right hand) over the back side of the mobile information terminal device and perform a predetermined motion or another predetermined motion, thereby enabling the user to perform a content switching operation or a content display mode control without the content or the like being hidden by the hand or the like displayed on the display. Therefore, the operation until the target information is reached can be performed only by the motion of the hand or the like behind the display, thereby ensuring visibility and improving operability. [Brief description of the drawings]
[0020] [Figure 1] 1 is an explanatory diagram showing a smartphone (hereinafter referred to as a "smartphone") according to one embodiment of the mobile information terminal device of the present invention and its program, and the hands and fingers of a user holding and operating it. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of the smartphone according to the present embodiment. [Diagram 3] 13 is a flowchart showing the flow of processing of a browsing operation assistance application program executed by the smartphone of this embodiment. [Figure 4] 4 is a flowchart showing the flow of a subroutine of the browsing operation assist application program shown in FIG. 3, in which (A) shows a first operation process and (B) shows a second operation process. [Diagram 5]An explanatory diagram showing an example of contents displayed on the display of a smartphone of this embodiment. (A) shows a display example of the results of searching for "chair", (B) shows a display example when a certain chair content is selected from the search results of (A), (C) shows a display example when the content of (B) is shifted to a related product content by a first action process, and (D) shows a display example when the display of (B) is scrolled upward by a second action process. [Figure 6] The figures show the state in which the scroller is attached to the smartphone. (A) is a plan view of the smartphone and scroller seen from the front side of the smartphone, (B) is a cross-sectional view showing the state in which the smartphone and scroller are cut along the line 6B-6B shown in (A), (C) is a side view seen from the direction of the arrow 6C shown in (A) and a bottom view of the smartphone and scroller, and (D) is a bottom view showing the state in which the scroller is being attached to or removed from the smartphone in the side view of (C). [Figure 7] 6(B) is an explanatory diagram showing an example of the operation of the roller part constituting the scroller, in the cross-sectional view equivalent to that shown in Fig. 6(B). (A) shows the case where the user rotates the roller part counterclockwise in the figure with his / her finger, (B) shows the state immediately after the corner of the roller part comes into contact with the touch panel from the state of (A), (C) shows the state where the corner is pressed against the touch panel and crushed as the roller part continues to rotate from the state of (B), and (D) shows the state immediately after the crushed corner returns to its original shape as the roller part continues to rotate from the state of (C). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] Next, embodiments to which a mobile information terminal device and a program for the mobile information terminal device of the present invention are applied will be described with reference to the drawings. First, the configuration of a smartphone 10 of the present embodiment will be described with reference to FIGS.
[0022] As shown in FIG. 1, the smartphone 10 is a commercially available type and is composed of a thin, rectangular main body case 10a that can be held by a user in one hand (left hand 90L in the figure), a display 11 provided on the inside and outside of the main body case 10a, a touch panel 12, a control unit 20, etc.
[0023] On the front side 10F of the smartphone 10, a display 11 is attached to a rectangular frame hole formed in the main body case 10a, and a speaker 14 and a front camera 15 are also provided. A rear camera 16 is provided on the back side 10B, and a microphone 13 is provided on the lower end side. The touch panel 12 is provided on the surface of the display 11 and is almost integral with the display 11. Therefore, please note that in the same figure, the display 11 and the touch panel 12 are depicted overlapping in the same rectangular shape (parts colored gray).
[0024] 2, the display 11, touch panel 12, front camera 15, etc. are electrically connected to and controlled by a control unit 20 housed in the main body case 10a. The control unit 20 is a main board having an MPU 21, memory 25, etc., and in addition to the display 11, etc., a wireless unit 17, various sensors 18, a battery 19, and further a memory card, a SIM card, etc. (not shown) are electrically connected to the control unit 20.
[0025] The display 11 is, for example, a 6-inch liquid crystal display device with an aspect ratio of 16:9, and is configured to be able to display image information output from the control unit 20. The touch panel 12 is a transparent film-type pointing device that is approximately the same size as the display 11 and is provided almost in front of the display surface, and is configured to be able to output position information on the display 11 that the user touches with his / her finger or stylus to the control unit 20. The touch panel 12 is sometimes called a touch screen or a contact screen. For position detection, for example, a resistive film method, a capacitive method, an ultrasonic surface acoustic wave method, or the like is used.
[0026] The microphone 13 and speaker 14 are used when making a call using the telephone function, and for recording and audio output by application programs (also called application software. These are hereinafter collectively referred to as "apps"). The microphone 13 is configured to be able to collect the user's voice and the like and output acoustic information to the control unit 20, and the speaker 14 is configured to be able to output acoustic information output from the control unit 20 to the outside.
[0027] The front camera 15 and the back camera 16 are imaging devices having a solid-state imaging element such as a CCD or CMOS and an optical lens, and are configured to be able to output captured image information to the control unit 20. The front camera 15 is provided on the front side 10F of the smartphone 10 and is configured to be able to capture, for example, the face of the user. The back camera 16 is provided on the back side 10B of the smartphone 10 and is configured to be able to capture, for example, scenery. The front camera 15 may be called an in-camera, and the back camera 16 may be called an out-camera.
[0028] The wireless unit 17 is a wireless transmitting / receiving device configured to be capable of voice communication and data communication via a wireless communication network of a mobile network operator (MNO), and the control unit 20 controls the calls and data communication. A SIM card (not shown) stores identification information unique to the smartphone 10 used during wireless communication. The various sensors 18 include a GPS sensor, an acceleration sensor, an illuminance sensor, a temperature sensor, and the like, and are configured to be able to output detection data to the control unit 20.
[0029] The battery 19 is a secondary battery (e.g., a lithium ion battery) that can be charged by DC power supplied from an external source, and is configured to be able to supply driving power to the control unit 20, the display 11, etc. Information about charging and discharging the battery 19 is also output to the control unit 20, and it is configured so that, for example, the control unit 20 can manage charging and discharging.
[0030] The control unit 20 is composed of an MPU 21, a GPU (Graphics Processing Unit) 23, a memory 25, and the like. The MPU 21 is a microcomputer, and is configured to control hardware such as the above-mentioned display 11 and execute a browsing application and a browsing operation assistance application described below by executing an OS (operating system software) such as Android (registered trademark) or iOS (registered trademark) stored in the memory 25. The GPU 23 is used for image data analysis processing, etc., which will be described later. As described in the section [Problems to be solved by the invention], the general smartphone 10 configured in this manner has problems such as the complicated operation required to reach the desired information, which may be troublesome for the user, and the visibility of the display 11 may be impaired.
[0031] Therefore, in the smartphone 10 of the present embodiment, a browsing operation assistance app that can solve these problems is installed. In this browsing operation assistance app, when the rear camera 16 captures (takes an image of) a motion of shaking the right hand 90R from side to side (first motion) as a motion of the hand, etc., the smartphone 10 is controlled so that the content currently displayed on the display 11 is switched to another display content, and when the rear camera 16 captures a motion of shaking the right hand 90R up and down (second motion) as a motion of the hand, etc., the smartphone 10 is controlled so that the content currently displayed on the display 11 is scrolled upward (or downward). From here, such a browsing operation assistance app will be described with reference to Figs. 3 to 5.
[0032] The file of the browsing operation assistance app is, for example, downloaded by data communication via a wireless communication network performed by the function of the wireless unit 17 and installed in the memory 25 of the control unit 20, or is pre-installed in the memory 25. The browsing operation assistance app may be executed independently, or may function in conjunction with, for example, a web browser or a file browser (file manager). That is, the browsing operation assistance app may function as a subroutine of a general-purpose plug-in compatible browsing app (for example, Google Chrome (registered trademark), Safari (registered trademark), Microsoft Edge (registered trademark), Internet Explorer (registered trademark), etc.). In this case, the browsing operation assistance app is an executable file that conforms to a predetermined plug-in format, and is plugged into these browsing apps and executed each time it is called.
[0033] In addition, prior to the execution of the browsing operation assistance application, it is necessary to register the above-mentioned first and second actions and the right hand 90R and left hand 90L that perform these actions in the memory 25 of the control unit 20 in advance. Although the description of the flow of the registration process is omitted, it is performed by a function of the browsing operation assistance application. For example, as a first action for switching the content currently displayed on the display 11 to another display content, the action of the user shaking his / her right hand 90R (or left hand 90L) from side to side and the right hand 90R (or left hand 90L) that performs this action are registered. In addition, as a second action for controlling the display mode of the content currently displayed on the display 11 (for example, scrolling upward), the action of shaking the right hand 90R (or left hand 90L) up and down and the right hand 90R (or left hand 90L) that performs this action are registered.
[0034] The direction of the right hand 90R (or left hand 90L) is in accordance with the up, down, left and right directions previously set (defined) in the rear camera 16. For example, as shown in FIG. 1, when the rear camera 16 is located above the smartphone 10 and the display 11 faces the user, the smartphone 10 sets the upper side of the smartphone 10 as "up", the lower side of the smartphone 10 as "down", the right side of the smartphone 10 as "right", and the left side of the smartphone 10 as "left". Therefore, for example, when the smartphone 10 is held by the user in a state rotated 90 degrees clockwise from the state shown in FIG. 1, the movement of the user waving the right hand 90R above (below) himself is analyzed as the movement of waving the right hand 90R to the left (right), and the movement of waving the right hand 90R to the left (right) is analyzed as the movement of waving the right hand 90R to the down (up) by the motion analysis process (S105) described later.
[0035] Instead of the right hand 90R and the left hand 90L, the movements of the fingers (e.g., index finger, thumb, etc.) and the fingers may be registered, and instead of the right hand 90R and the left hand 90L, for example, the movements of a stylus or touch pen held by the right hand 90R and the stylus that causes the movements may be registered. The right hand 90R, the left hand 90L, the fingers, the stylus or touch pen held by those hands, etc. are collectively referred to as "hands, etc." in the section "Mode for Carrying Out the Invention" below.
[0036] 3, the browsing operation assistance application executed by the control unit 20 first performs a predetermined initialization process in step S101. In this initialization process, for example, the work area and flags of the memory 25 are cleared, and video shooting by the back camera 16 is started. The captured image data is video data within a predetermined period (for example, 1 to 2 seconds).
[0037] In the next step S102, image data acquisition processing is performed. This image data is a series of image information captured by the backup camera 16 within a predetermined period, and is a series of still image information consisting of a plurality of still images captured (cut out) from video data within the predetermined period by frame advance, frame drop, or frame skip. Note that, if video analysis is possible in the image data analysis processing in step S103, the series of image information may be video-captured video image information for a predetermined period.
[0038] 1, when a user is looking at the display 11 while holding the smartphone 10 in the left hand 90L, the user often swipes or taps the touch panel 12 with the right hand 90R. Therefore, in the browsing operation assistance app, when the smartphone 10 is held in the left hand 90L (or right hand 90R), the rear camera 16 captures an image of the movement of the free right hand 90R (or left hand 90L).
[0039] For example, when a user swings his / her right hand 90R from side to side (in the direction of the arrow shown in FIG. 1) with the palm of his / her hand facing the rear camera 16, a series of still image information capturing the movement of the right hand 90R, 90R' is acquired in step S102. Therefore, in the image data analysis process in the following step S103, an analysis is performed to determine whether or not the series of still image information includes image information of the right hand 90R. This image data analysis process (S103) is mainly performed by the GPU 23.
[0040] In the image data analysis process in step S103, for example, pattern matching is used as a known image analysis method. In the image analysis by pattern matching, for example, image information of the user's right hand 90R is registered in advance in the memory 25 as template image information, and a match determination is performed between each of a series of still image information to be compared and the template image. In this image data analysis process, for example, the ratio of still image information determined to match the template image information among the series of still image information is obtained.
[0041] In step S104, a process is performed to determine whether or not the series of still image information includes the user's hand (right hand 90R in the example of FIG. 1). In the previous step S103, the ratio of still image information determined to match the template image information is obtained, and here, if the ratio is equal to or greater than a predetermined value, it is determined that the series of still image information includes a hand, and if it is less than the predetermined value, it is determined that the hand is not included. The predetermined value is predetermined based on the analysis accuracy of the image data analysis process in step S103, and is set arbitrarily within the range of, for example, 70 to 90%.
[0042] If it is determined by the determination process of step S104 that the series of still image information includes a hand or the like (S104; YES), a motion analysis process is performed in the following step S105. On the other hand, if it is determined that the series of still image information does not include a hand or the like (S104; NO), since there is a high probability that the image captured by the rear camera 16 includes something other than the right hand 90R, for example, the process proceeds to step S102 again to obtain image data captured within a predetermined period from this time onward.
[0043] In step S105, a motion analysis process is performed. In this process, an analysis is performed on what kind of movement (action) is being made in the image of the right hand 90R and the like included in the series of still image information acquired in step S102. This process is also performed mainly by the GPU 23 using, for example, pattern matching as a known image analysis method, similar to the image data analysis process in step S103. Note that, in the image data analysis process in step S103, when an analysis process having the same content as the motion analysis process in step S105 is also performed, the motion analysis process (S105) may be omitted.
[0044] As described above, in this embodiment, the first and second actions are registered in advance in the memory 25 of the control unit 20. Therefore, for example, when the action of the user shaking his / her right hand 90R etc. from side to side is registered as the first action, in this action analysis process, a plurality of still images captured by frame-by-frame advancement of the movement of the right hand 90R etc. are used as template image information, and a matching determination is made between each of the series of still image information to be compared and the template image. Also, for example, when the action of the user shaking his / her right hand 90R etc. up and down is registered as the second action, in this process, a plurality of still images captured by frame-by-frame advancement of the movement of the right hand 90R etc. are used as template image information, and a matching determination is made between each of the series of still image information to be compared and the template image.
[0045] In the motion analysis process (S105), a matching judgment result is obtained as to whether the motion matches either the first motion or the second motion, or whether the motion does not match either the first motion or the second motion. Therefore, in the motion judgment process in the following step S106, it is judged whether the motion (the first motion and the second motion) has been registered, and if it has been registered, whether it is the first motion or the second motion.
[0046] In addition, in the motion analysis process in step S105, the direction and speed of the right hand 90R are analyzed in the motion of waving the hand. For example, when the right hand 90R is waving left and right, directional information of "left and right" is obtained by analysis as the direction of waving the hand. Also, by comparing the speed of the hand moving from left to right (rightward) with the speed of the hand moving from right to left (leftward), information on the magnitude relationship of the speed in the left and right directions is also obtained by analysis. Similarly, when the right hand 90R is waving up and down, directional information of "up and down" is obtained by analysis as the direction of waving the hand, and by comparing the speed of the hand moving up and the speed of the hand moving down, information on the magnitude relationship of the speed in the up and down directions is also obtained by analysis. The information on the direction and speed of waving the hand is used in the second motion process (S108, FIG. 4(B)) described later.
[0047] If the action based on the image of the right hand 90R, etc. is determined to match the first action (e.g., the action of shaking the right hand 90R from side to side) by the action determination process of step S106 (S106; YES (first action)), the process proceeds to the first action process of step S107, and if the action based on the image of the right hand 90R, etc. is determined to match the second action (e.g., the action of shaking the right hand 90R up and down) (S106; YES (second action)), the process proceeds to the second action process of step S108. In contrast, if the action based on the image of the right hand 90R, etc. (e.g., the action of shaking the right hand 90R diagonally, etc.) is determined to match neither the first action nor the second action, the action captured by the back camera 16 is different from the predetermined action, etc. that has been determined in advance. In other words, there is a high possibility that the action of the user's right hand 90R, etc. is incorrect. Therefore, the process returns to step S102 to obtain image data captured within a predetermined period from this time onwards.
[0048] A flowchart of the first operation process (S107) and the second operation process (S108) is shown in Fig. 4. Also, an explanatory diagram showing an example of content displayed on the display 11 by these processes is shown in Fig. 5.
[0049] As shown in Fig. 4(A), in the first operation process (S107), first, a content information acquisition process is performed in step S201. Content refers to the substance or substance of information, and content information refers to information related to the content (for example, address information such as a URL where the content exists, file name, file type, file size, creation date, information on other content that is linked (associated), etc.). Here, information on the content currently being displayed on the display 11 is acquired from the viewing application.
[0050] For example, as shown in FIG. 5(A), when "chair" is entered into a search box 70 of a search engine provided via a browsing app on the Internet on the smartphone 10, search results are displayed on the screen of the display 11. In this example, reduced image information 71a, 71b, and 71c of a plurality of recommended products (recommended products) are displayed side by side from the left side of the screen to the right side in order of recommendation. In addition, in the same figure, among the reduced image information 71a and the like, a part of the reduced image information 71c that cannot be displayed on the screen due to the display size and hidden information 79 that cannot be displayed are represented by being colored in light gray on the outer right side of the display 11. The hidden information 79 is displayed by scrolling the screen display to the left.
[0051] Also, summary information 73 of mail-order sites selling "chairs" is displayed as text information below the reduced image information 71a to 71c on the screen. Although not shown in Fig. 5(A), multiple pieces of summary information 73 of mail-order sites not shown on the display 11 are obtained from the Internet as search results so that they can be displayed in order of ranking, and summary information 73 not shown in the figure can be displayed by scrolling the screen display upwards.
[0052] After the reduced image information 71a to 71c and the summary information 73 are displayed on the display 11, if the user selects the reduced image information 71a of "office stacking chair" via the touch panel 12 with a finger, a stylus, or a touch pen, for example, detailed information 74a of the stacking chair is displayed as content, as shown in Fig. 5(B). In this case, in step S201, information on the detailed information 74a of the stacking chair displayed on the display 11 is acquired as content information. Then, based on this acquired content information, the following step S202 is performed to determine whether or not there is a next content, that is, a content that has not yet been viewed this time.
[0053] In the example of the stacking chair content shown in FIG. 5(B), when scrolling upward, multiple related product information 75a, 75b, 75c ("transport cart," "set of 2 chairs," and "set of 5 chairs") displayed in the "★★★Related Products★★★" column of detailed information 74a' that subsequently appears on the display 11 are linked as other content information, as shown in FIG. 5(D). In this way, when other content information is linked to the currently viewed content or content that can be viewed by scrolling (hereinafter referred to as "viewed content, etc."), the next content exists. Therefore, in step S202, it is determined that the next content exists (S202; YES), and in the next step S203, processing is performed to obtain information on the next content from the viewing application or from the Internet via the viewing application.
[0054] Then, in the next step S205, a process is performed to determine whether the content is the same as the previously displayed content. In other words, if the next content has been viewed before, that is, the previous time, and the viewed content is displayed again on the display 11, the user will see the content that he or she has already viewed again, which may cause the user to feel disappointed or to feel that time has been wasted. In addition, if the content α being viewed is linked to the next content β, an infinite loop of viewing will be formed, such as content α → content β → content α → content β → ...
[0055] For this reason, in this embodiment, in step S205, a process is performed to determine whether the next content is the same as the previously viewed content. If the next content is the same as the previously viewed content (S205; YES), the process returns to step S202 to determine whether another next content exists, and if not (S205; NO), the process proceeds to step S206.
[0056] The determination as to whether these contents are the same or not is made based on content information such as the address information, file name, file size, creation date, etc. of the contents, and if this content information is the same, it is determined that these are the same contents. Also, if previously viewed contents are stored in the cache memory or cookie file of the control unit 20, and the same contents exist in the cache memory, it is determined that these are the same contents.
[0057] This prevents previously viewed content from being displayed again on the display 11, making it possible to avoid the occurrence of problems such as those described above. Note that if allowing previously displayed content to be viewed again is more advantageous than preventing such problems, the determination process (S205) is unnecessary. Therefore, in this case, the determination process in step S205 is deleted.
[0058] On the other hand, if it is determined in step S202 that there is no next content (S202; NO), that is, if other content information such as related product information 75a-75c of the browsed content is not linked, the process proceeds to step S204. Then, after a process of acquiring end information (predetermined data or a predetermined flag) indicating that there is no next content from memory 25, the process proceeds to step S206.
[0059] In addition, in the case where the content information acquired in step S201 is not linked to other content information, for example, the content currently displayed on the display 11 and other content existing in the same example in the display hierarchy (or data hierarchy) may be determined as the next content, and it may be determined in step S202 that a next content exists.
[0060] For example, if no other content information is linked to the detailed information 74a of the stacking chair shown in Fig. 5(B), the content may be returned to the previous content (or one level above in the display hierarchy) and other reduced image information 71b, 71c displayed in the same row as the reduced image information 71a of the stacking chair shown in Fig. 5(A) may be selected as the next content. For example, when the content of "GAMING CHAIR" of the reduced image information 71b displayed to the right of the reduced image information 71a according to the recommended order (from the left side to the right side of the screen) is selected, the detailed information of "GAMING CHAIR" is displayed in the same display format as the detailed information 74a of the stacking chair shown in Fig. 5(B) and the detailed information 74b of the transport cart shown in Fig. 5(C), as in the case of selecting the reduced image information 71 of "GAMING CHAIR" on the same screen.
[0061] In step S206, a display information output process is performed. In this process, the next content information acquired in step S203 or the end information acquired in step S204 is output to the browsing application. When the next content information is output to the browsing application, the browsing application acquires data of other content from the Internet based on the next content information (or from memory 25 if data of other content has already been downloaded by pre-reading or the like), and performs a process of switching the content currently displayed on display 11 to the display of the other content. In other words, the browsing application receives an instruction from the browsing operation assistance application and switches the display content of display 11 to the other content in response to the first operation performed by the user.
[0062] For example, in the detailed information 74a' of the stacking chair shown in Fig. 5(D), thumbnail images of the contents "transport dolly", "set of 2 chairs", and "set of 5 chairs" are linked as multiple related product information 75a to 75c, so the browsing application receives an instruction from the browsing operation assistance application and displays these contents as other contents on the display 11 in the display order on the display 11 (from the left side to the right side of the screen). In this example, the related product information 75a displayed on the left edge of the screen is a thumbnail image of the content "transport dolly", so detailed information 74b of "transport dolly for stacking chairs" is displayed on the display 11 as the content (next content) linked to it in response to the first action performed by the user (see Fig. 5(C)).
[0063] On the other hand, when the end information is output to the viewing application, there is no next content linked to the viewed content, etc., and the viewing application cannot obtain other content information, so it cannot switch the display on display 11 to other content. Therefore, in such a case, for example, even if the user performs the first action, the viewing application may maintain the content currently displayed on display 11 as it is, or may display information on display 11 that there is no content to switch to.
[0064] When the process in step S206 ends, the series of first operation processes is completed and the process returns to the main routine of the browsing operation assistance application shown in Fig. 3. Then, the process proceeds to step S102 again to obtain image data captured within a predetermined period from this time onward. In other words, the process prepares for the next hand movement or the like made by the user.
[0065] As shown in FIG. 4(B), in the second motion process (S108), first, hand information acquisition process is performed in step S301. The hand information is information on the direction in which the hand is waved and the speed at which it is waved, which is obtained by the motion analysis process (S105) shown in FIG. 3 described above. Generally, when a person waves their hand while being conscious of the direction, the speed at which they wave in that direction is faster than the speed at which they wave in the opposite direction to return their hand after waving. Therefore, from this information, it is possible to know that the hand is waved in the direction with the greater speed (upward if up>down, and downward if up<down), that is, the motion direction of the hand.
[0066] Therefore, if it is determined by the determination process in the next step S302 that the hand or the like is being waved in the upward direction (S302; upward direction), the process proceeds to step S303, where an upward scroll setting process is performed. Also, if it is determined by the same determination process that the hand or the like is being waved in the downward direction (S302; downward direction), the process proceeds to step S304, where a downward scroll setting process is performed. In these setting processes (S303, S304), for example, a process is performed in which a scroll command including a scroll direction (upward or downward direction) and the scroll amount is set in an output buffer. The scroll amount is a predetermined amount that is set in advance, and is set, for example, to a value that is approximately the same as or slightly smaller than the width (vertical width in this case) of the display 11 in the scroll direction (vertical direction in this case), that is, approximately equivalent to one page in the vertical direction of the screen.
[0067] In step S305, a scroll command output process is performed. In this process, the scroll command set in step S303 or step S304 is output to the browsing application. When the scroll command is output to the browsing application, the browsing application scrolls the content currently displayed on the display 11 upward or downward by a predetermined amount. This makes it possible to scroll the content displayed on the display 11 upward or downward by approximately one page in the vertical direction of the screen.
[0068] For example, in the example of the content of the stacking chair shown in Fig. 5(B), the viewing application scrolls the detailed information 74a displayed on the display 11 downward by approximately one page in the vertical direction of the screen. As a result, as shown in Fig. 5(D), in the detailed information 74a' displayed on the display 11 after scrolling, the display of "★★★ Related Products ★★★" that was located at the bottom of the screen before scrolling is displayed at the top of the screen, and multiple related product information 75a to 75c that were not displayed before scrolling are displayed below "★★★ Related Products ★★★". In addition, as "<<< Detailed product information >>>", the model number, size, weight, color, material, functions, precautions, and part of "<<< Product reviews >>>" are also displayed.
[0069] When the process in step S305 is completed, the series of second operation processes is completed and the process returns to the main routine of the browsing operation assistance application shown in Fig. 3, and the process again proceeds to step S102 to obtain image data captured within a predetermined period from this time onward. In other words, the process prepares for the next hand or other movement made by the user.
[0070] In addition, in the present embodiment, the browsing operation assistance app has been described as being plugged into a general-purpose browsing app and functioning as a subroutine thereof, but this is not limited to this. For example, the browsing app may be configured to have the functions of the browsing operation assistance app (Figures 3 and 4) pre-integrated into it as one function of a dedicated browsing app for smartphone 10.
[0071] As described above, in the smartphone 10 of this embodiment, the browsing operation assistance app installed thereon controls the control unit 20 via the browsing app or directly, so that when the control unit 20 determines that a series of image information contained within a predetermined period captured by the rear camera 16 includes an image of the hand or the like of the user of the smartphone 10 (S104; YES), and when the control unit 20 determines based on the series of image information that the movement of the hand or the like within the predetermined period is a predetermined first action (predetermined action) (S106; YES (first action)), the control unit 20 switches the detailed information 74a currently displayed on the display 11 to the display of related product information 75a to 75c (S206).
[0072] As a result, for example, instead of swiping or tapping on the display 11, the user of the smartphone 10 can switch the detailed information 74a currently displayed on the display 11 to the display of related product information 75a to 75c by moving the hand or the like so that the movement of the hand or the like captured by the back camera 16 becomes the first movement. For example, when the user holds the smartphone 10 with the left hand 90L (or right hand 90R), the user can hold the right hand 90R (or left hand 90L) over the back side 10B of the smartphone 10 and perform a predetermined movement, thereby making it possible to perform a content switching operation without the content or the like displayed on the display 11 being hidden by the right hand 90R (or left hand 90L). Therefore, since the operation until the target information is reached can be performed only by the movement of the right hand 90R (or left hand 90L) on the back side of the display 11, it is possible to improve operability while ensuring visibility.
[0073] Furthermore, in the smartphone 10 of this embodiment, when the control unit 20 determines that the movement of the hand or the like is a second action (another predetermined action) (S106; YES (second action)), it scrolls the screen display of the display 11 upward or downward as a control related to the display mode of the content. The control related to the display mode of the content may, for example, scroll the content currently displayed on the display 11 to the left or right, or change the display magnification of the content (enlarged or reduced display). The content displayed on the display 11 may also be returned to the content at the previous point in time (or one level above in the display hierarchy).
[0074] As a result, for example, instead of swiping the display 11, the user of the smartphone 10 can control various display modes, such as scrolling the content currently being displayed on the display 11 in the vertical or horizontal directions of the display 11, or enlarging or reducing the content, by moving the hand or other object captured by the rear camera 16 in a manner that corresponds to a second predetermined motion.
[0075] Furthermore, in the smartphone 10 of this embodiment, when a plurality of related product information 75 (other contents) are displayed on the display 11, and the control unit 20 judges that the movement of the hand or the like is the first movement a plurality of times in succession and switches the plurality of related product information 75a to 75c in sequence for each judgment, for example, when the content of the related product information 75b to be displayed next on the display 11 among the plurality of related product information 75a to 75c is the same as the displayed content already displayed on the display 11, the control unit 20 switches to the content of the related product information 75c to be displayed next (other content to be displayed after the next displayed content) instead of the content of the related product information 75b (next display content) and displays it on the display 11. This makes it possible to prevent detailed information with the same content as the displayed content from being displayed on the display 11. In other words, it is possible to prevent a situation in which the user feels disappointed or feels like time has been wasted by showing the content that has already been viewed again.
[0076] In the above embodiment of the browsing operation assistance application, when the browsing content, etc. is a recommended product (recommended product), the related product information 75a to 75c is exemplified as other content information associated with the recommended product. However, the associated other content information may be, for example, company information selling the recommended product, other product information sold by the company, etc. In addition, the browsing content, etc. is not limited to the recommended product provided by an Internet search engine using the browsing application, but may be text data, image data, video data, etc. provided from the Internet in association with information on the search results by the search engine. In addition, as long as it is electronic data that can be viewed on the display 11 using the browsing application, it may be, for example, text data, image data, video data, etc. stored in the memory 25 of the smartphone 10.
[0077] In the above embodiment of the browsing operation assistance application, an example has been described in which the screen display of the display 11 is scrolled upward or downward in response to the second action. However, if the control is related to the display mode of the content corresponding to the second action, the content currently displayed on the display 11 may be scrolled leftward or rightward, or the display magnification of the content may be changed (enlarged or reduced). The browsing operation assistance application may be configured to return the content displayed on the display 11 to the content at the previous point in time (or the content one level above in the display hierarchy) in response to the second action. The scrolling in the vertical or horizontal directions may be configured to be continuously scrolled at a constant speed, or the scrolling speed may be changed in response to the distance of the movement of the hand or the like in the second action. For example, the scrolling may be configured to be faster as the vertical movement distance of the hand or the like increases, and slower as the vertical movement distance decreases.
[0078] Furthermore, in the above-described embodiment of the browsing operation assistance application, an example has been described in which the display on display 11 is switched to other content in response to a first action, and an example in which the screen display on display 11 is scrolled upwards or downwards in response to a second action. However, in addition to these, a third action, a fourth action, a fifth action, etc. may be registered, and the display mode of the content being displayed on display 11 may be controlled in response to these actions in the action determination process (S106).
[0079] For example, the third action is a diagonal movement of waving the right hand 90R (or the left hand 90L) diagonally from the upper right to the lower left, the fourth action is a diagonal movement of waving the right hand 90R (or the left hand 90L) diagonally from the upper left to the lower right, and the fifth action is a movement of moving the right hand 90R (or the left hand 90L) back and forth away from or toward the smartphone 10. The browsing operation assistance app may be configured to scroll the content displayed on the display 11 left or right in response to the third action, change the display magnification of the content in response to the fourth action, or return the display on the display 11 to the content one point before (or one level above in the display hierarchy) in response to the fifth action. In addition, when the content displayed on the display 11 is video data, the browsing operation assistance app may be configured to play the video in response to the second action, stop the video being played in response to the third action, fast-forward the video in response to the fourth action, or rewind the video in response to the fifth action.
[0080] [Other inventions] In conventional mobile information terminal devices such as smartphones, a user can swipe or tap the display with a finger, stylus, or touch pen (hereinafter referred to as "finger, etc.") to display a large amount of information even on a small display that prioritizes portability over display size, but there are problems with operation, such as the following: Generally, the surface of the display of a mobile information terminal device is gloss-treated to make images such as photos and videos look beautiful, or is anti-glare-treated to reduce eye fatigue caused by reflection, but in either case, it often feels like a flat surface when touched with a finger, etc. (even anti-glare feels only rough to the touch).
[0081] For this reason, for example, when swiping the display with a finger, it is not possible to give the user the clicking sensation that is transmitted to the finger when the scroll wheel of a mouse is rotated. In other words, in conventional mobile information terminal devices, for example, it is not possible to give the user an intuitive sense of the amount of swiping when swiping the display. Therefore, in [Other Inventions], an operation device (also called a "scroll operation device") is provided that can give the user an intuitive sense of the amount of swiping when swiping the display.
[0082] [Other invention 1] A scroll operation device that is attached to a mobile information terminal device having a touch panel on the surface of a display, a roller portion having elastic protrusions on its outer periphery at substantially equal intervals in the circumferential direction; a frame portion that rotatably supports both ends of the roller portion; a support portion for mounting both ends or one end of the frame portion on the portable information terminal device so that the protrusion comes into contact with the touch panel when the roller portion rotates; A scroll operation device comprising: [Other invention 2] In the scroll operation device according to the above-mentioned Alternative Invention 1, the convex portion is present in a linear form in the axial direction of the roller portion. [Other invention 3] In the above-mentioned other invention 1 or 2, the scroll operation device is characterized in that the frame body portion is provided with a mechanism that imparts a clicking sensation to the tactile sensation of the user's finger touching the roller portion as the roller portion rotates, or a mechanism that imparts a clicking sound to the hearing of the user.
[0083] The above-mentioned another invention 1 is a scroll operation device that is attached to a mobile information terminal device having a touch panel on the surface of the display, and includes a roller unit, a frame unit, and a support unit. The roller unit has elastic convex parts on its outer periphery at approximately equal intervals in the circumferential direction, and both ends of the roller unit are supported by the frame unit so as to be freely rotatable. The frame unit is attached to the mobile information terminal device by the support unit at both ends or one end of the frame unit so that the convex parts of the roller unit come into contact with the touch panel when the roller unit rotates. The rotation means that the roller unit can rotate in both the forward direction and the reverse direction. The scroll operation device of this other invention is also simply called an "operation device" because it is capable of operations other than scrolling (screen switching operation).
[0084] As a result, when a user of a portable information terminal device rotates the roller part with his / her finger like rotating the scroll wheel of a mouse, "the convex parts that the roller part has on the outer periphery at approximately equal intervals in the circumferential direction" discontinuously abut against the touch panel of the portable information terminal device. Therefore, the roller part abuts against the touch panel as if the finger or the like were swiping in short strokes with a predetermined short stroke, and the user's finger is given a contact sensation of the convex parts that occurs in proportion to the amount of rotation as the roller part rotates. The amount of rotation of the roller part is the total amount of swipes made in short strokes (total amount of short swipes), and can correspond to the amount of swipe. In addition, the user can intuitively grasp the amount of rotation of the roller part from the contact sensation of the intermittent convex parts. Therefore, when swiping the display, the amount of swipe can be given to the user intuitively.
[0085] In the above-mentioned second other invention, the convex portion of the roller portion is linear in the axial direction of the roller portion, so that it is possible to swipe the touch panel over a wide range. As a result, even if the scroll operation device is attached to a mobile information terminal device in an inappropriate state in which a part of the roller portion is excessively raised from the touch panel in the axial direction of the roller portion, for example, it is possible to make the convex portion on the remaining part of the roller portion abut against the touch panel. Therefore, even in such an inappropriate attachment state, the amount of swipe can be given to the user intuitively when swiping the display.
[0086] In the third aspect of the present invention, the frame portion includes a mechanism for providing a clicking sensation to the tactile sensation of the user's finger touching the roller portion or a mechanism for providing a clicking sound to the user's hearing as the roller portion rotates. This allows the user to feel the amount of swiping more reliably and sensorily, since the user can receive a clicking sensation or clicking sound through the sense of touch or hearing as the roller portion rotates, in addition to the sensation of contact with the protrusions.
[0087] The scrolling device of the above [Other invention 1] to [Other invention 3] can be embodied as, for example, a scroller 30 as shown in Figures 6 and 7 as one embodiment. The scroller 30 is used by being attached to a smartphone 50 having a touch panel 52 on the surface of a display 51, and mainly includes a roller unit 35, a frame 31, and a click mechanism unit 36.
[0088] The display 51 of the smartphone 50 is, for example, a 6-inch liquid crystal display device with an aspect ratio of 16:9. The touch panel 52 is a transparent film-type pointing device that is approximately the same size as the display 51 and is provided almost on the front of the display surface, and is configured to be able to output position information on the display 51 where the user touches with a finger or a stylus to the control unit of the smartphone 50. The position detection of the touch panel 52 is, for example, a resistive film type, a capacitive type, an ultrasonic surface acoustic wave type, or the like.
[0089] The roller portion 35 is made of a material (e.g., conductive rubber) that is elastic and has good conductivity and is formed in a polygonal prism shape. In this embodiment, the roller portion 35 is formed in a regular hexagonal prism shape with an axial length set shorter than the width direction length of the display 51. As long as the cross-sectional shape is a regular polygon, it may be, for example, a regular octagonal prism shape or a regular decagonal prism shape. On the outer periphery of the roller portion 35, that is, on the peripheral surface portion 35a, corner portions 35b extending linearly in the axial direction are formed at approximately equal intervals in the circumferential direction. In this embodiment, since the roller portion 35 is in the shape of a regular hexagonal prism, the corner portions 35b are formed at six points on the circumference. A shaft member is provided so as to penetrate the center of the roller portion 35, and both ends thereof protrude as shaft portions 35c and 35d. In this embodiment, the corner portion 35b is formed as a linear strip in the axial direction between both ends of the roller portion 35. Therefore, even if the scroller 30 is attached to the smartphone 50 in an inappropriate state, such as when one end of the roller portion 35 is excessively raised from the touch panel 52, the corner portion 35b on the other side of the roller portion 35 can be abutted against the touch panel 52.
[0090] The frame 31 is composed of an upper frame 32 and a lower frame 33, which are configured to sandwich the smartphone 50 in both the front and back directions and the width (left and right) direction of the smartphone 50. The upper frame 32 is configured to support the roller part 35 so that it can rotate freely. The upper frame 32 and the lower frame 33 are made of, for example, high-density polyethylene, polypropylene, ABS resin, etc. The separation distance between the upper frame 32 and the lower frame 33 when the smartphone 50 is sandwiched is such that, as shown in FIG. 7(A), when the roller part 35 of the scroller 30 and the touch panel 12 of the smartphone 50 face each other, the peripheral part 35a of the roller part 35 has a gap SP, and the corner part 35b of the roller part 35 is pressed to such an extent that it is slightly crushed.
[0091] The upper frame 32 is formed in a thin plate shape having a rectangular shape longer than the width direction length of the smartphone 50, and a rectangular long hole (roller accommodating section 32a) capable of accommodating the roller section 35 in a state where it is laid down along the longitudinal direction of the upper frame 32 is formed in the approximate center. A hinge piece 32b constituting a hinge mechanism is formed on one end side of the upper frame 32, and a rotation shaft 34 penetrates the hinge piece 32b. The rotation shaft 34 may be integrally formed with the hinge piece 32b. A claw section 32c protruding in the longitudinal direction of the upper frame 32 is formed on the other end side of the upper frame 32. A click mechanism section 36 described later is accommodated in the side wall of the roller accommodating section 32a located on one end side of the upper frame 32, and the shaft section 35c of the roller section 35 is connected to the click mechanism section 36. Further, a bearing hole 32d capable of rotatably supporting a shaft portion 35d of the roller portion 35 is formed in the side wall of the roller accommodating portion 32a located on the other end side.
[0092] The lower frame 33 is formed in a rectangular shape having L-angle portions 33b and 33c at both ends that can hold both longitudinal ends of the upper frame 32. A cushioning material 39 is attached to the inside of the lower frame 33, that is, the inner surface of the smartphone storage portion 33a and the L-angle portions 33c and 33d. The cushioning material 39 is, for example, a sheet-like rubber sponge, and is attached to the inner surface of the lower frame 33 via double-sided tape or the like. The short-side width of the lower frame 33 is set to be almost the same as that of the upper frame 32. The L-angle portion 33b on one end side functions as a hinge piece that constitutes a hinge mechanism together with the hinge piece portion 32b of the upper frame 32. The L-angle portion 33c on the other end side is formed with a groove portion 33d that can engage the claw portion 32c of the upper frame 32. The claw portion 32c and the groove portion 33d constitute a locking mechanism.
[0093] The click mechanism 36 is a mechanism that gives a clicking sensation to the tactile sense of the user's finger touching the roller part 35 as the roller part 35 rotates or gives a clicking sound to the user's hearing, and has a known configuration that is used, for example, in a mouse wheel. Although not shown, specifically, the click mechanism 36 is composed of a disk part having a plurality of linear grooves radially from the center and connected to the shaft part 35c at one end of the roller part 35 as a rotating shaft, a bearing hole that rotatably supports the disk part, and a flexible claw that is fixed so as to be able to engage with the plurality of linear grooves of the rotating disk part. When the disk part rotates integrally with the roller part 35, the flexible claw overcomes the unevenness so as to cross the plurality of linear grooves of the disk part one after another as the disk part rotates or turns. As a result, the click mechanism 36 generates a clicking sound or vibration, so that the user is given a clicking sound or clicking sensation associated with the rotation of the roller part 35.
[0094] By configuring the scroller 30 in this manner, as shown in Fig. 6(D), the upper frame 32 and the lower frame 33 can rotate relatively around the rotation shaft 34 by a hinge mechanism configured on one end side of the upper frame 32 and the lower frame 33, and the smartphone 50 can be maintained in a sandwiched state by a lock mechanism configured on the other end side of the upper frame 32 and the lower frame 33. This makes it possible to attach the scroller 30 to the smartphone 50 in a positional relationship in which the roller part 35 can come into contact with the touch panel 52 (Figs. 6(A) to (C)).
[0095] The scroller 30 attached to the smartphone 50 functions as shown in FIG. 7. As shown in FIG. 7(A), when the user of the smartphone 50 rotates the roller part 35 by pulling it downward with his / her finger 91, the roller part 35 rotates in the direction of the arrow in the figure. Then, in the state of FIG. 7(A), a gap SP is formed between the peripheral surface part 35a of the roller part 35 and the touch panel 52, and the two (roller part 35 and touch panel 52) are not in contact with each other, but as the roller part 35 rotates, the corner part 35x of the roller part 35 comes into contact with the touch panel 52 (FIG. 7(B)). Then, by rotating the roller part 35 in the same direction, the corner part 35x' is pressed against the touch panel 52 and crushed (FIG. 7(C)), and the crushed corner part 35x' slides in the direction of the arrow, and by further rotating, the crushed corner part 35x' returns to its original shape (corner part 35x) (FIG. 7(D)).
[0096] Since the peripheral surface portion 35a of the roller portion 35 has a plurality of corners 35b formed at equal intervals in the circumferential direction, the plurality of corners 35b come into contact (contact) with the touch panel 52 intermittently (discontinuously) through three contact processes of point → surface → point, such as corner 35x → crushed corner 35x' → corner 35x, and the planar crushed corner 35x' moves on the touch panel 52 to slide upwards on the smartphone 50. As a result, the roller portion 35 comes into surface contact with the touch panel 52 as if the finger or the like were swiping toward the upper side of the smartphone 50 in short strokes with a predetermined short stroke, and the user's finger is given a contact sensation of the corners 35b that occurs in proportion to the amount of rotation as the roller portion 35 rotates. The amount of rotation of the roller portion 35 is the total amount of swipes made in short strokes (total amount of short swipes) and can correspond to the amount of swiping. Also, the user can intuitively grasp the amount of rotation of the roller portion 35 from the intermittent contact feeling of the corners 35b.
[0097] Therefore, when swiping the display 51, the user can sense the amount of swiping. Furthermore, the click mechanism 36 allows the user to feel a click or a click sound through touch or hearing with the rotation of the roller unit 35 in addition to the contact sensation of the corner 35b, so that the amount of swiping can be more reliably sensed. Furthermore, since the corner 35b of the roller unit 35 is formed as a line in the axial direction between both ends of the roller unit 35, even if the scroller 30 is attached to the smartphone 50 in an inappropriate state in which one side of the roller unit 35 is excessively raised from the touch panel 12, the amount of swiping can be sensed by the user when swiping the display 51.
[0098] The corners 35b of the roller portion 35 may be continuous (striated) or discontinuous (dot-like) in the axial direction of the roller portion 35. The roller portion 35 may be a cylinder with a curved circumferential surface, in which case it has a convex portion that is continuous (striated) or discontinuous (dot-like) in the axial direction. The support portion does not need to have both ends connected like the lower frame 33. It may be made up of two members with a U-shape facing each other on the opening side (open square bracket symbol "[" and closed square bracket symbol ]" facing each other). It may also be configured such that these two members are connected to both ends of the upper frame 32.
[0099] 7, the case where the user of the smartphone 50 pulls the roller unit 35 downward on the smartphone 50 with the finger 91 has been described, but even when the roller unit 35 is rotated so as to be pushed upward on the smartphone 50, the roller unit 35 and the click mechanism unit 36 function in the same manner as in the case of "pulling downward." However, in this case, the roller unit 35 rotates in the opposite direction to the arrow in the figure, so that the crushed corner portion 35x' slides downward on the touch panel 52. As a result, the roller unit 35 comes into surface contact with the touch panel 52 as if the finger or the like were swiping downward on the smartphone 50 in short predetermined strokes.
[0100] In the above-mentioned other aspect of the invention, a predetermined display control is performed by abutting the corner of the roller part against the touch panel (by detecting that the roller part has come into contact with the touch panel). Instead of this aspect, a scroller may be provided at a position where a camera (front camera) can detect (take an image of) the rotation (amount of rotation, angle of rotation) of the roller part (for example, the camera may be provided below the roller part), so that the amount of rotation of the roller part can be recognized and the display can be controlled even if the roller part and the touch panel (display screen) are not in contact with each other. Note that the roller part may have a smaller diameter than that shown in FIG. 6 so that the roller part does not come into contact with the touch panel (display screen) when rotated, thereby preventing the touch panel (display screen) from being damaged by the rotation operation of the roller part.
[0101] In the above-mentioned other aspect of the invention, by rotating the roller unit, it is possible to scroll the displayed content in the vertical direction of the display, or to switch the displayed content to another content. Also, by performing a predetermined input operation (predetermined input) in an application or the like installed on the smartphone, a scrolling operation of the content (first display control) can be performed in response to the "rotation operation of the roller unit", or Alternatively, the "rotation operation of the roller portion" may be associated with either the "first display control" or the "second display control" for switching the content. This allows the user to select which operation is to be performed by the "rotation operation of the roller portion." [Explanation of symbols]
[0102] 10,50 Smartphone (mobile information terminal device), 10B back side, 10F front side (surface having a display), 11,51 Display, 12,52 Touch panel, 15 Front camera, 16 Back camera (camera on the back side), 20 Control unit (image determination means, operation determination means, content switching means, display control means, computer for controlling the mobile information terminal device), 30 Scroller (scroll operation device), 32 Upper frame (frame body part, support part), 33 Lower frame (support part), 35 Roller part, 35b,35x,35x' Corner part (convex part), 35c,35d Shaft part (both ends of the roller part), 36 Click mechanism part (mechanism for providing a clicking sensation, mechanism for providing a clicking sound), 71a,71b,72 Reduced image information, 73 Summary information, 74a,74b,74a' Detailed information (content) 75a,75b,75c Related product information (other content), 90R right hand 90R (human hand), 90L left hand 90L (human hand), 91 finger 91, S103, S104 (image determination means), S105, S106 (motion determination means), S107 (content switching means), S108 (display control means).
Claims
1. A mobile information terminal device having a camera on a rear side of a surface having a display, a motion determination means for determining whether or not a motion of a human hand or finger (hereinafter referred to as "hand, etc.") within a predetermined period of time is a predetermined motion based on a series of image information captured by the camera when the series of image information includes an image of the human hand or finger (hereinafter referred to as "hand, etc."); a content switching means for switching the content currently being displayed on the display to another content when the motion determining means determines that the motion of the hand or the like is the predetermined motion, When a predetermined search target is input and a search result is obtained, and when a specific content selected from the plurality of contents is displayed on the display, if the motion determining means determines that the motion of the hand or the like is the predetermined motion, the content switching means switches the content currently displayed on the display to another content. A portable information terminal device comprising:
2. A computer that controls a mobile information terminal device having a camera on the back side of a surface having a display, a motion determination means for determining, when a series of image information captured by the camera within a predetermined period includes an image of a human hand or finger (hereinafter referred to as "hand, etc."), whether or not a movement of the hand, etc. within the predetermined period is a predetermined motion based on the series of image information; a program that functions as a content switching means that switches a content currently being displayed on the display to another content when the motion determination means determines that the motion of the hand or the like is the predetermined motion, In a case where there are a plurality of the other contents, and the content switching means determines a plurality of times in succession that the movement of the hand or the like is the predetermined movement, and the display of the plurality of the other contents is switched in sequence for each determination, When a next display content to be displayed next on the display among the plurality of other contents has the same content as a content already displayed on the display, the content switching means switches to the other content to be displayed next to the next display content instead of the next display content and displays the same on the display. A program characterized by:
Citation Information
Patent Citations
Camera-based user input for compact devices
JP2010526391A
Electronic apparatus
JP2016058061A
List scrolling, document translation, scaling, and rotation on touchscreen displays.
JP2010515978A
