Operating device

Abstract

The present invention provides an operating device that allows the user to intuitively input the amount of swipe when swiping a display. [Solution] A scroller 30 (scrolling device) used attached to a smartphone 50 (portable information terminal device) equipped with a touch panel 52 on the surface of a display 51, comprising: a roller portion 35; an upper frame 32 (frame portion) that rotatably supports both ends of the roller portion; a lower frame 33 (support portion) that attaches both ends or one end of the frame portion to the portable information terminal device; and a detection means for detecting the rotation of the roller portion, thereby controlling the display of the display in accordance with the rotation of the roller portion.

Description

Technical Field

[0001] The present invention relates to an operating device that is used by being mounted on a portable information terminal device having a camera on the back side of a surface having a display, and can be particularly applied to smartphones and the like.

Background Art

[0002] In recent years, portable information terminal devices have touch panels. A touch panel is a transparent film type pointing device arranged on a display, and may also be called a touch screen, a touch screen, a contact screen, or the like. In a portable information terminal device having such a touch panel, for example, as disclosed in Patent Document 1, after a finger or a stylus is brought into contact with the display (accurately, the touch panel on the display), the finger or the like is then moved so as to slide in a predetermined direction, whereby the content or the like displayed on the display can be moved in that direction or a predetermined operation can be performed on the device. Such an operation is called a swipe or a slide gesture.

[0003] By the way, a smartphone is a representative example of recent portable information terminal devices, and in addition to the original telephone function, it also has functions such as being able to use the Internet or take pictures of scenery or the like with an attached camera, so it has high utility value as a multifunctional device. In particular, in a scene where the Internet is used, it is also possible to view the content of a searched website or purchase a product being sold if the website is an EC (Electronic Commerce) site. Therefore, although it is desirable that more information be displayed on the display of a smartphone, there is a limit to the size of the display in consideration of portability.

[0004] Therefore, in smartphones and the like, by utilizing technologies such as those disclosed in Patent Document 1 below, it is possible to display other content by moving some of the content that cannot be displayed on the screen at once due to the limitations of the display size in the direction of a swipe operation, thereby moving it outside the display area. As a result, for example, a smartphone user can swipe the display left or right or up or down to scroll left or right or up or down and visually check the information of numerous content or products that are displayed as reduced images on the display like thumbnails. Furthermore, for content or products for which the user wants to know more details, they can tap the reduced display to display the detailed information on the display. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Special Publication No. 2010-515978 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] In this way, mobile information terminal devices such as smartphones allow users to swipe or tap the display with their fingers, enabling the display of a large amount of information even on small displays where portability is prioritized over display size. However, there are the following problems in terms of operation. Generally, the surface of the display of a mobile information terminal device is either glossy to make images such as photos and videos look good, or non-glare to reduce eye strain caused by reflections, but in either case, it often feels like a flat surface to the touch of a finger (even non-glare surfaces only feel slightly rough).

[0007] Therefore, for example, when swiping the display with a finger, it is not possible to provide the same click sensation to the finger as when rotating a mouse scroll wheel. In other words, conventional portable information terminal devices have not been able to intuitively convey the amount of swipe to the user when, for example, swiping the display.

[0008] The present invention aims to provide an operating device that can intuitively provide the user with the amount of swipe. [Means for solving the problem]

[0009] The operating device of the first invention for achieving the above objective is: An operating device used by being attached to a portable information terminal device that has a touch panel on the surface of the display, The roller section, A frame that rotatably supports both ends of the roller section, Support portion for attaching both ends or one end of the frame to the portable information terminal device, The system includes a detection means for detecting the rotation of the roller portion, By rotating the roller portion, the display of the display is controlled according to the rotation. It is characterized by the following:

[0010] Furthermore, the operating device of the second invention for achieving the above objective is the operating device of the first invention, The outer circumference of the roller portion is provided with elastic protrusions. The aforementioned protrusions are characterized in that they are present in a linear fashion in the axial direction of the roller portion.

[0011] Furthermore, the operating device of the third invention for achieving the above objective is the operating device of the first or second invention, The frame portion is characterized by comprising a mechanism that, as the roller portion rotates, provides a clicking sensation to the user's fingers that touch the roller portion, or a clicking sound to the user's hearing. [Effects of the Invention]

[0012] According to the operating device of the present invention, the user of a portable information terminal device can intuitively grasp the amount of rotation of the roller part by rotating the roller part with their finger, similar to how a mouse scroll wheel is rotated. [Brief explanation of the drawing]

[0013] [Figure 1] This diagram shows a smartphone (hereinafter referred to as "smartphone") according to one embodiment to which the mobile information terminal device and its program of the present invention are applied, and an explanatory diagram showing the hands and fingers of a user holding and operating it. [Figure 2] This is a block diagram showing an example of the functional configuration of the smartphone according to this embodiment. [Figure 3] This flowchart shows the processing flow of the browsing assistance application program executed by the smartphone of this embodiment. [Figure 4] Figure 3 is a flowchart showing the flow of subroutines in the browsing assistance application program, where (A) represents the first operation process and (B) represents the second operation process. [Figure 5] This is an explanatory diagram showing examples of content displayed on the smartphone screen of this embodiment. (A) shows an example of the display of search results for "chair", (B) shows an example of the display when a chair content is selected from the search results in (A), (C) shows an example of the display when the user moves from (B) to related product content by the first operation process, and (D) shows an example of the display when the display in (B) is scrolled upwards by the second operation process. [Figure 6] This diagram shows a smartphone with a scroller attached. (A) is a plan view of the smartphone and scroller seen from the front side of the smartphone, (B) is a cross-sectional view showing the smartphone and scroller cut along the line 6B-6B shown in (A), (C) is a side view and bottom view of the smartphone and scroller seen from the direction of arrow 6C shown in (A), and (D) is a bottom view showing the smartphone with the scroller being attached or removed in the side view of (C). [Figure 7] In the cross-sectional view corresponding to Fig. 6(B), it is an explanatory diagram showing an operation example of the roller part constituting the scroll bar. (A) represents the case where the user rotates the roller part counterclockwise in the figure with a finger. (B) represents the state immediately after the corner part of the roller part touches the touch panel from the state of (A). (C) represents the state where the corner part is pressed against the touch panel and crushed as the rotation of the roller part proceeds from the state of (B). (D) represents the state immediately after the crushed corner part returns to its original shape as the rotation of the roller part proceeds from the state of (C).

Embodiments for Carrying out the Invention

[0014] Next, each embodiment to which the portable information terminal device and its program of the present invention are applied will be described with reference to each drawing. First, the configuration of the smartphone 10 of this embodiment will be described with reference to Figs. 1 and 2.

[0015] As shown in Fig. 1, the smartphone 10 is a commercially available one, and has a main body case 10a having a rectangular thin plate shape that can be held by the user with one hand (left hand 90L in the figure), and is composed of a display 11, a touch panel 12, a control unit 20, etc. provided inside or outside the main body case 10a.

[0016] On the front surface 10F of the smartphone 10, the display 11 is mounted in a rectangular frame hole formed in the main body case 10a, and a speaker 14 and a front camera 15 are provided. A back camera 16 is provided on the back surface 10B, and a microphone 13 is provided on the lower end side surface. The touch panel 12 is provided on the surface of the display 11 and is provided substantially integrally with the display 11. Therefore, it should be noted that in the figure, the display 11 and the touch panel 12 are represented as overlapping with the same rectangular shape (the part colored in gray).

[0017] As shown in FIG. 2, the display 11, the touch panel 12, the front camera 15, etc. are electrically connected to and controlled by the control unit 20 housed in the main body case 10a. The control unit 20 is a main board having an MPU 21, a memory 25, etc. In addition to the display 11, etc., a wireless unit 17, various sensors 18, a battery 19, and further, a memory card, a SIM card, etc. (not shown) are electrically connected.

[0018] The display 11 is, for example, a 6-inch liquid crystal display device with an aspect ratio of 16:9, and is configured to be able to display the image information output from the control unit 20. The touch panel 12 is a transparent film type pointing device provided substantially in front of the display surface and having substantially the same size as the display 11, and is configured to be able to output the position information on the display 11 where the user touches with a finger or a stylus to the control unit 20. The touch panel 12 may also be called a touch screen, a contact screen, etc. For position detection, for example, a resistive film method, a capacitance method, an ultrasonic surface acoustic wave method, etc. are used.

[0019] The microphone 13 and the speaker 14 are used during a call of the telephone function, or for recording or audio output by an application program (also referred to as application software. These are collectively referred to as "apps" below). The microphone 13 is configured to be able to collect the user's voice, etc. and output acoustic information to the control unit 20, and the speaker 14 is configured to be able to output the acoustic information output from the control unit 20 to the outside.

[0020] The front camera 15 and the rear camera 16 are imaging devices having a solid-state image sensor such as a CCD or CMOS and an optical lens, and are configured to output captured image information to the control unit 20. The front camera 15 is located on the front side 10F of the smartphone 10 and is configured to capture, for example, the user's face. The rear camera 16 is located on the back side 10B of the smartphone 10 and is configured to capture, for example, a landscape. The front camera 15 is sometimes called the in-camera, and the rear camera 16 is sometimes called the out-camera.

[0021] The wireless unit 17 is a wireless transceiver configured to enable voice and data communication via the wireless communication network of a mobile network operator (MNO), and the control unit 20 controls calls and data communication. A SIM card (not shown) stores identification information unique to the smartphone 10 used during wireless communication. The various sensors 18 include a GPS sensor, an accelerometer, an illuminance sensor, a temperature sensor, etc., and all are configured to output detection data to the control unit 20.

[0022] The battery 19 is a rechargeable secondary battery (e.g., a lithium-ion battery) that can be charged by DC power supplied from an external source, and is configured to supply power to the control unit 20, the display 11, etc. Charge and discharge information of the battery 19 is also output to the control unit 20, and the control unit 20 is configured to manage charging and discharging, for example.

[0023] The control unit 20 consists of an MPU 21, a GPU (Graphics Processing Unit) 23, memory 25, etc. The MPU 21 is a microcomputer and is configured to control hardware such as the display 11 mentioned above, or to run browsing applications and browsing operation assistance applications, which will be described next, by executing an OS (basic operating system) such as Android® or iOS® stored in the memory 25. The GPU 23 is used for image data analysis processing, which will be described later. A typical smartphone 10 configured in this way has problems such as the fact that the operation required to reach the desired information is complicated and may be troublesome for the user, as described in the [Problems to be Solved by the Invention] section, and that the visibility of the display 11 may be impaired.

[0024] Therefore, the smartphone 10 of this embodiment has a browsing operation assistance application installed that can solve these problems. With this browsing operation assistance application, for example, when the back camera 16 captures (images) a movement of the hand, such as shaking the right hand 90R from side to side (first movement), the smartphone 10 switches the content currently displayed on the display 11 to other displayed content. Also, when the back camera 16 captures a movement of the hand, such as shaking the right hand 90R up and down (second movement), the smartphone 10 controls the smartphone 10 to scroll the content currently displayed on the display 11 upwards (or downwards). From here on, we will explain this browsing operation assistance application with reference to Figures 3 to 5.

[0025] Furthermore, the files for the browsing assistance application are downloaded, for example, via data communication over a wireless communication network performed by the functions of the wireless unit 17 and installed in the memory 25 of the control unit 20, or they may be pre-installed in the memory 25. The browsing assistance application may run independently, but it may also function in conjunction with, for example, a web browser or file browser (file manager). That is, the browsing assistance application may function as a subroutine of a general-purpose plugin-compatible browsing application (for example, Google Chrome®, Safari®, Microsoft Edge®, Internet Explorer®, etc.). In this case, the browsing assistance application is an executable file that conforms to a predetermined plugin format and is executed each time it is plugged into and called by these browsing applications.

[0026] Furthermore, prior to running the browsing assistance application, the first and second actions described above, as well as the right hand 90R and left hand 90L that perform these actions, must be registered in the memory 25 of the control unit 20. The flow of the registration process will not be explained here, but it is performed by the functions of the browsing assistance application. For example, as the first action to switch the content currently displayed on the display 11 to other display content, the user registers the action of swinging their right hand 90R (or left hand 90L) from side to side, and the right hand 90R (or left hand 90L) that performs this action. Also, as the second action to control the display mode of the content currently displayed on the display 11 (for example, scrolling upwards), the user registers the action of swinging their right hand 90R (or left hand 90L) up and down, and the right hand 90R (or left hand 90L) that performs this action.

[0027] The direction in which the right hand 90R (or left hand 90L) is swung follows the up, down, left, and right directions pre-set (defined) in the rear camera 16. For example, as shown in Figure 1, when the rear camera 16 is positioned above the smartphone 10 and the display 11 is facing the user, and the user is holding the smartphone 10, the smartphone 10 is set so that the upper part of its longitudinal direction is "up", the lower part of its longitudinal direction is "down", the right part of its short direction is "right", and the left part of its short direction is "left". Therefore, for example, when the smartphone 10 is held by the user after being rotated 90 degrees clockwise from the state shown in Figure 1, the user's movement of swinging their right hand 90R upwards (downwards) is analyzed as a movement of swinging their right hand 90R to the left (right), and the movement of swinging their right hand 90R to the left (right) is analyzed as a movement of swinging their right hand 90R downwards (upwards) by the motion analysis processing (S105) described later.

[0028] Furthermore, instead of registering the right hand 90R or left hand 90L, you may register the movements of those fingers (for example, the index finger or thumb) or those fingers themselves. Alternatively, instead of the right hand 90R or left hand 90L, you may register, for example, the movements of a stylus or touch pen held in the right hand 90R, or the stylus that causes those movements. Incidentally, the right hand 90R, left hand 90L, those fingers, and the stylus or touch pen held in those hands will be collectively referred to as "hand, etc." in the following section [Modes for Carrying Out the Invention].

[0029] As shown in Figure 3, the browsing operation assistance application executed by the control unit 20 first undergoes a predetermined initialization process in step S101. This initialization process includes, for example, clearing the work area and flags of the memory 25, and starting video recording by the back camera 16. The captured image data is video data within a predetermined period (for example, 1 to 2 seconds).

[0030] In the next step, S102, image data acquisition processing is performed. This image data is a series of image information captured by the rear camera 16 within a predetermined period, and consists of a series of still image information consisting of multiple still images captured (extracted) from video data within the predetermined period by frame-by-frame, frame-by-frame, or frame-by-frame. If video analysis is possible in the image data analysis processing in step S103, the series of image information may also be video captured video information for the predetermined period.

[0031] For example, as shown in Figure 1, when a user is holding the smartphone 10 with their left hand 90L and looking at the display 11, they often swipe or tap the touch panel 12 with their right hand 90R. Therefore, the browsing assistance application captures the movement of the free right hand 90R (or left hand 90L) when the smartphone 10 is being held with the left hand 90L (or right hand 90R) using the back camera 16.

[0032] If the user points their palm towards the back camera 16 and, for example, shakes their right hand 90R from side to side (in the direction of the arrows shown in Figure 1), a series of still image information capturing the movement of the right hand 90R, 90R' is acquired in step S102. Therefore, in the subsequent image data analysis process in step S103, an analysis is performed to determine whether or not image information of the right hand 90R is included in this series of still image information. This image data analysis process (S103) is mainly performed by the GPU 23.

[0033] In the image data analysis process in step S103, for example, pattern matching is used as a known image analysis method. In image analysis using pattern matching, for example, image information of the user's right hand at 90R is registered in memory 25 in advance as template image information, and a matching determination is made between each of the series of still image information to be compared and the template image. In this image data analysis process, for example, the proportion of still image information that is determined to match the template image information out of the series of still image information is obtained.

[0034] In step S104, a process is performed to determine whether the series of still image information includes the user's hand (right hand 90R in the example of Figure 1). In the previous step S103, the percentage of still image information that was determined to match the template image information is obtained. Here, if that percentage is greater than or equal to a predetermined value, it is determined that the series of still image information includes the hand, etc. If it is less than the predetermined value, it is determined that the hand, etc. does not exist. The predetermined value is set in advance based on the analysis accuracy of the image data analysis process in step S103, and can be arbitrarily set in the range of 70 to 90%, for example.

[0035] If the determination process in step S104 determines that a hand or similar object is included in the series of still image information (S104; YES), the motion analysis process in the following step S105 is performed. On the other hand, if it is determined that a hand or similar object is not included in the series of still image information (S104; NO), there is a high probability that the image taken by the rear camera 16 contains something other than, for example, the right hand 90R. Therefore, the process returns to step S102 to acquire image data captured within a predetermined period from this point onward.

[0036] Step S105 performs motion analysis. This process analyzes the movement (action) of images such as the right hand 90R included in the series of still image information acquired in step S102. Similar to the image data analysis process in step S103, this process is mainly performed by the GPU23, using, for example, pattern matching as a known image analysis method. Note that if the image data analysis process in step S103 also performs the same analysis as the motion analysis process in step S105, the motion analysis process (S105) may be omitted.

[0037] As described above, in this embodiment, the first and second actions are pre-registered in the memory 25 of the control unit 20. Therefore, for example, if the first action registered is the user swinging their right hand 90R, etc., from side to side, the action analysis process uses multiple still images captured by frame-by-frame playback of the movement of the right hand 90R, etc., as template image information, and performs a matching determination between each of the series of still image information to be compared and the template image. Similarly, if the second action registered is the user swinging their right hand 90R, etc., up and down, the process uses multiple still images captured by frame-by-frame playback of the movement of the right hand 90R, etc., as template image information, and performs a matching determination between each of the series of still image information to be compared and the template image.

[0038] In this motion analysis process (S105), the matching result is obtained as to whether it matches either the first or second motion, or whether it does not match either the first or second motion. Therefore, in the subsequent motion determination process in step S106, it is determined whether or not it is a registered motion (first motion or second motion), and if it is a registered motion, it is determined whether or not it is the first or second motion.

[0039] Furthermore, in the motion analysis process performed in step S105, the direction and speed of the swing of the hand, such as the right hand 90R, are analyzed. For example, if the right hand 90R is swung from side to side, the direction information "left and right" is obtained through the analysis. By comparing the speed at which the hand moves from left to right (to the right) with the speed at which it moves from right to left (to the left), information on the relative magnitude of the velocities in the left and right directions is also obtained through the analysis. Similarly, if the right hand 90R is swung up and down, the direction information "up and down" is obtained through the analysis, and by comparing the speed at which the hand moves upward with the speed at which it moves downward, information on the relative magnitude of the velocities in the up and down directions is also obtained through the analysis. This information on the direction and speed of the swing is used in the second motion processing (S108, Figure 4(B)) described later.

[0040] If the motion determination process in step S106 determines that the motion based on the image of the right hand 90R matches the first motion (for example, the motion of shaking the right hand 90R from side to side) (S106; YES (first motion)), the process proceeds to the first motion processing in step S107. If the motion based on the image of the right hand 90R matches the second motion (for example, the motion of shaking the right hand 90R up and down) (S106; YES (second motion)), the process proceeds to the second motion processing in step S108. However, if the motion based on the image of the right hand 90R (for example, the motion of shaking the right hand 90R diagonally) does not match either the first or second motion, the motion captured by the back camera 16 is different from a predetermined motion. In other words, there is a high possibility that the user's motion of the right hand 90R is incorrect. Therefore, the process returns to step S102 to acquire image data captured within the predetermined period from this point onward.

[0041] The flowcharts for the first operation process (S107) and the second operation process (S108) are shown in Figure 4. Furthermore, an explanatory diagram showing an example of the content displayed on the display 11 as a result of these processes is shown in Figure 5.

[0042] As shown in Figure 4(A), in the first operation process (S107), the content information acquisition process is first performed in step S201. Content refers to the content or substance of information, and content information refers to information about the content (for example, address information such as the URL where the content exists, file name, file type, file size, creation date, information about other content to which it is associated, etc.). Here, information about the content currently displayed on display 11 is acquired from the browsing application.

[0043] For example, as shown in Figure 5(A), when a user enters "chair" into the search box 70 of a search engine provided via a browsing app on a smartphone 10, the search results are displayed on the screen of the display 11. In this example, thumbnail images 71a, 71b, and 71c of multiple recommended products are displayed horizontally from left to right on the screen in order of recommendation. In this figure, some of the thumbnail images 71a, etc., that cannot be fully displayed on the screen due to the display size, as well as some of the hidden information 79, are represented in light gray on the right edge of the display 11. The hidden information 79 can be viewed by scrolling the screen to the left.

[0044] Furthermore, below the reduced image information 71a~71c on the screen, summary information 73 of an online shopping site selling "chairs" is displayed as text. Although not shown in Figure 5(A), multiple summary information 73 of online shopping sites and other sites not displayed on display 11 are obtained from the internet and can be displayed in order of search ranking as part of the search results. Summary information 73 not shown in the figure can be displayed by scrolling the screen upwards.

[0045] After these reduced image information 71a~71c and summary information 73 are displayed on the display 11, if the user selects the reduced image information 71a of the "office stacking chair" via the touch panel 12 with their finger, stylus, or touch pen, then, for example, as shown in Figure 5(B), the detailed information 74a of the stacking chair will be displayed as content. In this case, in step S201, information regarding the detailed information 74a of the stacking chair displayed on the display 11 is acquired as content information. Then, based on this acquired content information, in the subsequent step S202, a process is performed to determine whether or not there is any next content, i.e., content that has not yet been viewed.

[0046] In the example of the stacking chair content shown in Figure 5(B), when scrolling upwards, as shown in Figure 5(D), multiple related product information items 75a, 75b, and 75c ("transport cart," "set of 2 chairs," and "set of 5 chairs") are linked as other content information in the "★★★Related Products★★★" section of the detailed information 74a' that subsequently appears on the display 11. In this way, if other content information is linked to the content currently being viewed or the content that can be viewed by scrolling (hereinafter referred to as "viewed content, etc."), then there is next content. Therefore, in step S202, it is determined that there is next content (S202; YES), and in the next step S203, the process of obtaining information about the next content from the viewing application or from the internet via the viewing application is performed.

[0047] Then, in the following step S205, a process is performed to determine whether or not the content is the same as previously displayed content. In other words, if the next content has been viewed in the current or previous instance, and that previously viewed content is displayed again on the display 11, the user will see content they have already viewed, which may give the user a feeling of disappointment or that they have wasted their time. Also, if the content α being viewed is linked to the next content β, it will create an infinite viewing loop of content α → content β → content α → content β →…

[0048] Therefore, in this embodiment, step S205 is used to determine whether the next content is the same as previously viewed content. If the next content is the same as previously viewed content (S205; YES), the process returns to step S202 to determine whether there is another next content. If it is not the same (S205; NO), the process proceeds to step S206.

[0049] The determination of whether these contents are the same is made based on content information such as the content address, file name, file size, and creation date. If this content information is identical, it is determined that they are the same content. Furthermore, if previously viewed content is stored in the control unit 20's cache memory or cookie file, and the same content exists in these cache memories, it is determined that they are the same content.

[0050] This prevents previously viewed content from being displayed on display 11 again, thus avoiding the problem described above. However, if the benefit of allowing previously viewed content to be viewed again outweighs the risk of this problem, then the judgment process (S205) is unnecessary. In this case, the judgment process in step S205 is deleted.

[0051] On the other hand, if step S202 determines that there is no next content (S202; NO), that is, if other content information such as related product information 75a~75c of the viewed content is not associated, the process moves to step S204. Then, after the process of obtaining end information (predetermined data or predetermined flags) indicating that there is no next content is performed from memory 25, the process moves to step S206.

[0052] Furthermore, if no other content information is associated with the content information obtained in step S201, for example, another piece of content that exists in the same display hierarchy (or data hierarchy) as the content currently displayed on display 11 may be used as the next content, and it may be determined in step S202 that there is next content.

[0053] For example, if no other content information is linked to the detailed information 74a of the stacking chair shown in Figure 5(B), the user may return to the previous content (or one level up in the display hierarchy) and select other thumbnail images 71b and 71c displayed in the same row as the thumbnail image information 71a of the stacking chair shown in Figure 5(A) as the next content. For example, if the user selects the "GAMING CHAIR" content of thumbnail image information 71b, which is displayed to the right of thumbnail image information 71a, in accordance with the recommended order (from left to right on the screen), the detailed information of "GAMING CHAIR" will be displayed in the same display format as the detailed information 74a of the stacking chair shown in Figure 5(B) and the detailed information 74b of the transport cart shown in Figure 5(C), just as when the thumbnail image information 71 of "GAMING CHAIR" is selected on the same screen.

[0054] In step S206, display information output processing is performed. This process outputs the next content information obtained in step S203, or the end information obtained in step S204, to the browsing application. When the next content information is output to the browsing application, the browsing application retrieves data for other content from the internet (or from memory 25 if data for other content has already been downloaded through pre-loading, etc.) based on the next content information, and performs processing to switch the content currently displayed on the display 11 to the display of other content. In other words, the browsing application, upon receiving instructions from the browsing operation assistance application, switches the content displayed on the display 11 to other content in response to the first action taken by the user.

[0055] For example, in the detailed information 74a' of the stacking chair shown in Figure 5(D), thumbnail images of the contents for "transport cart," "set of 2 chairs," and "set of 5 chairs" are linked as multiple related product information items 75a to 75c. Therefore, the browsing application, upon receiving instructions from the browsing operation assistance application, displays these contents on the display 11 as other contents in the order they appear on the display 11 (from left to right on the screen). In this example, the related product information 75a displayed on the far left of the screen is a thumbnail image of the "transport cart" content. As the content linked to it (next content), the detailed information 74b for "transport cart for stacking chairs" is displayed on the display 11 in response to the first action performed by the user (see Figure 5(C)).

[0056] In contrast, if end information is output to the browsing application, there is no next content associated with the browsing content, and the browsing application cannot obtain other content information, so it cannot switch the display on display 11 to other content. Therefore, in such cases, for example, even if the user performs the first action, the browsing application will either maintain the content currently displayed on display 11 or display information on display 11 indicating that there is no content to switch to.

[0057] Once the processing in step S206 is complete, the series of first operation processes is finished and the system returns to the main routine of the browsing operation assistance application shown in Figure 3. Then, the process moves back to step S102 to acquire image data captured within the predetermined period from this point onward. In other words, it prepares for the next hand or other movement the user will make.

[0058] As shown in Figure 4(B), in the second motion processing (S108), the first step, S301, is used to acquire hand information. The hand information is the direction and speed of the swing of the hand, which is obtained from the motion analysis processing (S105) shown in Figure 3. Generally, when a person swings their hand with a sense of direction, the speed at which they swing in that direction is faster than the speed at which they swing in the opposite direction to return the hand after the initial swing. Therefore, from this information, it is possible to determine that the hand is being swung in the direction of greater speed (upward if up > down, downward if up < down), that is, the direction of the hand's movement.

[0059] Therefore, if the determination process in the next step S302 determines that a hand or other object is being waved upwards (S302; upwards), the process moves to step S303 to perform the upward scroll setting process. Also, if the determination process determines that a hand or other object is being waved downwards (S302; downwards), the process moves to step S304 to perform the downward scroll setting process. In these setting processes (S303, S304), for example, a scroll command including the scroll direction (upwards or downwards) and the scroll amount is set in the output buffer. The scroll amount is a predetermined amount set in advance, for example, a value that is approximately the same as or slightly smaller than the width of the display 11 (in this case, the vertical width) in the scroll direction (in this case, the vertical direction), that is, approximately the equivalent of one page in the vertical direction of the screen.

[0060] In step S305, a scroll command output process is performed. This process outputs the scroll command set in steps S303 and S304 to the browsing application. When the scroll command is output to the browsing application, the browsing application scrolls the content currently displayed on display 11 upwards or downwards by a predetermined amount. This makes it possible to scroll the content displayed on display 11 upwards or downwards by approximately one page in the vertical direction of the screen.

[0061] For example, in the example of the stacking chair content shown in Figure 5(B), the browsing application scrolls the detailed information 74a displayed on the display 11 downwards by approximately one page vertically. As a result, as shown in Figure 5(D), in the detailed information 74a' displayed on the display 11 after scrolling, the "★★★Related Products★★★" display, which was located at the bottom of the screen before scrolling, is now located at the top of the screen, and several related product information items 75a~75c, which were not displayed before scrolling, are displayed below "★★★Related Products★★★". In addition, the model number, size, weight, color, material, functions and precautions, as well as part of the "<<<Product Reviews>>>", are also displayed as "<<<Product Details>>>".

[0062] Once the processing in step S305 is complete, the series of second operation processes are finished, and the system returns to the main routine of the browsing operation assistance application shown in Figure 3. The process then proceeds back to step S102 to acquire image data captured within the predetermined period from this point onward. In other words, it prepares for the next hand or other movement the user may make.

[0063] In this embodiment, the browsing operation assistance application was described as an example of being plugged into a general-purpose browsing application and functioning as a subroutine. However, it is not limited to this, and for example, the functions of the browsing operation assistance application (Figures 3 and 4) may be pre-integrated into a dedicated browsing application for the smartphone 10 as one of its functions.

[0064] As described above, in the smartphone 10 of this embodiment, the control unit 20 controls the control unit 20 either via the browsing operation assistance application installed thereon, either through the browsing application or directly, and when the control unit 20 determines that the series of image information captured by the back camera 16 within a predetermined period includes an image of the user's hands, etc. (S104; YES), and when the control unit 20 determines that the movement of the hands, etc. within the predetermined period based on the series of image information is a predetermined first action (predetermined action) (S106; YES (first action)), the control unit 20 switches the display on the display 11 from the detailed information 74a currently displayed to the display of related product information 75a to 75c (S206).

[0065] As a result, for example, the user of the smartphone 10 can switch from the detailed information 74a currently displayed on the display 11 to the display of related product information 75a~75c by moving their hand, etc., so that the movement of their hand, etc., captured by the rear camera 16 becomes the first action, instead of swiping or tapping the display 11. For example, if the user is holding the smartphone 10 with their left hand 90L (or right hand 90R), they can perform a predetermined action by holding their right hand 90R (or left hand 90L) over the back side 10B of the smartphone 10, thereby enabling them to switch content without the content displayed on the display 11 being obscured by their right hand 90R (or left hand 90L). Therefore, since the operation to reach the desired information can be performed solely by moving the right hand 90R (or left hand 90L) behind the display 11, visibility can be ensured while improving operability.

[0066] Furthermore, in the smartphone 10 of this embodiment, if the control unit 20 determines that a hand movement or the like is a second action (another predetermined action) (S106; YES (second action)), it scrolls the screen display of the display 11 upwards or downwards as a control related to the display mode of the content. Control related to the display mode of the content may, for example, be to scroll the content currently displayed on the display 11 to the left or right, or to change the display magnification of the content (enlarge or reduce display). Alternatively, the content displayed on the display 11 may be returned to the content of the previous point in time (or one level up in the display hierarchy).

[0067] As a result, for example, the user of the smartphone 10 can control various display modes, such as scrolling the content currently displayed on the display 11 vertically or horizontally, or zooming in or out, by moving their hand or other body, which is captured by the rear camera 16, so that it corresponds to a second predetermined action, instead of swiping the display 11.

[0068] Furthermore, in the smartphone 10 of this embodiment, when multiple related product information 75 (other content) is displayed on the display 11, and the control unit 20 repeatedly determines that a hand movement or the like is the first action, and each time it determines, the multiple related product information 75a to 75c are switched sequentially, for example, if the content of related product information 75b, which is scheduled to be displayed next on the display 11, is the same as content already displayed on the display 11, the control unit 20 switches from the content of related product information 75b (next displayed content) to the content of related product information 75c (other content scheduled to be displayed after the next displayed content) and displays it on the display 11. This prevents detailed information with the same content as already displayed content from being displayed on the display 11. In other words, it is possible to prevent situations where the user is shown content that has already been viewed again, which could give them a feeling of disappointment or that they have wasted their time.

[0069] In the embodiment of the browsing assistance application described above, when the browsing content is a recommended product, related product information 75a to 75c was used as an example to illustrate the other content information associated with it. However, the other content information associated with it may be, for example, company information selling the recommended product or information on other products sold by that company. Furthermore, the browsing content is not limited to recommended products provided by an internet search engine using the browsing application, but may also be text data, image data, video data, etc., provided from the internet in association with the search results from the search engine. In addition, any electronic data that can be viewed on the display 11 using the browsing application may be, for example, text data, image data, video data, etc., stored in the memory 25 of the smartphone 10.

[0070] Furthermore, in the embodiment of the browsing assistance application described above, an example was explained in which the screen display on the display 11 is scrolled upward or downward in response to the second action. However, as long as the control relates to the display mode of the content in response to the second action, for example, the content currently displayed on the display 11 may be scrolled left or right, or the display magnification of the content may be changed (enlarged or reduced). Alternatively, the browsing assistance application may be configured to return the content displayed on the display 11 to the content of the previous point in time (or one level up in the display hierarchy) in response to the second action. Note that scrolling in the vertical or horizontal directions may be configured to scroll continuously at a constant speed, or the scrolling speed may be configured to change in response to the distance the hand or other object is moved in the second action. For example, the scrolling speed may increase as the vertical movement distance of the hand or other object increases, and decrease as the vertical movement distance decreases.

[0071] Furthermore, in the embodiment of the browsing assistance application described above, examples were explained in which the display on the display 11 is switched to other content in response to the first operation, and in which the screen display on the display 11 is scrolled upwards or downwards in response to the second operation. However, in addition to these, a third operation, a fourth operation, a fifth operation, etc., may be registered, and the operation determination process (S106) may be configured to control the display mode of the content displayed on the display 11 in response to these operations.

[0072] For example, the third action may be a motion of swinging the right hand 90R (or left hand 90L) diagonally upwards and downwards to the left, the fourth action may be a motion of swinging the right hand 90R (or left hand 90L) diagonally upwards and downwards to the right, and the fifth action may be a motion of moving the right hand 90R (or left hand 90L) back and forth, moving it away from or closer to the smartphone 10. The browsing assistance app may be configured to scroll the content displayed on the display 11 to the left or right in response to the third action, change the display magnification of the same content in response to the fourth action, or return the display on the display 11 from the current content to the content one step back in the display hierarchy in response to the fifth action. Furthermore, if the content displayed on the display 11 is video data, the browsing assistance app may be configured to, for example, play the video with the second action, stop the playing video with the third action, fast-forward the video with the fourth action, or rewind the video with the fifth action.

[0073] [Other inventions] Conventional mobile information terminal devices such as smartphones allow users to swipe or tap the display with their fingers, stylus, or touch pen (hereinafter referred to as "fingers, etc."), enabling the display of a large amount of information even on small displays where portability is prioritized over display size. However, there are the following problems in terms of operation. Generally, the surface of the display of a mobile information terminal device is treated with a glossy finish to make images such as photos and videos look good, or with an anti-glare finish to reduce eye strain caused by reflections. In either case, the touch of a finger, etc. often feels like a flat surface (even an anti-glare finish only feels slightly rough).

[0074] Therefore, for example, when swiping the display with a finger, it is not possible to provide the same click sensation to the finger as when rotating a mouse scroll wheel. In other words, conventional portable information terminal devices have not been able to intuitively convey the amount of swipe to the user when swiping the display. Therefore, in [other inventions], an operating device (also called a "scrolling device") is provided that can intuitively convey the amount of swipe to the user when swiping the display.

[0075] [Other Invention 1] A scroll operation device used by being attached to a portable information terminal device equipped with a touch panel on the surface of a display, A roller portion having elastic protrusions on its outer circumference at approximately equal intervals in the circumferential direction, A frame that rotatably supports both ends of the roller section, A support portion is attached to the portable information terminal device at both ends or one end of the frame portion such that the protrusion contacts the touch panel when the roller portion rotates, A scroll operation device characterized by comprising: [Other Invention 2] The scroll operating device, wherein the protrusions are present in a linear fashion in the axial direction of the roller portion, in the other invention 1 described above. [Other Inventions 3] A scroll operation device characterized in that, in the above-mentioned other invention 1 or other invention 2, the frame portion is provided with a mechanism that provides a clicking sensation to the user's fingers that touch the roller portion, or a clicking sound to the user's hearing, in conjunction with the rotation of the roller portion.

[0076] The above-mentioned alternative invention 1 is a scroll operation device used when attached to a portable information terminal device equipped with a touch panel on the surface of a display, and comprises a roller part, a frame part, and a support part. The roller part has elastic protrusions on its outer circumference at approximately equal intervals in the circumferential direction, and both ends of the roller part are rotatably supported by the frame part, and the frame part is attached to the portable information terminal device by the support part at both ends or one end of the frame part so that the protrusions of the roller part come into contact with the touch panel when the roller part rotates. Note that rotation means that the roller part can rotate in both the forward direction and the reverse direction. Note that the scroll operation device of this alternative invention is also capable of operations other than scrolling (screen switching operation), and is therefore also simply called an "operation device".

[0077] As a result, when a user of a mobile device rotates the roller with their finger, similar to how a mouse scroll wheel is rotated, the "protrusions on the outer circumference of the roller, which are spaced almost equally in the circumferential direction," come into discontinuous contact with the touch panel of the mobile device. Therefore, the roller contacts the touch panel as if a finger were swiping in short, precise strokes, and the user's finger is given a tactile sensation of the protrusions that occurs in proportion to the amount of rotation of the roller. The amount of rotation of the roller is the sum of the small swipes (total amount of small swipes), and can be considered equivalent to the swipe amount. Furthermore, the user can intuitively grasp the amount of rotation of the roller from the intermittent tactile sensation of the protrusions. Therefore, when swiping the display, the amount of swipe can be intuitively conveyed to the user.

[0078] In the above-mentioned alternative invention 2, since the protrusions of the roller portion are linear in the axial direction of the roller portion, it becomes possible to swipe over a wide area on the touch panel. As a result, even if the scroll operation device is mounted on a portable information terminal in an improper state, such as when a part of the roller portion is excessively lifted away from the touch panel in the axial direction of the roller portion, it becomes possible to bring the protrusions on the remaining part of the roller portion into contact with the touch panel. Therefore, even in such an improper mounting state, the amount of swipe can be intuitively conveyed to the user when swiping the display.

[0079] In the above-mentioned other invention 3, the frame portion includes a mechanism that provides a click sensation to the user's fingers touching the roller portion, or a click sound to the user's hearing, as the roller portion rotates. As a result, the user can obtain a click sensation or click sound via touch or hearing as the roller portion rotates, in addition to the contact sensation of the convex portion, thus more reliably providing the user with a swiping amount intuitively.

[0080] The scroll operation devices described in [Other Invention 1] to [Other Invention 3] above can be embodied, for example, as a scroller 30 as shown in Figures 6 and 7. The scroller 30 is used when attached to a smartphone 50 equipped with a touch panel 52 on the surface of a display 51, and mainly comprises a roller section 35, a frame 31, and a click mechanism section 36.

[0081] The display 51 of the smartphone 50 is, for example, a 6-inch liquid crystal display with an aspect ratio of 16:9. The touch panel 52 is a transparent film-type pointing device that is approximately the same size as the display 51 and is located almost in front of the display surface. It is configured to output position information on the display 51 where the user has touched with their finger or stylus to the control unit of the smartphone 50. Position detection of the touch panel 52 can be done using, for example, a resistive method, a capacitive method, or an ultrasonic surface acoustic wave method.

[0082] The roller portion 35 is made of an elastic and highly conductive material (for example, conductive rubber) and is formed in a polygonal prism shape. In this embodiment, it is formed in a regular hexagonal prism shape with an axial length set shorter than the width of the display 51. As long as the cross-sectional shape is a regular polygon, it may also be a regular octagonal prism shape or a regular decagonal prism shape, for example. On the outer circumference of the roller portion 35, i.e., the circumferential surface portion 35a, corner portions 35b extending linearly in the axial direction are formed at approximately equal intervals in the circumferential direction. In this embodiment, since the roller portion 35 is a regular hexagonal prism shape, the corner portions 35b are formed at six locations on the circumference. A shaft member is provided so as to pass through the center of the roller portion 35, and both ends of it protrude as shaft portions 35c and 35d. In this embodiment, since the corner portion 35b is formed as a linear axial groove between both ends of the roller portion 35, even if the scroll roller 30 is attached to the smartphone 50 in an improper state, such as when one end of the roller portion 35 is excessively lifted from the touch panel 52, it is possible to bring the corner portion 35b on the other side of the roller portion 35 into contact with the touch panel 52.

[0083] The frame 31 is composed of an upper frame 32 and a lower frame 33, which are configured to hold the smartphone 50 between them, both front and back, and in the lateral (left and right) direction. The upper frame 32 is configured to rotatably support the roller portion 35. The upper frame 32 and the lower frame 33 are made of, for example, high-density polyethylene, polypropylene, or ABS resin. When the smartphone 50 is held in place, the separation distance between the upper frame 32 and the lower frame 33 is maintained such that, as shown in Figure 7(A), when the roller portion 35 of the scroll roller 30 and the touch panel 12 of the smartphone 50 face each other, a gap SP is created at the circumferential surface 35a of the roller portion 35, and the corner portion 35b of the roller portion 35 is pressed against it to the extent that it is slightly compressed.

[0084] The upper frame 32 is formed in the shape of a thin strip, longer than the width of the smartphone 50, and has a rectangular elongated hole (roller housing portion 32a) in its approximate center, which can accommodate the roller portion 35 when it is laid flat along the longitudinal direction of the upper frame 32. A hinge piece 32b, which constitutes a hinge mechanism, is formed at one end of the upper frame 32, and a pivot shaft 34 passes through the hinge piece 32b. The pivot shaft 34 may be formed integrally with the hinge piece 32b. A claw portion 32c is formed at the other end of the upper frame 32, which protrudes in the longitudinal direction of the upper frame 32. A click mechanism portion 36, which will be described later, is housed in the side wall of the roller housing portion 32a located at one end of the upper frame 32, and the shaft portion 35c of the roller portion 35 is connected to the click mechanism portion 36. Furthermore, a bearing hole 32d is formed in the side wall of the roller housing portion 32a located on the other end, which is capable of rotatably supporting the shaft portion 35d of the roller portion 35.

[0085] The lower frame 33 is formed in a strip shape with L-angle portions 33b and 33c at both ends that can sandwich the longitudinal ends of the upper frame 32. Cushioning material 39 is attached to the inside of the lower frame 33, that is, to the inner surfaces of the smartphone storage portion 33a and the L-angle portions 33c and 33d. The cushioning material 39 is, for example, a sheet of rubber sponge and is attached to the inner surface of the lower frame 33 via double-sided tape or the like. The width of the lower frame 33 in the short direction is set to be approximately the same as that of the upper frame 32. The L-angle portion 33b at one end functions as a hinge piece that forms a hinge mechanism together with the hinge piece 32b of the upper frame 32. The L-angle portion 33c at the other end has a groove portion 33d that can engage with the claw portion 32c of the upper frame 32. The claw portion 32c and the groove portion 33d form a locking mechanism.

[0086] The click mechanism 36 is a mechanism that provides a click sensation to the user's fingers that touch the roller 35 as the roller 35 rotates, and provides a click sound to the user's hearing. For example, it has a known configuration used in mouse wheels. Although not shown in the figures, it specifically consists of a disc portion having multiple linear grooves radiating from the center and connected to a shaft portion 35c at one end of the roller 35, which serves as the pivot axis; a bearing hole that pivotally supports this disc portion; and a flexible claw fixed so as to be able to engage with the multiple linear grooves of the rotating disc portion. When the disc portion rotates together with the roller 35, the flexible claw overcomes the unevenness by successively traversing the multiple linear grooves of the disc portion as the disc portion rotates. As a result, the click mechanism 36 emits a clicking sound and vibration, providing the user with a click sound and a click sensation associated with the rotation of the roller 35.

[0087] By configuring the scroller 30 in this way, as shown in Figure 6(D), the hinge mechanism configured on one end of the upper frame 32 and the lower frame 33 allows the upper frame 32 and the lower frame 33 to rotate relative to each other around the pivot axis 34, and the locking mechanism configured on the other end of the upper frame 32 and the lower frame 33 allows the smartphone 50 to be held in place. As a result, the scroller 30 can be attached to the smartphone 50 in a position where the roller portion 35 can contact the touch panel 52 (Figures 6(A) to (C)).

[0088] The scroller 30 attached to the smartphone 50 functions as shown in Figure 7. As shown in Figure 7(A), when the user of the smartphone 50 rotates the roller part 35 by pulling it downwards with their finger 91, the roller part 35 rotates in the direction of the arrow in the figure. Then, in the state shown in Figure 7(A), a gap SP is formed between the circumferential surface 35a of the roller part 35 and the touch panel 52, and the two (roller part 35 and touch panel 52) are not in contact with each other. However, as the roller part 35 rotates, the corner 35x of the roller part 35 comes into contact with the touch panel 52 (Figure 7(B)). By rotating the roller part 35 in the same direction, the corner 35x' is pressed against the touch panel 52 and crushed (Figure 7(C)), the crushed corner 35x' slides in the direction of the arrow, and as the rotation continues, the crushed corner 35x' returns to its original shape (corner 35x) (Figure 7(D)).

[0089] Since multiple corners 35b are formed on the circumferential surface 35a of the roller portion 35 at equal intervals in the circumferential direction, these multiple corners 35b intermittently (discontinuously) contact the touch panel 52 through three contact processes: corner 35x → flattened corner 35x' → corner 35x, and the flattened corner 35x' moves on the touch panel 52 as if sliding upwards on the smartphone 50. As a result, the roller portion 35 makes surface contact with the touch panel 52 as if the smartphone 50 were being swiped upwards with a predetermined short stroke using a finger, and the user's finger is given a feeling of contact with the corners 35b that is generated in proportion to the amount of rotation of the roller portion 35. The amount of rotation of the roller portion 35 is the total amount of small swipes (total amount of small swipes), and can be said to correspond to the amount of swiping. Furthermore, the user can intuitively grasp the amount of rotation of the roller portion 35 from the intermittent contact sensation of the corner portion 35b.

[0090] Therefore, when swiping the display 51, the amount of swipe can be intuitively conveyed to the user. Furthermore, the click mechanism 36 allows the user to obtain a click sensation or click sound via touch or hearing in addition to the contact sensation of the corner 35b, as the roller 35 rotates, thus more reliably conveying the amount of swipe intuitively to the user. In addition, since the corner 35b of the roller 35 is formed as a linear groove in the axial direction between both ends of the roller 35, even if the scroll roller 30 is attached to the smartphone 50 in an inappropriate state where one side of the roller 35 is excessively lifted from the touch panel 12, the amount of swipe can still be intuitively conveyed to the user when swiping the display 51.

[0091] Furthermore, the corner portion 35b of the roller portion 35 may be continuous (linear) or discontinuous (point-like) in the axial direction of the roller portion 35. Also, the roller portion 35 may be a cylinder with a curved circumferential surface, in which case it has a convex portion that is continuous (linear) or discontinuous (point-like) in the axial direction. In addition, the support portion does not have to be connected at both ends, like the lower frame 33. It may be composed of two members with a U-shape facing each other on the opening side (open square bracket symbol "[" and closed square bracket symbol "[]" facing each other). Alternatively, a configuration in which these two members are connected to both ends of the upper frame 32 may be adopted.

[0092] Furthermore, while the example shown in Figure 7 describes the case where the user of the smartphone 50 pulls the roller part 35 downwards with their finger 91, the roller part 35 and the click mechanism part 36 function in the same way as in the "downward pull" case when the roller part 35 is rotated to push upwards on the smartphone 50. However, in this case, the roller part 35 rotates in the opposite direction to the arrow in the figure, so the flattened corner part 35x' moves as if sliding downwards on the touch panel 52. As a result, the roller part 35 makes surface contact with the touch panel 52 as if the smartphone 50 were being swiped downwards with a predetermined short stroke using a finger or the like.

[0093] Furthermore, in the aforementioned embodiments of the invention, a predetermined display control is performed by bringing the corner of the roller part into contact with the touch panel (detecting that the roller part has come into contact with the touch panel). Instead of this embodiment, a scroll roller may be provided in a position where the rotation of the roller part (amount of rotation, rotation angle) can be detected (imaged) by a camera (front camera) (for example, by positioning the camera below the roller part), so that the amount of rotation of the roller part can be recognized and the display control can be performed even if the roller part and the touch panel (display screen) are not in contact. In addition, by making the diameter of the roller part smaller than that shown in Figure 6, it is possible to prevent the roller part from coming into contact with the touch panel (display screen) when the roller part is rotated, thereby preventing damage to the touch panel (display screen) due to the rotation operation of the roller part.

[0094] Furthermore, in the other embodiments of the invention described above, by rotating the roller part, it becomes possible to scroll the displayed content vertically on the display or to switch the displayed content to other content. Also, by performing a predetermined input operation (predetermined input) using an app installed on a smartphone, the content scrolling operation (first display control) or It may be possible to associate either the content switching operation (second display control) with the other operation. This allows the user to switch which operation is performed by the "rotation operation of the roller part".

[0095] [Reference invention] Furthermore, while mobile information terminal devices such as smartphones allow users to display a large amount of information even on small displays where portability is prioritized over display size, by swiping or tapping the display with their fingers, there are the following problems in terms of operation. For example, if a user selects a thumbnail image of a particular piece of content from among many thumbnail images to display detailed information, and that detailed information differs from what was expected, the user may need to perform a "back operation" by tapping the display again with their finger to display the many thumbnail images, or a "selection operation" by tapping a thumbnail image of another piece of content to display detailed information.

[0096] In other words, with conventional mobile information terminal devices, users had to repeatedly swipe and tap the display to select and go back until they found the desired content or product information. This process was cumbersome and potentially inconvenient for the user.

[0097] Furthermore, because the touch panel is located on the display, swiping or tapping the display with a finger or stylus can obscure some of the content or information displayed on the screen, making it difficult for the user to see. This has resulted in a problem where the visibility of the display may be impaired.

[0098] This reference invention was made to solve the above-mentioned problems, and aims to provide a portable information terminal device and its program that can improve operability while ensuring visibility.

[0099] To achieve the above objective, the portable information terminal device of Reference Invention 1 is: A portable information terminal device equipped with a camera on the back side of the surface having a display, If the series of image information captured by the camera within a predetermined period includes an image of a human hand or finger (hereinafter referred to as "hand, etc."), the motion determination means determines whether the movement of the hand, etc. within the predetermined period is a predetermined action based on the series of image information, The system includes a content switching means that, when the motion determination means determines that the movement of the hand or the like is a predetermined action, switches the content currently displayed on the display to display other content. When a search is performed using a predetermined search target, and multiple content items are output as a result, and a specific content item selected from the multiple content items is displayed on the display, if the motion determination means determines that the movement of the hand or other body part is the predetermined action, the content switching means switches the content currently displayed on the display to display other content. It is characterized by the following:

[0100] Furthermore, the program of Reference Invention 2 for achieving the above objective is A computer that controls a portable information terminal device equipped with a camera on the back side of the display surface, If the series of image information captured by the camera within a predetermined period includes an image of a human hand or finger (hereinafter referred to as "hand, etc."), an action determination means determines, based on the series of image information, whether the movement of the hand, etc. within the predetermined period is a predetermined action. A program that functions as a content switching means, which switches the content currently displayed on the display to the display of other content when the motion determination means determines that the movement of the hand or the like is a predetermined action, In a case where there are multiple other contents, and the content switching means determines multiple times in a row that the movement of the hand, etc., is the predetermined action, and the display of the multiple other contents is switched sequentially each time such determination is made, The content switching means, when the next content to be displayed on the display from among the multiple other content items is the same as content already displayed on the display, switches to displaying the other content that is scheduled to be displayed after the next content, instead of the previously displayed content, and displays it on the display. It is characterized by the following:

[0101] Furthermore, if the portable information terminal device contains images of a person's (typically the user of the portable information terminal device) hand or fingers (hereinafter referred to as "hand, etc." in the sections [Means for Solving the Problem] and [Effects of the Invention]) in a series of image information captured by the camera within a predetermined period, and the motion determination means determines that the movement of the hand, etc. within the predetermined period is a predetermined action based on the series of image information, the content switching means switches the content currently displayed on the display to the display of other content.

[0102] This allows, for example, the user of the mobile information terminal to switch the currently displayed content to other content by moving their hand, etc., which is being captured by the camera, to perform a predetermined action, instead of swiping or tapping the display. For example, if the user is holding the mobile information terminal in their left hand, they can perform a predetermined action by holding their right hand over the back of the mobile information terminal, allowing them to switch content without their hand obscuring the content displayed on the screen. In this specification, "camera" mainly refers to a digital camera that forms an image of a subject onto a solid-state image sensor.

[0103] Furthermore, if there are multiple other contents and the content switching means repeatedly determines that a hand or other movement is a predetermined action, and the display of multiple other contents is switched sequentially each time such determination is made, the content switching means will, if the next content to be displayed on the display is the same as content already displayed on the display, switch to another content that is scheduled to be displayed after the next content, and display that instead. As a result, content with the same content as already displayed content will not be displayed on the display, thus preventing the display of duplicate content. In other words, it is possible to prevent situations where the user of the mobile information terminal is shown content that has already been viewed, which could give them a feeling of disappointment or that they have wasted their time.

[0104] Furthermore, the system may be configured to include an image determination means for determining whether or not an image (image information, specific image) that satisfies control conditions is included in a series of image information captured by the rear-side camera within a predetermined period. Also, if the image determination means determines that an image of a person's (typically the user of the mobile information terminal device) hand or fingers is included in a series of image information captured by the camera within a predetermined period, and the operation determination means determines that the movement of the hand, etc., within the predetermined period based on the series of image information is a predetermined operation, the content switching means may be configured to switch the content currently displayed on the display to the display of other content.

[0105] Here, "image that satisfies the control conditions (specific image)" refers to an image (series of image information) that the camera on the back of the portable information terminal device recognizes as an appropriate image within the imaging area set for executing the display control of the content of the present invention. Furthermore, if the image determination means and the operation determination means recognize an image within a predetermined imaging area, they can recognize it as a target image even if it is not an image of a human hand or finger (for example, an image of a stylus or other object), and execute display control of the content displayed on the display. However, it is preferable that at least an image of a human hand or finger can be recognized as a target image. In addition to this configuration, by adding certain conditions to the image determination means (for example, shape conditions or temperature conditions, etc.), display control of the content displayed on the display may be performed only when a human hand or finger, or something similar (i.e., something that satisfies certain conditions common to a human hand or finger), is recognized.

[0106] Furthermore, this reference invention assumes that a camera on the back of a portable information terminal device captures (detects) an image of a human hand or finger, and controls the display of content shown on the display. For this reason, the imaging area in which the captured image can be recognized as a proper image is set to a distance (depth) from the camera on the back of the portable information terminal device between approximately 0 cm and approximately 25 cm. More preferably, the imaging area may be set between approximately 1 cm and approximately 15 cm.

[0107] Furthermore, this imaging area can be appropriately set according to the performance of the rear-side camera of the portable information terminal device, and the present invention is not limited to such a specific range (approximately 0 cm to approximately 25 cm). However, if the distance from the camera is too close, it becomes difficult to recognize (identify) a variety of movements such as hand movements, and if the distance from the camera is too far, there is a risk of recognizing other image information that the user did not intend (for example, the movements of someone other than the user). For this reason, it is preferable to limit it to the specific range (approximately 0 cm to approximately 25 cm) as described above.

[0108] According to the portable information terminal device and the computer program that controls it of this reference invention, the user can, for example, switch the content currently displayed on the display to other content, or control the display mode, such as scrolling up and down or left and right, or zooming in and out, by moving their hand or other hand, which is captured by the camera, to perform a predetermined action or another predetermined action, instead of swiping or tapping the display. For example, if the user is holding the portable information terminal device with one hand (for example, the left hand), they can perform a predetermined action or another predetermined action by holding the other hand (for example, the right hand) over the back of the portable information terminal device, thereby enabling content switching operations and control of content display mode without the content displayed on the display being obscured by the hand or other hand. Therefore, since operations to reach the desired information can be performed solely by hand movements on the back of the display, operability can be improved while ensuring visibility. [Explanation of Symbols]

[0109] 10, 50 Smartphone (Mobile Information Terminal Device), 10B Back side, 10F Front side (Side with display), 11, 51 Display, 12, 52 Touch panel, 15 Front camera, 16 Back camera (Camera on the back side), 20 Control unit (Image determination means, motion determination means, content switching means, display control means, computer that controls the mobile information terminal device), 30 Scroller (Scroll operation device), 32 Upper frame (Frame body part, support part), 33 Lower frame (Support part), 35 Roller part, 35b, 35x, 35x' Corner part (Convex part), 35c, 35d Shaft part (Both ends of the roller part), 36 Click mechanism part (Mechanism that provides a click sensation, mechanism that provides a click sound), 71a, 71b, 72 Reduced image information, 73 Summary information, 74a, 74b, 74a' Detailed information (Content) 75a, 75b, 75c Related product information (other content), 90R Right hand 90R (human hand), 90L Left hand 90L (human hand), 91 Finger 91, S103, S104 (image determination means), S105, S106 (action determination means), S107 (content switching means), S108 (display control means).

Claims

1. An operating device used by being attached to a portable information terminal device that has a touch panel on the surface of the display, The roller section, A frame that rotatably supports both ends of the roller section, Support portion for attaching both ends or one end of the frame to the portable information terminal device, The system includes a detection means for detecting the rotation of the roller portion, An operating device characterized by controlling the display of the display in accordance with the rotation of the roller portion.

2. The outer circumference of the roller portion is provided with elastic protrusions. The operating device according to claim 1, characterized in that the protrusion is present in a linear fashion in the axial direction of the roller portion.

3. The operating device according to claim 1 or 2, characterized in that the frame portion includes a mechanism that provides a clicking sensation to the user's fingers touching the roller portion as the roller portion rotates, or a mechanism that provides a clicking sound to the user's hearing.

Images