System, information processing device, information processing method, and program

Abstract

This system includes: a controller which is operated by a user; and one or a plurality of processors. The controller includes: a plurality of independent buttons that can be pressed down; a first sensor capable of detecting the pressing of the buttons; and a second sensor capable of detecting the approach or contact of the user's finger with respect to the plurality of buttons. When there is a shifting operation in which the user's finger approaches or touches two or more buttons in sequence, the processors execute first processing on the basis of the order in which the buttons were approached or contacted in said shifting operation.

Description

System, information processing device, information processing method, and program

[0001] The present disclosure relates to a system, an information processing device, an information processing method, and a program.

[0002] 2. Description of the Related Art Touchpads capable of detecting touch operations are used in various products to improve user operability.

[0003] For example, Japanese Patent Laid-Open Publication No. 2006-191232 (Patent Document 1) discloses an imaging device having a circular touchpad and a press-type decision button located in the center of the touchpad, which controls the display of a menu screen in response to the detection results of the direction and speed of rotation of a finger on the touchpad.

[0004] Japanese Patent Application Laid-Open No. 2006-191232

[0005] In the above-mentioned prior art, the touchpad makes it difficult for the user to determine whether the touch input is being accepted, and it may be difficult for the device to determine the intention of the user's touch operation. In addition, the touchpad and the enter button are separately provided, and space is required for their placement.

[0006] An object of the present disclosure is to provide a controller with improved usability and a process that follows operations given to the controller.

[0007] (Configuration 1) A system according to one embodiment includes a controller operated by a user and one or more processors. The controller includes a plurality of independent buttons that can be pressed, a first sensor that can detect button presses, and a second sensor that can detect approach or contact of the user's finger with the plurality of buttons. When the user's finger performs a movement operation in which the user's finger approaches or contacts two or more buttons in sequence, the processor executes a first process based on the order of the buttons approached or contacted in the movement operation.

[0008] According to configuration 1, since it is possible to detect the approach or contact of a user's finger to multiple buttons, the system can obtain information on user operations such as approaching or contacting a button, as well as information on user operations such as pressing a button. Furthermore, since multiple independent buttons are used that can be pressed, the user can specify the function assigned to each button press, and can distinguish between each button by the tactile sensation of their fingertip, and then provide the intended instruction to the system by approaching or contacting the desired button. This makes it possible to realize a controller with improved usability and a process that follows operations given to the controller.

[0009] (Configuration 2) In configuration 1, the processor may execute a second process based on pressing any one of the plurality of buttons after executing the first process.

[0010] According to the second aspect, the user can execute the second process by pressing any button after the first process is executed, which allows the user to perform an intuitive operation.

[0011] (Configuration 3) In configuration 2, the processor may execute the second process when any one of the plurality of buttons is pressed.

[0012] According to configuration 3, for example, assuming that a specific button among the multiple buttons must be pressed to execute the second process, depending on the content of the immediately preceding movement operation, an additional finger movement may be required to press the specific button, and this additional movement may be erroneously determined to be part of the immediately preceding movement operation. In contrast, by executing the second process regardless of which button among the multiple buttons is pressed, the possibility of executing a process unintended by the user can be reduced.

[0013] (Configuration 4) In configuration 2 or 3, the processor may execute the second process based on the fact that the last button in the movement operation is pressed while the user's finger is approaching or in contact with the button.

[0014] According to configuration 4, the second process is not executed unless the user presses the last button in the movement operation while the user's finger is close to or in contact with the button. This reduces the possibility of the second process being executed by mistake when the user ends the movement operation and attempts to perform another operation.

[0015] (Configuration 5) In configurations 2 or 3, the processor may execute the second process based on the fact that a button is pressed within a predetermined time after the user's finger is released from the button that was last approached or touched in the movement operation.

[0016] According to configuration 5, the second process is not executed unless the user presses a button within a predetermined time after the user releases the finger from the button that was last approached or touched during the movement operation. This reduces the possibility of the second process being executed by mistake when the user ends the movement operation and attempts to perform another operation.

[0017] (Configuration 6) In any of configurations 1 to 5, the plurality of buttons may include three or more buttons, and the movement operation may include the user's finger sequentially approaching or touching the three or more buttons.

[0018] According to configuration 6, the first process is executed when the user's finger approaches or touches three or more buttons in sequence, so that even if the user accidentally approaches or touches two buttons, the possibility of the first process being executed unintentionally by the user can be reduced.

[0019] (Configuration 7) In any of configurations 1 to 6, the processor may vary the processing content of the first process depending on whether the order of the buttons approached or touched in the movement operation is clockwise or counterclockwise.

[0020] According to configuration 7, the user can select two types of processing simply by switching the order in which the user's finger approaches or touches the buttons clockwise or counterclockwise, thereby improving usability.

[0021] (Configuration 8) In any of configurations 1 to 6, the first process may include a process of moving a cursor to select one of the displayed items.

[0022] According to configuration 8, the user can move the cursor more intuitively by moving their finger closer to or in contact with two or more buttons in turn. Also, according to configuration 8, the user can operate the buttons while feeling their presence with the touch of their fingertip, which makes it easier for the user to understand the operation they are giving to the system compared to operating a flat touch panel.

[0023] (Configuration 9) In configuration 1, the processor may execute the first process based on the fact that at least one of the plurality of buttons or a button different from the plurality of buttons is pressed following the movement operation.

[0024] According to the ninth aspect, the first process is not executed unless the button is pressed, thereby reducing the possibility that the first process will be executed unintentionally by the user.

[0025] (Configuration 10) In configuration 9, the processor may execute the first process based on the fact that, following a movement operation, the button that was last approached or touched in the movement operation is pressed.

[0026] According to configuration 10, since the user only needs to press the button that was last approached or touched during the movement operation, the first process can be executed more easily if the user intends to execute the first process.

[0027] (Configuration 11) In configuration 9 or 10, the processor may execute the first process based on the fact that the last button in the movement operation is pressed while the user's finger is approaching or in contact with the button.

[0028] According to configuration 11, unless the user's finger is close to or in contact with the last button in the movement operation and the button is pressed, the first process will not be executed, thereby reducing the possibility that the first process will be executed unintentionally by the user.

[0029] (Configuration 12) In configurations 9 or 10, the processor may execute the first process based on the fact that a button is pressed within a predetermined time after the user's finger is released from the button that was last approached or touched in the movement operation.

[0030] According to configuration 12, if the user's finger is released from the button that was last approached or touched during the movement operation and the button is not pressed within a predetermined time, the first process will not be executed, thereby reducing the possibility that the first process will be executed unintentionally by the user.

[0031] (Configuration 13) In any of configurations 1 to 12, the processor may be configured not to execute processing in response to a user operation when the user's finger approaches or touches only one button.

[0032] According to configuration 13, it is possible to reduce the possibility that an unintended process will be executed when the user unintentionally approaches or touches any one of the buttons.

[0033] (Configuration 14) In configuration 13, the controller may be configured to be able to be held by a user. The plurality of buttons may be provided in a first area operable with one finger of a user holding the controller.

[0034] According to configuration 14, the user holding the controller can operate multiple buttons with one finger, thereby improving operability for the user.

[0035] (Configuration 15) In configuration 14, a plurality of independent buttons that can be pressed may be provided in a second region different from the first region. The plurality of buttons provided in the first region and the plurality of buttons provided in the second region may be configured to be able to independently detect movement operations on the plurality of buttons.

[0036] According to configuration 15, the user can operate the plurality of buttons provided in the first area and the plurality of buttons provided in the second area, thereby improving operability for the user.

[0037] (Configuration 16) In any of Configurations 1 to 15, the plurality of buttons may be four buttons. The four buttons may be arranged in a ring shape.

[0038] According to configuration 16, the buttons are arranged in a ring shape, so that the user can more easily perform operations such as moving his or her finger around the buttons.

[0039] (Configuration 17) In configuration 16, when the movement operation is such that the user's finger approaches or contacts a second button from a first button among the four buttons, the second button being different from the third and fourth buttons adjacent to the first button, the processor may perform a first process based on the user's finger approaching or contacting the third and fourth buttons among the four buttons.

[0040] According to configuration 17, in a movement operation from one button to another button that is not adjacent to the button, the detection resolution can be improved by using information on approach or contact with a button adjacent to the button from which the movement originates.

[0041] (Configuration 18) In configuration 17, the processor may perform a first process based on the user's finger approaching or touching both the third button and the fourth button.

[0042] According to configuration 18, the first process is executed on the condition that the user's finger approaches or touches both the third button and the fourth button, so that the user's intention can be more reliably reflected.

[0043] (Configuration 19) In configuration 17, the processor may execute the same first process when the user's finger approaches or touches either the third button or the fourth button, and when the user's finger approaches or touches both the third button and the fourth button.

[0044] According to configuration 19, even if one of the buttons is not approached or touched for some reason, the same first process is executed, thereby reducing the possibility of the user feeling uncomfortable.

[0045] (Configuration 20) In any of configurations 1 to 19, the processor may execute the processing and output an image generated by the execution of the processing. The first processing may include processing that changes the output image.

[0046] According to configuration 20, when a user operates a controller, the result of the operation is visually given, thereby improving usability.

[0047] (Configuration 21) An information processing device according to another embodiment is connected to a controller operated by a user. The controller includes a plurality of independent buttons that can be pressed, a first sensor that can detect button presses, and a second sensor that can detect approach or contact of the user's finger with the plurality of buttons. The information processing device includes one or more processors. When a user's finger performs a movement operation in which the user's finger approaches or contacts two or more buttons in sequence, the processor executes a first process based on the order of the buttons approached or contacted in the movement operation.

[0048] (Configuration 22) An information processing method according to yet another embodiment is executed in a system including a controller operated by a user. The controller includes a plurality of independent buttons that can be pressed, a first sensor that can detect the pressing of the buttons, and a second sensor that can detect the approach or contact of the user's finger with the plurality of buttons. The information processing method includes the steps of accepting a user's operation on the controller, and, when the user's finger performs a movement operation in which the user's finger approaches or contacts two or more buttons in sequence, executing a first process based on the order of the buttons approached or contacted in the movement operation.

[0049] (Configuration 23) A program according to yet another embodiment is executed on a computer connected to a controller operated by a user. The controller includes a plurality of independent buttons that can be pressed, a first sensor that can detect pressing of the buttons, and a second sensor that can detect approach or contact of the plurality of buttons by the user's finger. The program causes the computer to perform the following steps: accepting a user's operation on the controller; and, when the user's finger performs a movement operation in which the user's finger approaches or contacts two or more buttons in sequence, performing a first process based on the order of the buttons that were approached or contacted in the movement operation.

[0050] According to the present disclosure, it is possible to provide a controller with improved usability and a process according to an operation given to the controller.

[0051] FIG. 10 is a schematic diagram showing an example of the overall configuration of a system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of the system according to the present embodiment in which a controller is used while attached to a main unit. FIG. 10 is a schematic diagram showing an example of the system according to the present embodiment in which a controller is used while detached from the main unit. FIG. 10 is a schematic diagram showing an example of the hardware configuration of a main unit of the system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of the cross-sectional structure of a button of a controller of the system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of the hardware configuration of a controller of the system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of cursor movement processing in the system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of a user operation corresponding to the cursor movement processing shown in FIG. 7. FIG. 10 is a schematic diagram showing another example of a user operation corresponding to the cursor movement processing in the system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of a user operation for canceling a cursor movement input wait or a cursor movement mode in the system according to the present embodiment. FIG. 10 is a flowchart showing a processing procedure for cursor movement processing in the system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of a user operation for transitioning to a cursor movement input wait state in the system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of function allocation in cursor movement mode in the system according to the present embodiment. FIG. 10 is a schematic diagram showing an example of a user operation corresponding to cursor movement processing when two controllers are used in the system according to the present embodiment. FIG. 10 is a schematic diagram showing another example of cursor movement processing in the system according to the present embodiment. FIG. 11 is a schematic diagram showing an example of game processing in the system according to the present embodiment. FIG. 12 is a schematic diagram showing an example of user operation corresponding to the example of game processing in the system according to the present embodiment. FIG. 13 is a schematic diagram showing an example of user operation corresponding to the example of game processing in the system according to the present embodiment when two controllers are used in the system according to the present embodiment. FIG. 14 is a schematic diagram showing yet another example of game processing in the system according to the present embodiment. FIG. 15 is a schematic diagram showing an example of character input processing in the system according to the present embodiment. FIG. 16 is a schematic diagram showing an example of an operation guide function in the system according to the present embodiment. FIG. 17 is a schematic diagram showing an example of a combination of an operation on a controller and a touch operation in the system according to the present embodiment.10A and 10B are schematic diagrams showing examples of combinations of operations and touch operations on a direction indicator of a controller in the system according to the present embodiment.

[0052] The present embodiment will be described in detail with reference to the drawings, in which the same or corresponding parts are designated by the same reference numerals and description thereof will not be repeated.

[0053] [A. Overall Configuration Example] First, an overall configuration example of a system according to this embodiment will be described. In the following explanation, a system that executes game processing will be mainly described, but the system according to this embodiment and the processing described below can be applied to various applications other than game processing.

[0054] Furthermore, the system according to this embodiment may be configured with any electronic device such as a smartphone, a tablet, or a personal computer.

[0055] 1 is a schematic diagram showing an example of the overall configuration of a system 1 according to the present embodiment. Referring to FIG. 1, system 1 includes a main unit 100, which is an example of an information processing device, and one or more controllers 200 operated by a user.

[0056] Main unit 100 progresses through an application such as a game in accordance with data indicating user operations from each of controllers 200 .

[0057] Each of the controllers 200 accepts user operations. More specifically, the controller 200 includes a button operation unit 206 consisting of a plurality of independent buttons that can be pressed, and a direction indication unit 208. The button operation unit 206 and the direction indication unit 208 are provided in an area that can be operated with one finger of the user holding the controller 200.

[0058] For example, the button operation unit 206 includes four buttons 202_1 to 202_4 (hereinafter, sometimes collectively referred to as "buttons 202"). Touch sensors 204_1 to 204_4 (hereinafter, sometimes collectively referred to as "touch sensors 204") that can detect the approach of or contact with the buttons 202_1 to 202_4 by the user are provided on the upper surfaces (exposed surfaces) of the buttons 202_1 to 202_4 that accept user operations. By employing such a configuration, the button operation unit 206 can detect the approach of or contact with any of the buttons 202 by the user, in addition to being able to detect the user's pressing any of the buttons 202.

[0059] In this specification, a "touch" refers to the approach or contact of a user's finger (or a part of the user's body) with any of the buttons 202. Note that the extent to which the user's finger must approach the button 202 before the system 1 determines that a "touch" has occurred can be designed as desired.

[0060] The direction indicator 208 receives a direction indication (for example, one of four directions or an angle) from the user. For example, the direction indicator 208 may be a slide stick that the user tilts a protrusion to indicate a direction, an analog stick that the user slides a protrusion to indicate a direction, a cross-shaped button, or a set of buttons arranged in each of the four directions.

[0061] Specific examples of processing in response to user operations on controller 200 will be described later. Controller 200 may be attachable to main unit 100. When controller 200 is separated from main unit 100, data is transmitted and received between main unit 100 and controller 200 via wireless communication. When controller 200 is attached to main unit 100, data is transmitted and received between main unit 100 and controller 200 via wired communication and / or wireless communication.

[0062] The controller 200 can be held by the user either by itself or attached to the main unit 100 .

[0063] 2 is a schematic diagram showing an example in which controllers 200 are used in a state where they are attached to main unit 100 in system 1 according to the present embodiment. Referring to FIG. 2 , a user can use main unit 100 by holding main unit 100 with a pair of controllers 200 attached.

[0064] 2 , when two controllers 200 are used while attached to main unit 100, a button operation unit 206 and a direction indication unit 208 are provided in an area operable with one finger of a user holding one controller 200, and another button operation unit 206 and a direction indication unit 208 are provided in another area operable with one finger of a user holding the other controller 200. In this configuration, the button operation unit 206 (comprising a plurality of buttons 202) provided on one controller 200 and the other button operation unit 206 (comprising a plurality of buttons 202) provided on the other controller 200 can independently detect user touches to the plurality of buttons 202.

[0065] 3 is a schematic diagram showing an example in which controller 200 is used in system 1 according to the present embodiment while being detached from main device 100. Referring to FIG. 3 , with main device 100 placed in dock 302, one or more users operate controller 200 while viewing an image output on external display 300.

[0066] As another example of a case in which controller 200 is used while detached from main device 100, main device 100 may be placed so that the user can see display 106, and one or more users may operate controller 200 while looking at the image output on display 106.

[0067] 2 and 3, the four buttons 202_1 to 202_4 that make up the button operation unit 206 of the controller 200 are arranged in a ring. This arrangement allows the user to perform cyclical touch operations on the buttons 202_1 to 202_4 in any direction. This makes it easier for the user to perform operations such as sequentially touching the buttons in a clockwise and / or counterclockwise direction, for example.

[0068] FIG. 1 shows an example of a configuration in which buttons 202 are arranged on the top, bottom, left and right sides, but the buttons 202 may be arranged in any way as long as the user can perform touch operations in a circular manner.

[0069] [B. Example of Hardware Configuration of System 1] Next, an example of a hardware configuration of system 1 according to the present embodiment will be described.

[0070] (b1: Main unit 100) Fig. 4 is a schematic diagram showing an example of a hardware configuration of main unit 100 of system 1 according to the present embodiment. Referring to Fig. 4, main unit 100 includes one or more processors 102, memory 104, storage 120, display 106, speaker 108, wireless communication module 110, and wired communication module 112.

[0071] The processor 102 is a processing entity that executes processes provided by the main device 100. The processor 102 executes various processes and outputs images generated by the execution of the processes. The memory 104 is a storage device accessible by the processor 102, and is, for example, a volatile storage device such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). The storage 120 is, for example, a non-volatile storage device such as a flash memory.

[0072] The processor 102 performs the processing described below by reading a program stored in the storage 120, expanding it into the memory 104, and executing it. The storage 120 stores, for example, a system program 122 that provides libraries and the like necessary for program execution, an application program 124 consisting of computer-readable instruction codes for realizing any information processing, and application data 126 that is referenced when the application program 124 is executed.

[0073] In this specification, the term "processor" not only refers to a processing circuit that executes processing according to instruction codes written in a program, such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), but also includes hardwired circuits such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). Hardwired circuits such as an ASIC or FPGA have pre-formed circuits corresponding to the processing to be executed. Furthermore, in this specification, the term "processor" also includes circuits that integrate multiple functions, such as an SoC (System on Chip), and also includes a combination of a processor in the narrow sense and a hardwired circuit. Thus, a "processor" herein may also be referred to as a processing circuitry.

[0074] The display 106 displays an image based on the processing result of the processor 102. The speaker 108 generates any sound around the main device 100.

[0075] The wireless communication module 110 transmits and receives wireless signals to and from any device. For example, when transmitting and receiving wireless signals to and from the controller 200 operated by the user, the wireless communication module 110 can employ any wireless method, such as Bluetooth (registered trademark), ZigBee (registered trademark), wireless LAN (IEEE802.11), or infrared communication.

[0076] When the controller 200 is attached to the main unit 100 , the wired communication module 112 transmits and receives wired signals to and from the attached controller 200 .

[0077] In addition, the main unit 100 may have a wireless communication unit for sending and receiving wireless signals between a wireless repeater connected to the Internet, etc., and an image output unit for outputting images to the external display 300 via the dock 302.

[0078] (b2: Controller 200) Fig. 5 is a schematic diagram showing an example of the cross-sectional structure of button 202 of controller 200 of system 1 according to the present embodiment. Referring to Fig. 5, button 202 includes key top 220 provided such that a portion of button 202 protrudes from an opening provided in housing 214 of controller 200. Key top 220 of button 202 is configured independently from other buttons 202. Key top 220 may be made of a non-plastic, non-conductive resin. As a variation, key top 220 may be made of a conductive material.

[0079] A touch sensor 204 is provided on the top surface of the key top 220 that receives user operations.

[0080] A key rubber 218 is provided on the inside of the controller 200 of the key top 220. The key rubber 218 as a whole elastically deforms due to a force received from the top surface of the key top 220. The key rubber 218 may be made of an elastically deformable material (for example, it may be made of a flexible non-conductive resin or rubber).

[0081] A fixed contact 212 consisting of two separate conductors is provided on a substrate 216 provided inside the controller 200. A movable contact 210 is provided on a portion of the key rubber 218 facing the fixed contact 212. The movable contact 210 is made of a conductive material such as conductive carbon. When the key top 220 is pressed, the key rubber 218 and the movable contact 210 move toward the inside of the controller 200, and the movable contact 210 comes into contact with the fixed contact 212, electrically connecting the two conductors that make up the fixed contact 212.

[0082] 6 is a schematic diagram showing an example of a hardware configuration of controller 200 of system 1 according to the present embodiment. Referring to Fig. 6, controller 200 includes a press determination unit 230 electrically connected to four buttons 202_1 to 202_4, a touch detection unit 232 electrically connected to touch sensors 204_1 to 204_4, and an output processing unit 234.

[0083] Each of the buttons 202_1 to 202_4 includes a fixed contact 212 and a movable contact 210 as a sensor component capable of detecting pressing of the button 202_1 to 202_4. When the position of the movable contact 210 changes due to a user operation, the movable contact 210 brings the fixed contact 212 into conduction. The press determination unit 230 determines whether the corresponding button 202 is pressed or not based on the conduction state of the fixed contact 212.

[0084] Each of the touch sensors 204_1 to 204_4 changes its own capacitance depending on, for example, the distance from the user's finger. The touch detection unit 232 determines whether or not the corresponding button 202 has been touched based on the capacitance generated in each of the touch sensors 204_1 to 204_4. In this case, whether or not each of the buttons 202 has been touched can be digitally detected.

[0085] Furthermore, based on the capacitance generated in each of the touch sensors 204_1 to 204_4, it may be possible to detect in which direction the user's finger is present on the button 202. In this case, it is possible to detect the direction of movement of the touch on the button 202. In other words, it becomes possible to detect the user's gesture on the button 202.

[0086] The detection method of the touch sensor 204 is not limited to the capacitance method, and any method such as an ultrasonic method, an optical method, or a resistive film method may be adopted.

[0087] The output processing unit 234 outputs the determination results of the press determination unit 230 and the touch detection unit 232. For example, the output processing unit 234 outputs a press signal indicating whether each of the buttons 202_1 to 202_4 is pressed or not, and a touch signal indicating the state of touch on each of the buttons 202_1 to 202_4.

[0088] The output processing unit 234 may have a circuit for generating wireless and / or wired signals for transmitting and receiving press signals and touch signals to and from the main unit 100 .

[0089] The touch signal output by the output processing unit 234 may be a signal indicating whether the button 202 is being touched (for example, a binary value of "ON" when it is determined that the button 202 is being touched, and "OFF" otherwise), or a signal indicating approach and contact with the button 202 (for example, a ternary value of "1" when the button 202 is being approached but not being contacted, "2" when the button 202 is being contacted, and "0" otherwise). The touch signal may also be a signal that analogically indicates the degree of approach according to the capacitance generated in the touch sensor 204 (for example, a value normalized to a range of 0 to 100, with 100 representing a state of contact).

[0090] The process for determining the presence or absence of a touch or the state thereof may be implemented in controller 200, in main unit 100, or in both.

[0091] With the structure shown in FIGS. 5 and 6, it is possible to detect whether the button 202 is pressed and whether the button 202 is touched.

[0092] FIG. 5 shows an example of a structure in which the touch sensor 204 is provided on the top surface of the button 202 that receives user operations, but the present invention is not limited to this, and the touch sensor 204 may be provided in any position as long as it can detect a touch on the button 202.

[0093] Furthermore, instead of providing a touch sensor 204 for each button 202, a common touch sensor 204 may be provided for multiple buttons 202. For example, a touch sensor may be provided on the inner surface of the housing 214 of the controller 200 shown in FIG. 5 so as to encompass the area where buttons 202_1 to 202_4 are present, or multiple touch sensors, four or less or five or more, may be provided near buttons 202_1 to 202_4. In this case, it is possible to detect (calculate) whether one or multiple buttons 202 are approaching or in contact with each other based on the touch position detected by the touch sensor (the position where the user's finger is detected as approaching or in contact with each button 202).

[0094] (b3: Modification) While the configuration of the controller 200 that can be held by a user has been shown, a configuration capable of detecting button presses and touches by the user may also be employed in, for example, a stationary controller. The stationary controller may include, for example, a joystick operated with the user's left hand and multiple buttons operated with the user's right hand (e.g., two buttons each corresponding to the index finger, middle finger, and ring finger, for a total of six buttons), and may be provided integrally with a stationary game device or may be placed on the floor for use. Each button may be provided with a touch sensor as described above.

[0095] Note that in any controller, a directional input unit (analog stick, slide stick, joystick, etc.) is not a required component, and only buttons (and touch sensors) may be provided. The number and layout of buttons can also be designed as desired.

[0096] An example of a user experience provided by system 1 according to the present embodiment will now be described. In the user experience described below, a process is executed to change an image output from main unit 100 in response to a user operation on controller 200.

[0097] C. User Experience Example 1 As an example of the user experience, a process of moving a cursor to select one of a plurality of displayed items (hereinafter also referred to as a "cursor movement process") will be described.

[0098] (c1: Cursor Movement Processing Example 1) Fig. 7 is a schematic diagram showing an example of cursor movement processing in the system according to the present embodiment. Fig. 8 is a schematic diagram showing an example of a user operation corresponding to the cursor movement processing shown in Fig. 7.

[0099] 7, a plurality of items are displayed as selection candidates, and a cursor 312 is displayed for selecting one of the plurality of items included in item group 310. The user can operate controller 200 to move cursor 312 so as to select another item.

[0100] FIG. 7 shows, as examples of movement of the cursor 312, a movement process MP1 for moving the cursor 312 to the adjacent upper position and a movement process MP2 for moving the cursor 312 to the adjacent lower position.

[0101] 8A shows an example of the operation of the controller 200 corresponding to the movement process MP1 in FIG. 7, and FIG. 8B shows an example of the operation of the controller 200 corresponding to the movement process MP2 in FIG.

[0102] As shown in Figures 8(A) and 8(B), an operation in which a user touches (approaches or contacts) two or more buttons 202 with their finger in sequence is also referred to below as a "movement operation." Main unit 100 executes various processes based on the order in which buttons 202 are touched in the user's movement operation. That is, the process to be executed is determined based on the button 202 touched just before or before the button 202 that is finally touched in the series of movement operations.

[0103] 8(A) and 8(B), when sequential touches on three different buttons 202 on the controller 200 are detected, a process of moving the cursor 312 is executed. That is, in the example shown in Figures 8(A) and 8(B), the target movement operation includes the user's finger touching three or more buttons 202 in sequence.

[0104] 8A shows an example of an operation in which the user sequentially touches three buttons 202 in a counterclockwise direction. More specifically, at time t1, the user touches button 202_1, at the following time t2, the user touches button 202_4, and at the following time t3, the user touches button 202_3. In response to touching the three buttons 202 from time t1 to time t3, the cursor 312 moves to the adjacent button above by the movement process MP1 shown in FIG.

[0105] Furthermore, at time t4 following time t3, the user presses the button 202_3 that he has been touching, and the item selected by the cursor 312 at time t4 is then confirmed.

[0106] 8B shows an example of an operation in which the user sequentially touches three buttons 202 in a clockwise direction. More specifically, at time t1, the user touches button 202_1, at subsequent time t2, the user touches button 202_2, and at subsequent time t3, the user touches button 202_3. In response to touching the three buttons 202 from time t1 to time t3, the cursor 312 moves to the adjacent button below by the movement process MP2 shown in FIG.

[0107] Furthermore, at time t4 following time t3, the user presses the button 202_3 that he has been touching, and the item selected by the cursor 312 at time t4 is then confirmed.

[0108] In this way, the operation of confirming the selection of an item may be performed by the user pressing the button 202 that the user is touching while the cursor 312 is selecting the desired item. In this case, regardless of which button 202 is ultimately touched after the movement operation, the process of confirming the selection of the item may be performed by pressing the button 202. In other words, the type of button 202 for performing the process of confirming the selection of the item may not matter.

[0109] Furthermore, the process of determining the selection of an item may be executed based on the user's pressing of the last button 202 in the movement operation while the button 202 is still in touch with the user's finger. That is, in order for the button 202_4 to be pressed and the process of determining the selection of an item to be executed at time t4 in FIGS. 8(A) and 8(B), the state in which the user's finger is touching the button 202_4 may be maintained as a condition. Note that when the user's finger releases its touch on the last button 202 in the movement operation (is no longer in proximity or contact), and the button 202 is pressed again, a function previously assigned to the button 202 may be executed.

[0110] As shown at times t1 to t3 in Figures 8A and 8B, when a user performs a movement operation in which three or more buttons 202 are touched in sequence with their finger, main unit 100 executes a cursor movement process (first process) based on the order in which buttons 202 are touched in the movement operation. At this time, different cursor movement processes are executed depending on the content of the movement operation in which the user performs a movement operation in which three or more buttons 202 are touched in sequence with their finger. In other words, main unit 100 changes the content of the cursor movement process depending on whether the order in which buttons 202 are touched in the movement operation is clockwise or counterclockwise.

[0111] Also, as shown at time t4 in Figures 8 (A) and 8 (B), after executing the cursor movement process (first process), the main unit 100 executes a process (second process) to determine the selection of an item based on the button 202 that is last touched among the multiple buttons 202 being pressed.

[0112] 8, if the button 202 is pressed while the cursor movement process is not being executed, the process of confirming the selection of an item is not executed. If the button 202 is pressed alone while the cursor movement process is not being executed, no process may be executed. Alternatively, a function previously assigned to the pressed button 202 may be executed, and in this case, a function of confirming the selection of an item may be assigned to one of the buttons 202.

[0113] Furthermore, the button 202 for executing the process of determining the selection of an item while the cursor movement process is being executed does not have to be limited to the button 202 that is last touched. That is, the main unit 100 may be configured to execute the process of determining the selection of an item (second process) when a specific button 202 out of the plurality of buttons 202 is pressed while the cursor movement process is being executed. Furthermore, the process of determining the selection of an item may be executed regardless of which button 202 is pressed.

[0114] For ease of explanation, Figures 8(A) and 8(B) show an example in which only one button 202 is touched at each time, but this is not limited to this, and the cursor movement process may be performed even if multiple buttons 202 are touched simultaneously at a certain time.

[0115] More specifically, when a touch on a certain button 202 is detected followed by a touch on another button 202 adjacent to the button 202, it may be determined that the two buttons 202 have been touched in succession, regardless of whether the touch on the previously touched button 202 continues. In other words, when two buttons 202 have been touched in succession, it is not necessary to use the fact that the previously touched button 202 has ceased to be touched as a determination condition. This is because, for example, when the buttons 202 are arranged closely to each other, it is possible that the previously touched button 202 remains touched while another button 202 is touched.

[0116] On the other hand, the determination condition may be that a touch on the previously touched button 202 is no longer detected. That is, it may be determined that two buttons 202 have been touched sequentially only when a touch on a button 202 adjacent to the previously touched button 202 is detected after the touch on the previously touched button 202 is no longer detected. In this case, even if a touch on another button 202 adjacent to the adjacent button 202 is also detected in addition to the adjacent button 202, it may be determined that at least two buttons 202 have been touched sequentially.

[0117] 9 is a schematic diagram showing another example of a user operation corresponding to the cursor movement process in the system according to the present embodiment. Referring to FIG. 9 , an example of an operation is shown in which the user touches three buttons 202 in a clockwise direction in succession, and then touches another button 202. More specifically, at time t1, the user touches button 202_1, at subsequent time t2, the user touches button 202_2, at subsequent time t3, the user touches button 202_3, and at subsequent time t4, the user touches button 202_4.

[0118] At time t1 and time t2, two buttons 202 are touched, resulting in a transition to a state waiting for cursor movement input. In the state waiting for cursor movement input, at time t3, button 202_3 is further touched, which activates cursor movement mode and moves the cursor 312.

[0119] Here, the cursor movement mode is a state in which, when a touch on a certain button 202 is detected, a cursor movement process is executed based on the positional relationship between the touched button 202 and the previously touched button 202. The cursor movement input wait state is a state in which a user operation (touch on a specific button 202) to activate the cursor movement mode is waited for.

[0120] For example, at time t4, button 202_4 is touched, which further moves cursor 312. Similarly, as buttons 202 are successively touched, cursor 312 continues to move in a manner corresponding to the order in which buttons 202 are touched.

[0121] For ease of explanation, Figure 9 shows an example in which the buttons 202 are touched in a fixed order (clockwise), but in cursor movement mode, touches on the buttons 202 that satisfy a predetermined rule (e.g., touching adjacent buttons 202 sequentially) may be treated as valid operations. Therefore, for example, even if the buttons 202 are touched sequentially in a clockwise direction and then touched sequentially in a counterclockwise direction, the cursor movement process may continue. When the buttons 202 are touched sequentially in a counterclockwise direction, the cursor 312 may move in a different direction than when the buttons 202 are touched sequentially in a clockwise direction.

[0122] On the other hand, if a certain button 202 is touched and then another button 202 is not touched within a predetermined time, the cursor movement input wait or cursor movement mode may be cancelled. That is, when a predetermined time has elapsed since the detected touch movement stopped, the cursor movement input wait or cursor movement mode may be cancelled. Similarly, when a predetermined time has elapsed since none of the buttons 202 was touched, the cursor movement input wait or cursor movement mode may be cancelled.

[0123] Furthermore, the process of determining the selection of an item may be executed based on whether the button 202 is pressed within a predetermined time period after the cursor movement process is executed. That is, the process of determining the selection of an item may be executed when any button 202 is pressed within a predetermined time period after the user's finger touches the last button 202 in the movement operation, or within a predetermined time period after the user releases touch on the last button 202 in the movement operation.

[0124] Furthermore, when one button 202 is touched and then another button 202 is touched, the cursor movement input wait or cursor movement mode may be cancelled even if the two buttons 202 are not adjacent to each other. For example, when opposing buttons 202 are touched rather than when adjacent buttons 202 are touched sequentially, the cursor movement input wait or cursor movement mode may be cancelled.

[0125] 10 is a schematic diagram showing an example of a user operation that cancels the cursor movement input wait or cursor movement mode in the system according to the present embodiment. Referring to FIG. 10, assume that at time t1, the user touches button 202_1, and then at time t2, the user touches button 202_2. The user operations at times t1 and t2 result in a transition to a cursor movement input wait state. In the cursor movement input wait state, it is expected that buttons 202 will be touched sequentially in a clockwise direction.

[0126] At the next time t3, the user touches the button 202_4 opposite the button 202_2, instead of the button 202_3 adjacent to the button 202_2. This violates the rule that adjacent buttons 202 are touched sequentially, and the wait for cursor movement input is canceled. Similarly, even when the cursor movement mode is enabled, the cursor movement mode may be canceled when a touch on the button 202_4 opposite the button 202_2 is detected following a touch on the button 202_2.

[0127] 11 is a flowchart showing the procedure for cursor movement processing in the system according to the present embodiment. Each step shown in FIG. 11 may be realized by one or more processors 102 of main unit 100 executing system program 122 and / or application program 124.

[0128] Referring to FIG. 11, the processor 102 of the main device 100 displays a screen including a plurality of items in response to a user operation (step S100), and displays a cursor 312 corresponding to one of the displayed items in accordance with a predetermined initial setting (step S102).

[0129] Next, the processor 102 determines whether a touch to any of the buttons 202 has been detected (step S104). The button 202 for which a touch has been detected in step S104 is referred to as the "first-touched button 202." If a touch to any of the buttons 202 has not been detected (NO in step S104), the process of step S104 is repeated.

[0130] When a touch on any of the buttons 202 is detected (YES in step S104), the processor 102 identifies one or more buttons 202 adjacent to the initially touched button 202 (step S106). The processor 102 then determines whether a touch on the identified button 202 is detected (step S108). If a touch on the identified button 202 is not detected (NO in step S108), the process of step S108 is repeated. Note that, for example, if a touch on a button (not shown) other than the identified button is detected, the button on which the touch was detected may be set as the "first touched button," and the processes from step S106 onward may be executed.

[0131] When a touch on the identified button 202 is detected (YES in step S108), processor 102 transitions to a state waiting for cursor movement input (step S110). The button for which a touch is detected in step S108 is referred to as the "second-touched button 202." Then, processor 102 identifies the button 202 to be touched third based on the positional relationship between the first-touched button 202 and the second-touched button 202 (step S112).

[0132] If a touch on an adjacent button is not detected within a predetermined time from the detection of a touch on the first touched button 202, the state may not transition to a state waiting for cursor movement input.

[0133] Next, processor 102 determines whether a touch on button 202 that should be touched third has been detected (step S114). If a touch on button 202 that should be touched third has not been detected (NO in step S114), processor 102 determines whether a condition for canceling the wait for cursor movement input has been met (step S116).

[0134] Conditions for canceling the state of waiting for cursor movement input include (1) when the third button 202 to be touched is not touched within a predetermined time from when the first button 202 was touched, and (2) when a state in which none of the buttons 202 is touched continues for more than a predetermined time. The conditions for canceling the state of waiting for cursor movement input may adopt only a part of these two conditions, or may include other conditions.

[0135] If the condition for canceling the wait for cursor movement input is met (YES in step S116), processor 102 cancels the wait for cursor movement input state (step S118). Then, processor 102 determines whether the condition for ending the display of a screen including multiple items is met (step S140).

[0136] If the condition for canceling the wait for cursor movement input is not met (NO in step S116), the process of step S114 is repeated.

[0137] When a touch on the button 202 that should be touched third is detected (YES in step S114), the processor 102 activates a cursor movement mode (step S120) and moves the displayed cursor 312 in a direction corresponding to the order of the second-touched button and the third-touched button 202 (step S122).Then, the processor 102 identifies one or more buttons adjacent to the most recently touched button (step S124).

[0138] Next, processor 102 determines whether a touch to any of buttons 202 identified in step S124 has been detected (step S126). If a touch to any of buttons 202 identified in step S124 has been detected (YES in step S126), processor 102 moves displayed cursor 312 in a direction corresponding to the order of the detected touches (step S128). Then, the processing from step S124 onward is repeated.

[0139] In this way, when a movement operation is performed in which the user's finger touches two or more buttons 202 in sequence, the main unit 100 executes a cursor movement process (first process) based on the order in which the buttons 202 are touched in the movement operation.

[0140] If a touch on the button 202 identified in step S124 is not detected (NO in step S126), processor 102 determines whether pressing of the button 202 for which a touch is currently detected has been detected (step S130). If pressing of the button 202 for which a touch is currently detected is detected (YES in step S130), processor 102 determines the selection of the item corresponding to the current cursor 312 (step S132). Next, processor 102 executes processing associated with the determination of the selection of the item (step S134). Note that the processing associated with the determination of the selection of the item may be, for example, processing such as displaying details of the selected item. Then, the cursor movement processing ends.

[0141] If the pressing of button 202, for which a touch is currently detected, is not detected (NO in step S130), processor 102 determines whether a condition for canceling the cursor movement mode is met (step S136).

[0142] As described above, the conditions for canceling the cursor movement mode include (1) when another button 202 is not touched within a predetermined time period after the most recent touch of the button 202, (2) when a state in which none of the buttons 202 is touched continues for more than a predetermined time period, and (3) when a button 202 other than the adjacent button 202 is touched. The conditions for canceling the cursor movement mode may adopt only some of these three conditions, or may include other conditions.

[0143] If the condition for canceling the cursor movement mode is not met (NO in step S136), the processing from step S126 onwards is repeated.

[0144] If the condition for canceling the cursor movement mode is met (YES in step S136), processor 102 cancels the cursor movement mode (step S138). Then, processor 102 determines whether the condition for ending the display of a screen including multiple items is met (step S140).

[0145] If the condition for ending the display of the screen including a plurality of items is not met (NO in step S140), the processing from step S104 onwards is repeated.

[0146] If the condition for ending the display of the screen including multiple items is met (YES in step S140), the process ends.

[0147] In this way, in the cursor movement process according to the present embodiment, cursor 312 is moved on the condition that three different buttons 202 on controller 200 are touched. By setting three different buttons 202 as the condition, it is possible to prevent unintended movement of cursor 312, even if the user accidentally operates two buttons 202.

[0148] The correspondence between the sequence pattern of the user's touch on the button 202 (for example, touching clockwise or counterclockwise) and the movement direction and amount of movement of the cursor 312 can be designed arbitrarily.

[0149] In the above description, an example of processing has been shown in which cursor 312 is moved in response to detection of touch on three different buttons 202, but this is not limiting, and cursor 312 may be moved in response to detection of touch on two different buttons 202, or detection of touch on four (or more) different buttons 202. For example, if the condition is detection of touch on two different buttons 202, cursor 312 may be moved when a touch on another button 202 is detected after a touch on the first button 202 has been detected.

[0150] (c2: Cursor Movement Processing Example 2) In the above-described cursor movement processing example 1, a transition to a state waiting for cursor movement input occurs when two buttons 202 are touched. The operation of transitioning to the state waiting for cursor movement input may include not only touching two buttons 202 but also pressing two buttons 202 simultaneously.

[0151] 12 is a schematic diagram showing an example of a user operation for transitioning to a state waiting for cursor movement input in the system according to this embodiment. Referring to FIG. 12, assume that the user simultaneously presses buttons 202_1 and 202_2. The pressed buttons 202_1 and 202_2 are adjacent to each other, and this operation transitions to a state waiting for cursor movement input. Subsequently, when the user touches button 202_3, cursor movement mode is enabled, and cursor 312 moves to the adjacent lower position by movement processing MP2 shown in FIG. 7.

[0152] Note that even if the user touches the button 202_4 instead of the button 202_3, the cursor movement mode is enabled and the cursor 312 moves to the upper adjacent position by the movement process MP1 shown in FIG.

[0153] If the user touches buttons 202_1 and 202_2 at the same time, and then releases touch on button 202_1 while maintaining touch on button 202_2, the cursor movement mode may be activated by touching button 202_3 next, and the cursor 312 may be moved to the adjacent button below. In this case, even if button 202_1 and / or button 202_4 is touched, the cursor movement mode may not be activated, and the cursor 312 may not be moved.

[0154] In this way, the condition for transitioning to the state waiting for cursor movement input may include the operation of simultaneously pressing two adjacent buttons 202. In this processing example, a movement operation in which the user's finger touches two or more buttons 202 in sequence is not necessarily required, and the processing is executed by a combination of pressing and touching buttons 202.

[0155] (c3: Cursor Movement Processing Example 3) In the cursor movement mode, a function specific to the cursor movement mode may be assigned to the button 202 of the controller 200 .

[0156] 13 is a schematic diagram showing an example of function assignment in the cursor movement mode in the system according to the present embodiment. Referring to FIG. 13, it is assumed that the user touches button 202_1 and then button 202_2. This operation causes the system to transition to a state waiting for cursor movement input.

[0157] When the user touches the button 202_3 in the state waiting for cursor movement input, the cursor movement mode is activated, and the cursor 312 moves to the next lower position in the movement process MP2 shown in FIG.

[0158] In addition, in the cursor movement mode, functions specific to the cursor movement mode may be assigned to at least some of the buttons 202_1 to 202_4 of the controller 200. Examples of functions specific to the cursor movement mode include confirm, cancel, page forward, and page back.

[0159] 13 shows an example in which "page forward" is assigned to button 202_3. When the user presses button 202_3, a page forward process is executed. Similarly, when the other buttons 202 are pressed, the functions assigned to the respective buttons 202 may be executed.

[0160] Note that the assignment of functions specific to the cursor movement mode to at least some of the buttons 202_1 to 202_4 of the controller 200 may be performed in the cursor movement mode, and when the cursor movement mode is released, other functions may be assigned to the buttons 202_1 to 202_4.

[0161] In this way, a function (second process) specific to the cursor movement mode may be executed based on the fact that at least one of the plurality of buttons 202 is pressed following the movement operation.

[0162] The cursor movement mode may be cancelled if a state in which none of the buttons 202 is touched continues for a predetermined time (e.g., three seconds) or more. For example, the predetermined time may be determined taking into consideration the operation time required for the user to press the button 202 after touching the button 202.

[0163] (c4: Cursor Movement Processing Example 4) The user may not only operate one controller 200, but may also operate two controllers 200 simultaneously (see FIG. 2, etc.). Alternatively, one controller 200 may have a group of touch-detectable buttons on both the right and left sides, and these button groups may be operated simultaneously. Even in such a case, the above-described cursor movement processing can be performed.

[0164] 14 is a schematic diagram showing an example of a user operation corresponding to a cursor movement process when two controllers 200 are used in a system according to the present embodiment. With reference to Fig. 14, for example, the user operates controller 200L with his left hand and controller 200R with his right hand.

[0165] Suppose the user touches the button 202_4 of the controller 200L and then touches the button 202_1. Then, after waiting for cursor movement input, the cursor movement mode is enabled. In the cursor movement mode, functions specific to the cursor movement mode may be assigned to the buttons 202 of the controller 200L and the controller 200R.

[0166] For example, functions such as enter, cancel, page forward, and page back may be assigned to at least some of the buttons 202_1 to 202_4 of the controller 200R.

[0167] Furthermore, the buttons 202_1 to 202_4 of the controller 200L may be assigned with functions of moving the cursor 312 to the adjacent right, below, to the adjacent left, and above, respectively. That is, in the cursor movement mode, the user can move the cursor 312 by pressing any of the buttons 202_1 to 202_4 of the controller 200L.

[0168] In this way, following a movement operation on button 202 of controller 200R, a function specific to the cursor movement mode (first process) may be executed based on the pressing of at least one of buttons 202 of controller 200L, rather than button 202 of controller 200R.

[0169] On the other hand, it may be difficult to determine the user's intention when the user simply touches one of the buttons 202_1 to 202_4 on the controller 200L, and specific processing in the cursor movement mode may not be executed.

[0170] The cursor movement mode may be enabled by operating the button 202 of the controller 200R.

[0171] (c5: Cursor Movement Process Example 5) In the above description, a process example in which the cursor 312 moves along items arranged in a line has been exemplified, but the items may also be arranged in a matrix.

[0172] 15 is a schematic diagram showing another example of cursor movement processing in the system according to the present embodiment. Referring to FIG. 15, a cursor 312 for selecting one of items in a group 310 of items arranged in a matrix may be configured to be movable.

[0173] 15, in response to a user operation, a movement process may be executed to move the cursor 312 to the adjacent right or left, or to move the cursor diagonally. Furthermore, instead of moving the cursor 312 to the adjacent position, the cursor 312 may be moved a plurality of positions, such as two positions forward.

[0174] In this way, the number of items that the cursor 312 moves over and the arrangement method thereof can be designed as desired.

[0175] [D. User Experience Example 2] As another example of the user experience, a game process that progresses based on the detection results of pressing and touching the button 202 by the user will be described.

[0176] (d1: Game Processing 1) Fig. 16 is a schematic diagram showing an example of game processing in the system according to the present embodiment. Referring to Fig. 16, a user operates character object 330 with controller 200. The user can move character object 330 by operating direction indicating unit 208.

[0177] 16(A) and 16(B) show an example in which a plurality of buttons 202 are touched in sequence to activate a technique (skill) set in advance for the character object 330. In FIG.

[0178] More specifically, referring to Fig. 16(A), skill 1 of character object 330 is activated when the user touches button 202_3 and then button 202_4. Referring to Fig. 16(B), skill 2 of character object 330 is activated when the user touches button 202_4 and then button 202_3. Note that, during a movement operation, the appearance of character object 330 may not change due to the movement operation. Conversely, an effect suggesting that a movement operation is in progress may be displayed.

[0179] In this way, character object 330 may be made to perform a specific action on the condition that touches on a plurality of different buttons 202 are detected. That is, when a movement operation in which the user's finger touches two or more buttons 202 in sequence is performed, main device 100 executes activation of a skill (first process) based on the order of buttons 202 touched in the movement operation. By setting the condition that touches on a plurality of different buttons 202 are detected, it is possible to prevent character object 330 from moving in a manner unintended by the user.

[0180] 16(C) and 16(D) show an example in which the character object 330 is operated by touching the button 202 and then pressing the button 202.

[0181] More specifically, referring to FIG. 16(C), when the user touches the button 202_3, the character object 330 enters a state of preparation for attack, and when the user subsequently presses the button 202_3, the character object 330 performs an attack.

[0182] Referring to FIG. 16(D), when the user touches the button 202_4, the character object 330 enters a jump preparation state, and when the user subsequently presses the button 202_4, the character object 330 jumps.

[0183] Note that actions such as attack and jump that are executed by pressing buttons 202_3 and 202_4 may or may not be executed on the condition that touches on buttons 202_3 and 202_4 have been detected immediately beforehand. In the latter case, pressing buttons 202_3 and 202_4 causes character object 330 to perform the corresponding action.

[0184] The behavior of the character object 330 shown in FIG. 16 is an example, and the behavior of the character object 330 corresponding to touching multiple different buttons 202 and / or a combination of touching a button 202 and pressing a button 202 can be designed arbitrarily.

[0185] (d2: Game Process 2) In the above-described game process 1, an example was shown in which the skill of the character object 330 is activated by sequentially touching a plurality of buttons 202, but pressing a button 202 may also be included in the activation conditions.

[0186] 17 is a schematic diagram showing an example of a user operation corresponding to an example of game processing in the system according to the present embodiment. The user operation shown in FIG. 17 causes the character object 330 to move as shown in FIG.

[0187] 17(A) and 17(B) show an example in which a plurality of buttons 202 are touched in sequence and then a button 202 is pressed down, thereby activating a skill of a character object 330.

[0188] More specifically, referring to FIG. 17(A), when the user touches button 202_3, then touches button 202_4, and then presses button 202_4, skill 1 of character object 330 is activated.

[0189] Referring to FIG. 17(B), when the user touches button 202_4, then touches button 202_3, and then presses button 202_3, skill 2 of character object 330 is activated.

[0190] In this way, after touches to a plurality of different buttons 202 are detected, character object 330 may be caused to perform a specific action on the condition that button 202 is pressed. That is, main device 100 may execute activation of a skill (first process) based on the pressing of at least one of the plurality of buttons 202 following a movement operation. By making pressing of button 202 a condition in addition to detecting touches to a plurality of different buttons 202, it is possible to prevent character object 330 from performing an action unintended by the user.

[0191] Note that a skill may be activated when any button 202 is pressed, but for example, as shown in Figures 17(A) and 17(B), a skill may be executed based on the button 202 that was last touched in a movement operation being pressed following the movement operation. Also, a specific button for activating a skill may be provided, which is pressed last to activate the skill.

[0192] Furthermore, the activation of a skill may be executed based on the user's finger touching the last button 202 in the movement operation and then pressing the button 202. That is, in Fig. 17(A) or 17(B), in order for the process of pressing the button 202_4 or 202_3 to activate a skill, the state in which the user's finger is touching the button 202_4 or 202_3 may be maintained as a condition.

[0193] Furthermore, the skill may be activated based on the button 202 being pressed within a predetermined time after the user's finger is released from the button 202 that was last touched by the user's finger during a movement operation. The skill can be activated by pressing the button 202 within the predetermined time after the touch on the button 202 is released. Therefore, even if the user unintentionally releases their finger from the button 202, they can continue the operation as intended.

[0194] In addition, some or all of buttons 202_1 to 202_4 may be assigned a function such that when pressed alone (i.e., without a previous touch being detected), the character object 330 performs a specific action (e.g., attack, jump, etc.).

[0195] (d3: Game Processing 3) The user may operate not only one controller 200 but also two controllers 200 simultaneously (see FIG. 2, etc.). Even in such a case, the above-described game processing can be executed.

[0196] 18 is a schematic diagram showing an example of a user operation corresponding to game processing in a system according to the present embodiment when two controllers 200 are used. The user operation shown in FIG. 18 causes character object 330 to move as shown in FIG.

[0197] Figures 18(A) and 18(B) show an example in which a user sequentially touches multiple buttons 202 on controller 200L with his / her left hand and then presses button 202 on controller 200R with his / her right hand, thereby activating the skill of character object 330.

[0198] More specifically, referring to Figure 18 (A), the user touches button 202_2 on controller 200L, then touches both buttons 202_2 and 202_1, then touches button 202_1, and then presses button 202_3 on controller 200R, thereby activating skill 1 of character object 330.

[0199] In this way, following a movement operation on button 202 of controller 200R, the activation of a skill (first process) may be performed based on the pressing of at least one of buttons 202 of controller 200L, rather than button 202 of controller 200R.

[0200] For example, in a game process according to the related art, a specific skill is activated by operating the direction instruction unit 208 to sequentially input predetermined directions, followed by pressing the button 202. When such a user operation is employed, there is a possibility that the character object 330 will be moved even though moving the character object 330 is not the purpose, since the direction instruction unit 208 is operated.

[0201] In contrast to this, in the above-described game process 3, since operation of the direction instruction unit 208 is not required to activate a skill, the character object 330 will not be moved unintentionally.

[0202] [E. User Experience Example 3] Controller 200 according to this embodiment can detect that a user's finger has approached button 202. Therefore, it is possible to estimate a user operation on controller 200 based on a change in the detection results of touch sensors 204_1 to 204_4. Below, a processing example that can detect a user operation more precisely will be described.

[0203] Figure 19 is a schematic diagram showing yet another example of game processing in the system according to the present embodiment. Referring to Figure 19, a user operates character object 332 with controller 200. Figure 19 shows an example in which the user operates buttons 202_1 to 202_4 of controller 200, causing character object 332 to bend and stretch in the vertical direction.

[0204] As shown in FIG. 19, when a movement operation is performed in which the user's finger touches two or more buttons 202 in sequence, the main unit 100 executes game processing (first processing) based on the order in which the buttons 202 are touched in the movement operation.

[0205] More specifically, the user touches button 202_1 and then touches button 202_3. At this time, while the user's finger is moving from button 202_1 to button 202_3, the user's finger approaches buttons 202_2 and 202_4. That is, touch sensors 204_2 and 204_4 arranged on buttons 202_2 and 202_4, respectively, output signals indicating that the user's finger is approaching (signals indicating a lower degree of approach compared to when the user's finger is in contact).

[0206] Based on the touch signal from the touch sensor 204_2 and / or the touch sensor 204_4, the main device 100 expresses an intermediate state between a state in which the character object 332 is stretched to the uppermost position and a state in which the character object 332 is contracted to the lowermost position.

[0207] In this way, when the movement operation involves the user's finger touching sequentially from button 202_1 to button 202_4, which is different from buttons 202_2 and 202_4 adjacent to button 202_1, the main unit 100 may execute game processing based on the user's finger touching buttons 202_2 and 202_4 of the four buttons 202_1 to 202_4.

[0208] As a process for representing this intermediate state, the main device 100 may execute game processing based on the user's finger touching both button 202_2 and button 202_4, or may execute game processing based on the user's finger touching one of button 202_2 and button 202_4.

[0209] In this case, the same game processing may be executed both when the user's finger touches both button 202_2 and button 202_4 and when the user's finger touches only one of button 202_2 and button 202_4.

[0210] By adopting such a configuration, even if the user does not touch the button 202 unintentionally, it is possible to realize image display that is in line with the user's original intention.

[0211] Note that Figure 19 illustrates an example of a user's movement between button 202_1 (touch sensor 204_1) and button 202_3 (touch sensor 204_3), but in the case of a user's movement between button 202_ (touch sensor 204_2) and button 202_4 (touch sensor 204_4), an intermediate state can be similarly expressed based on the touch signals from touch sensor 204_1 and touch sensor 204_3.

[0212] [F. Other User Experience Examples] Next, other user experience examples according to this embodiment will be described.

[0213] (f1: Character Input Process) Fig. 20 is a schematic diagram showing an example of character input process in the system according to the present embodiment. In the example of character input process shown in Fig. 20, the moving direction of a touch on button 202 can be detected.

[0214] 20 , for example, when the user touches button 202_1, character candidate object 340 is displayed. Character candidate object 340 includes four characters ("B," "C," "D," and "E" in the example shown in FIG. 20 ) associated with four directions in which button 202_1 is traced. Furthermore, "A" is positioned at the center of character candidate object 340.

[0215] Although not shown, by touching another button 202, character candidate objects 340 including other characters (for example, "F", "G", "H", "I", and "J") are displayed.

[0216] That is, each of the buttons 202 is assigned one letter (e.g., "A", "F", ...), and by touching any of the buttons 202, a character candidate object 340 is displayed that includes the letter (e.g., "A") assigned to the touched button 202 and multiple characters related to that letter (e.g., "B", "C", "D", "E").

[0217] When the user traces button 202 in any direction, a character corresponding to that direction is selected from a plurality of related characters included in character candidate object 340. FIG. 20 shows an example in which the character "C" is selected. On the other hand, when button 202 is pressed while character candidate object 340 is displayed, the character "A" located at the center of character candidate object 340 is selected. Note that when selecting a character located at the center of character candidate object 340, a movement operation in which the user's finger touches two or more buttons 202 in sequence is not necessarily required; the character is selected by a combination of pressing and touching buttons 202.

[0218] In this way, efficient character input can be performed using controller 200. (f2: Operation Guide Function) Fig. 21 is a schematic diagram showing an example of an operation guide function in a system according to the present embodiment. Referring to Fig. 21, when any process (e.g., game process) is being executed, touching any button 202 may cause notification object 350 to be displayed, which explains the function assigned to the touched button 202, etc.

[0219] By displaying such a notification object 350, the user can easily understand the functions assigned to each button 202 of the controller 200.

[0220] The operation guide function may have the following configurations (A) to (F). It may have only one of the following configurations, or may have multiple configurations. It may also have additional configurations.

[0221] (A) When the touched button 202 is pressed, the display of the notification object 350 corresponding to the button 202 may be terminated. This is because it is considered that the user pressed the button 202 after understanding the function assigned to the button 202.

[0222] (B) When the button 202 is touched and the state in which the button 202 is not pressed continues for a predetermined time or more, the notification object 350 may be displayed.

[0223] When the user operates the controller 200, the user may unintentionally touch the button 202. In such a case, the user may feel annoyed if the notification object 350 is displayed. Furthermore, when the user presses the button 202, the button 202 is inevitably touched. However, if the user presses the button 202 after understanding the function of the button 202, the user may feel annoyed if the notification object 350 is displayed. For this reason, a predetermined time may be set before the notification object 350 is displayed.

[0224] The predetermined time may be set to a time longer than the time generally required to press the button 202. For example, it may be 0.5 seconds or more. It may also be 1 second or more.

[0225] (C) When the notification object 350 continues to be displayed for a predetermined time or longer while the button 202 is touched but not pressed, the display of the notification object 350 may be terminated. This is because the user has already viewed the notification object 350 and further display may not be necessary. In this case, the predetermined time for which the notification object 350 is displayed may be, for example, two seconds or longer.

[0226] (D) When the first button 202 is touched and a second button 202 different from the first button 202 is also touched, the notification objects 350 for explaining the functions assigned to the first button 202 and the second button 202 may not be displayed. This is because when two buttons 202 are touched, it is unclear which button 202 the user wants to know the function of, and there is a possibility that the wrong button 202 has been touched. In another example, all of the notification objects 350 corresponding to both (multiple) touched buttons 202 may be displayed.

[0227] (E) When the first button 202 is touched and a second button 202 different from the first button 202 is pressed, the notification object 350 for explaining at least the function assigned to the first button 202 may not be displayed.

[0228] Examples of such cases include a case where the second button 202 is pressed while the first button 202 is being touched, and a case where the first button 202 is touched while the second button 202 is being pressed. In either case, there is a possibility that one button 202 is unintentionally touched when the other button 202 is pressed, and in such a case, the user may find it annoying to display the notification object 350 for one button 202.

[0229] As a more detailed example, if the second button 202 is pressed while the first button 202 is being touched and before a predetermined time required for displaying the notification object 350 has elapsed, the notification object 350 corresponding to the first button 202 may not be displayed even after the predetermined time has elapsed. Note that, for example, if the touch on the first button 202 continues for a certain period of time even after the pressing of the second button 202 has ended, the notification object 350 may be displayed.

[0230] As another detailed example, if the first button 202 is touched while the second button 202 is being pressed, the notification object 350 corresponding to the first button 202 may not be displayed even if a predetermined time that would normally be required to display the notification object 350 has elapsed. Similarly, if the first button 202 continues to be touched even after the pressing of the second button 202 has ended, the notification object 350 may be displayed.

[0231] (F) The likelihood of displaying the notification object 350 may change depending on the game state. For example, if a new function or a function different from the previous function is assigned to a certain button 202 as the game progresses, the notification object 350 may be made more likely to be displayed by touching the button 202 than before.

[0232] Alternatively, for example, when the game is played for the first time or if a predetermined time has passed since the last time the game was played, the notification object 350 may be made more likely to be displayed when the game is ended or paused and then resumed.

[0233] Alternatively, if a notification object 350 corresponding to a certain button 202 has already been displayed or if the button 202 has been pressed, the notification object 350 corresponding to the button 202 may be made less likely to be displayed thereafter.

[0234] Note that "easy to display" may include, for example, a short touch time on the button 202 required for the notification object 350 to be displayed. Also, "easy to display" may refer to the notification object 350 not being displayed, and "easy to display" may refer to the notification object 350 being displayed.

[0235] (f3: Combination with Operation of Controller 200) If the controller 200 is equipped with a gyro sensor, the user can perform an input operation by tilting the controller 200. Therefore, an operation on the controller 200 may be combined with touching the button 202.

[0236] 22 is a schematic diagram showing an example of a combination of operations and touch operations on controller 200 in a system according to the present embodiment. As shown in FIG. 22(A), a predetermined process is executed when the user sequentially touches different buttons 202. As shown in FIG. 22(B), a different process may be executed even when the user performs a touch similar to that shown in FIG. 22(A) by tilting controller 200. In this way, even when a movement operation is performed in which the user's finger sequentially touches two or more buttons 202, the content of the control executed may be changed depending on the state (e.g., tilt angle) or change in posture (e.g., acceleration) of controller 200.

[0237] By applying the user operations shown in FIGS. 22(A) and 22(B) to the same character object, it is possible to provide a variety of operability, thereby improving usability.

[0238] (f4: Combination with Operation of Direction Indicator 208) An operation on the direction indicator 208 of the controller 200 and a touch on the button 202 may be combined.

[0239] Fig. 23 is a schematic diagram showing an example of a combination of an operation and a touch operation on direction indicating unit 208 of controller 200 in the system according to the present embodiment. Fig. 23 shows an example of game processing in which a game object 360 indicating a target is aimed at a balloon game object 362.

[0240] The game object 360 can be adjusted by both operating the direction indicator 208 and touching the button 202. However, the effect on the game object 360 differs depending on the operation.

[0241] For example, the game object 360 moves more in response to an operation on the direction indicator 208, and moves more in response to a touch (movement operation) on the button 202. This allows the user to roughly adjust the aim by operating the direction indicator 208 and to finely adjust the aim by touching the button 202.

[0242] Note that pressing button 202, which is used for fine adjustment, may have a specific effect on the aiming direction. Conversely, the game object 360 may move more slightly in response to an operation on the direction indicator 208, and may move more significantly in response to a touch on button 202. In this case, operating the direction indicator 208 may have a specific effect on the aiming direction. Note that pressing button 202 may have a specific effect on the aiming direction.

[0243] As another example of operation, the direction in which the aim is pointed (line of sight direction) may be adjusted in response to an operation on the direction indicator 208, and the focal position of the aim (depth direction) may be adjusted by touching the button 202.

[0244] In this way, by applying an operation on the direction indicator 208 of the controller 200 and a touch operation on the button 202 to the same game object but in different operation modes, usability can be improved.

[0245] (f5: Cyclic Touch Operation) In the above-described cursor movement process, the user sequentially touches the multiple buttons 202 in a clockwise or counterclockwise direction to move the cursor 312. This type of cyclical touch operation may be employed in game processing in which force or power is accumulated by continuously performing the cyclical touch operation. The user progresses through the game by continuously performing the cyclical touch operation until the desired force or power is accumulated.

[0246] The factor of how long to continue the cyclical touch operation can be applied to any game process.

[0247] (f6: E-Book) As with the above-described cursor movement process, the user may sequentially touch a plurality of buttons 202 in a clockwise or counterclockwise direction to move forward or backward in a page of an e-book.

[0248] Furthermore, if the direction in which the user traces the buttons 202 can be detected, operations such as zooming in / out and changing the display range may be performed depending on the direction in which any of the buttons 202 is traced.

[0249] (f7: Rhythm Game) In the system according to the present embodiment, it is possible to detect pressing of button 202 and touching of button 202. Furthermore, it is also possible to detect gesture inputs such as up / down, left / right, diagonal, and rotation based on the order of touches on button 202. Furthermore, it is also possible to detect the direction in which the user traces button 202.

[0250] By using these many input operations, it is possible to realize, for example, a rhythm game with improved usability.

[0251] (f8: Scrolling Process) In the above-described cursor movement process, an example was shown in which the cursor moves while the display of the items on the screen is maintained, but this may also be applied to a scrolling process that changes the range displayed on the screen. In this case, all or part of the items displayed on the screen may be moved in response to a user operation.

[0252] For example, in the above-mentioned FIG. 7, nine items, Item 1 to Item 9, are displayed, but the scrolling process displays the following items, for example, Item 10 and subsequent items.

[0253] In addition, the cursor 312 may also move in conjunction with the scrolling process, or may be placed at a predetermined position (for example, the first item displayed) on the screen at each point in time.

[0254] [G. Modifications] The above examples show configurations in which processing is executed in response to an operation (movement operation) in which a user sequentially touches (approaches or contacts) two or more buttons 202 with their finger. In these configuration examples, main device 100 may not execute processing in response to the user's operation when the user's finger only touches (approaches or contacts) one button 202 alone, or may execute a predetermined processing when a specific condition is met, even when the user's finger only touches one button 202.

[0255] The specific conditions may be, for example, when one button 202 is touched off and the adjacent button 202 is not touched within a predetermined time, or when a button 202 that is not adjacent is touched even within the predetermined time.

[0256] [H. Advantages] According to the present embodiment, it is possible to detect the approach or contact of a user's finger with a plurality of buttons 202, and therefore system 1 can obtain information on user operations such as approaching or contacting buttons 202, in addition to information on user operations such as pressing buttons 202.

[0257] Furthermore, because multiple independent buttons 202 that can be pressed are employed, the user can specify the function assigned to pressing each button 202, and can distinguish between the buttons 202 by the feel of their fingertip, and can then give the intended instruction to the system 1 by approaching or touching the desired button 202. This makes it possible to realize a controller 200 with improved usability and processing that follows operations given to the controller 200.

[0258] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims.

[0259] 1 System, 100 Main unit, 102 Processor, 104 Memory, 106 Display, 108 Speaker, 110 Wireless communication module, 112 Wired communication module, 120 Storage, 122 System program, 124 Application program, 126 Application data, 200, 200L, 200R Controller, 202, 202_1 to 202_4 Button, 204, 204_1 to 204_4 Touch sensor, 206 Button operation unit, 208 Direction indication unit, 210 Movable contact, 212 Fixed contact, 214 Housing, 216 Board, 218 Key rubber, 230 Press determination unit, 232 Touch detection unit, 234 Output processing unit, 300 External display, 302 Dock, 310 Item group, 312 Cursor, 330, 332 Character object, 340 Character candidate object, 350 notification object, 360, 362 game objects, MP1, MP2 movement processing.

Claims

1. A system comprising a user-operated controller and one or more processors, The controller: Multiple independent buttons that can be pressed; A first sensor capable of detecting pressing of the button; a second sensor capable of detecting an approach or contact of a user's finger to the plurality of buttons; the processor, when a movement operation is performed in which the user's finger approaches or touches two or more of the buttons in sequence, executes a first process based on the order of the buttons approached or touched in the movement operation, and, after execution of the first process, executes a second process based on pressing of any one of the plurality of buttons; The processor executes the second process regardless of which of the plurality of buttons is pressed.

2. The system according to claim 1 , wherein the processor executes the second process based on the fact that a last button in the movement operation is pressed with the user's finger approaching or in contact with the last button.

3. The system according to claim 1 or 2, wherein the processor executes the second process based on the button being pressed before a predetermined time has elapsed after the user's finger is released from the button that was last approached or touched in the movement operation.

4. A program executed on a computer, the program causing the computer to: receiving a user's operation on a controller including a plurality of independent buttons that can be pressed, the controller being operated by the user; when detecting a movement operation in which the user's finger approaches or touches two or more of the buttons in sequence, executing a first process based on the order of the buttons that are approached or touched in the movement operation; and executing a second process based on a pressing of any one of the plurality of buttons after the first process is executed. The second process is executed when any one of the plurality of buttons is pressed.

5. An information processing method executed in a system having a controller operated by a user, the controller including a plurality of independent buttons that can be pressed, comprising: receiving a user operation on the controller; when detecting a movement operation in which the user's finger approaches or touches two or more of the buttons in sequence, executing a first process based on the order of the buttons that are approached or touched in the movement operation; and executing a second process based on pressing of any one of the plurality of buttons after the first process is executed. The information processing method, wherein the second process is executed regardless of which button among the plurality of buttons is pressed.

6. A system comprising a controller operated by a user and one or more processors, the controller includes a first button, a second button, and a third button that are depressible independently; A first sensor capable of detecting pressing of each of the buttons; a second sensor capable of detecting the approach or contact of a user's finger to each of the buttons; The processor executes a first process based on the user's finger moving from a state of approaching or touching the first button to approaching or touching both the second button and the third button.

7. The controller further includes a depressible independent fourth button; the first sensor is capable of detecting pressing of the fourth button; The second sensor is capable of detecting an approach or contact of a user's finger to the fourth button; the first button, the second button, the fourth button, and the third button are arranged in a circular manner in this order; 7. The system of claim 6, wherein in a movement operation in which the user's finger moves from approaching or touching the first button to approaching or touching the fourth button, the processor executes the first process based on the user's finger approaching or touching both the second button and the third button.

8. A program executed by a computer, the program causing the computer to: receiving a user's operation on a controller including a first button, a second button, and a third button that are independently depressible and operated by the user; and executing a first process based on detection that the user's finger has moved from a state in which the finger was in proximity to or in contact with the first button to a state in which the finger is in proximity to or in contact with both the second button and the third button.

9. An information processing method executed in a system having a controller including a first button, a second button, and a third button that are operable by a user and are independent and depressible, comprising: receiving a user operation on the controller; and executing a first process based on detecting that the user's finger has moved from a state of approaching or touching the first button to approaching or touching both the second button and the third button.

10. A system comprising a controller operated by a user and one or more processors, The controller: first to fourth buttons that are arranged in a ring and can be pressed independently by one finger of a user holding the controller; A first sensor capable of detecting pressing of each of the buttons; a second sensor capable of detecting the approach or contact of a user's finger to each of the buttons; The processor executes a first process when the user's finger moves from any one of the first to fourth buttons to adjacent buttons in one direction in sequence, and executes a second process when the user's finger moves to adjacent buttons in the other direction in sequence.

11. The first process includes a process of moving a displayed cursor in a first direction, The system of claim 10 , wherein the second operation includes moving a displayed cursor in a second direction.

12. A system as described in claim 10 or 11, wherein the controller further includes a direction indication unit that accepts directional instructions from a user.

13. A program executed on a computer, the program causing the computer to: a step of receiving an operation by a user of a controller operated by a user, the controller including first to fourth buttons arranged in a ring and operable by one finger of the user holding the controller, the first to fourth buttons being pressable independently, and the first to fourth buttons being arranged in a ring and operable by one finger of the user holding the controller; executing a first process when detecting a movement operation in which the user's finger approaches or touches one of the first to fourth buttons in order from one button to an adjacent button in one direction; and executing a second process when detecting a movement operation in which the user's finger approaches or contacts one of the first to fourth buttons in sequence to an adjacent button in the other direction.

14. An information processing method executed in a system including a controller operated by a user, the controller including first to fourth independent buttons arranged in a ring and operable with one finger of a user holding the controller, the information processing method comprising: receiving a user operation on the controller; executing a first process when detecting a movement operation of the user's finger from any one of the first to fourth buttons to approach or contact an adjacent button in one direction in sequence; and executing a second process when detecting a movement operation in which the user's finger approaches or contacts one of the first to fourth buttons in sequence to an adjacent button in the other direction.