Information processing method, information processing system, and information processing program

Abstract

To provide a novel control method using an input device having a mouse sensor. [Solution] Based on the output of the mouse sensor of the input device, a first state is determined in which the opening of the mouse sensor is closed. Based on the output of the inertial sensor of the input device, a second state is determined in which the bottom of the input device is facing downwards, and a third state is determined in which the input device is stationary around an axis parallel to the bottom. When at least the first to third states are determined, the displayed virtual object is controlled based on the output of the mouse sensor.

Description

Technical Field

[0001] This disclosure relates to information processing such as games.

Background Art

[0002] Conventionally, games using a mouse have been known (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There has been a demand for a new control method using an input device having a mouse sensor.

[0005] Therefore, an object of the present invention is to provide a new control method or the like using an input device having a mouse sensor.

Means for Solving the Problems

[0006] To achieve the above object, for example, the following configuration examples can be cited.

[0007] One configuration example is an information processing method using an input device having an inertial sensor, an optical mouse sensor, and a housing having a bottom portion where a light guide path for guiding light to the mouse sensor opens. Based on the output of the mouse sensor, a first state in which the opening is blocked is determined. Based on the output of the inertial sensor, a second state in which the bottom is facing downward is determined. Based on the output of the inertial sensor, a third state in which the input device is stationary around at least a first axis among the axes parallel to the bottom is determined. When at least the first state, the second state, and the third state are determined, the virtual object being displayed is controlled based on the output of the mouse sensor.

[0008] According to the above configuration example, it is possible to accurately estimate whether the input device is being used by the user like a mouse, and to control virtual objects in accordance with the user's intentions.

[0009] In another configuration example, the input device has an upper part opposite to the bottom, and a front part that connects the bottom and the upper part at the front and a rear part that connects them at the rear, with respect to the front-to-back direction which is parallel to the bottom and is longitudinal, and a first button is provided on the front side of the upper part or at least one of the front parts, and the third state may be a state in which the device is stationary at least around a first axis that is parallel to the bottom and extends in the left-to-right direction perpendicular to the front-to-back direction.

[0010] When an input device is used as a mouse, it may be less likely to tilt forward or backward. According to the above configuration example, by determining that the device is stationary at least around the first axis, it is possible to accurately estimate that it is intended to be used as a mouse.

[0011] As another example of configuration, the third state may be a state in which the first parameter, which indicates the degree of rotation around the first axis, is less than or equal to the first threshold.

[0012] According to the above configuration example, a certain degree of rotation around the first axis of the input device can be permitted, so for example, the input device can be used as a mouse on a mounting surface that has a curved surface.

[0013] As another example of configuration, the third state may further be a state in which the second parameter, which indicates the degree of rotation around the second axis extending in the front-back direction, is less than or equal to the second threshold, which is greater than the first threshold.

[0014] According to the above configuration example, by considering the left-right tilt of the input device, it is possible to accurately determine whether the input device is being used as a mouse. Furthermore, since left-right tilting is more likely to occur than forward-backward tilting when the input device is being used as a mouse, setting the threshold for determining left-right tilting to be relatively higher than the threshold for determining forward-backward tilting can suppress misdetermination.

[0015] As another example of configuration, the third state may be a state in which the first parameter is less than or equal to the first threshold, regardless of the value of the second parameter, which indicates the degree of rotation around the second axis extending in the front-rear direction.

[0016] When using an input device as a mouse, lateral tilting is likely to occur. According to the above configuration example, lateral tilting is tolerated, while forward and backward tilting is detected, thereby suppressing misdetection.

[0017] As another example of configuration, the input device may be longer in the vertical direction (vertical to the front-to-back and left-to-right directions) than in the left-to-right direction.

[0018] According to the above configuration example, the input device is easy for the user to hold when using it as a mouse, and is also easy to operate, for example, on the lap.

[0019] As another example of a configuration, the second state may be a state in which the third axis perpendicular to the bottom is tilted at an angle of 45 degrees or less with respect to the direction of gravity.

[0020] According to the above configuration example, a certain degree of inclination relative to the downward direction can be tolerated.

[0021] As another configuration example, based on the determination of at least the first, second, and third states, the system may switch from a first information processing state in which virtual objects are not displayed to a second information processing state in which virtual objects are displayed.

[0022] According to the above configuration example, since the virtual object is displayed when the virtual object can be operated, it is easy for the user to understand that the virtual object can be operated, that is, the state has transitioned to an operable state.

[0023] As another configuration example, based on the output of the mouse sensor, the input device determines a fourth state in which it is moving on the placement surface. When, in addition to the first state, the second state, and the third state being determined, the fourth state is determined, the displayed virtual object may be controlled based on the output of the mouse sensor.

[0024] As another configuration example, the input device has an upper part opposite to the bottom part, and a front part connecting the bottom part and the upper part in the front and a rear part connecting them in the rear in the front-rear direction that is parallel to the bottom part and is the longitudinal direction. The upper part has a first button on at least one of the front part side or the front part. When at least the first state, the second state, and the third state are determined, a predetermined process may be executed by operating the first button, and when such determination is not made, the predetermined process may not be executed by such operation.

[0025] According to the above configuration example, the input device can be preferably operated according to the operation state of the input device.

[0026] As another configuration example, the input device has a second button, and when at least one of the first state, the second state, and the third state is not determined, a predetermined process may be executed by operating the second button.

[0027] According to the above configuration example, the input device can be preferably operated according to the operation state of the input device.

[0028] As another configuration example, the third state may include a state in which the input device is stationary for a first period around a first axis.

[0029] According to the above configuration example, misjudgment of the input device can be suppressed.

[0030] In another configuration example, the input device may have an upper part opposite to the bottom, and based on the output of an inertial sensor, it may determine a fifth state in which an acceleration greater than a predetermined value is generated in the direction from the bottom to the top, and when at least the first, second, and fifth states are determined, it may manipulate the displayed virtual object based on the output of a mouse sensor.

[0031] According to the above configuration example, by determining the occurrence of upward acceleration in the input device, it is determined that the input device is placed on the mounting surface, thus enabling a quick transition of the operating state in accordance with the user's intentions.

[0032] As another example of configuration, a sixth state may be determined based on the output of the inertial sensor, where the input device is stationary around the first axis for a second period shorter than the first period. If the sixth state is not determined while controlling the displayed virtual object based on the output of the mouse sensor, control of the displayed virtual object based on the mouse sensor output may be stopped.

[0033] According to the above configuration example, the system can quickly detect when the input device is lifted from the mounting surface, enabling a rapid transition of the operating state in accordance with the user's intentions.

[0034] As another example of a configuration, the input device may have an upper part opposite to the bottom, and when in a first information processing state where no virtual object is displayed, it may switch to a third information processing state where no virtual object is displayed based on the determination of at least the first, second, and third states; a fourth state where at least the input device is moving on the mounting surface based on the output of the mouse sensor; a sixth state where the input device is stationary around the first axis for a second period less than the first period based on the output of the inertial sensor; and when in the third information processing state, it may switch to a second information processing state where no virtual object is displayed based on the determination of at least the first, second, fourth, and sixth states.

[0035] According to the above configuration example, the presence of a third information processing state allows for the construction of a more flexible control system.

[0036] As another example of configuration, the input device may perform at least operation system processing and application processing, and in the operation system processing, it may store information regarding the determination result of the first state and information regarding the determination result of at least the first state, second state, and third state being determined, in a manner that can be used in application processing.

[0037] According to the above configuration example, the application can be controlled not only by the determination results of the first, second, and third states, but also by information regarding the determination result of the first state.

[0038] As another example of configuration, the input device may perform at least operation system processing, and the operation system processing may at least determine whether the input device is set to portrait or landscape orientation, and in either case, determine the first state, second state, and third state.

[0039] According to the above configuration example, the application can be given more flexibility in its processing. [Effects of the Invention]

[0040] According to this embodiment, it is possible to provide a new control method, etc., using an input device having a mouse sensor. [Brief explanation of the drawing]

[0041] [Figure 1] This diagram shows an example of the main unit 2 with the right controller 3 and left controller 4 attached. [Figure 2] A six-view drawing showing an example of the right controller 3. [Figure 3] A six-view drawing showing an example of the left controller 4. [Figure 4] Block diagram showing an example of the internal configuration of the main unit 2. [Figure 5]Block diagram showing an example of the internal configuration of the main unit 2, the right controller 3, and the left controller 4. [Figure 6] This diagram shows an example of how the right controller 3 is held and operated with the right hand. [Figure 7] This diagram shows an example of how the right controller 3 is held and operated with the right hand. [Figure 8] This diagram shows an example of how the right controller 3 is held and operated with both hands. [Figure 9] A diagram illustrating controller state transitions. [Figure 10] Functional block diagram to explain the functions of the main unit 2. [Figure 11] Diagram showing an example of screen display. [Figure 12] This diagram shows examples of various data stored in DRAM69. [Figure 13] An example of an information processing flowchart [Figure 14] An example of an information processing flowchart [Figure 15] An example of an information processing flowchart [Figure 16] An example of an information processing flowchart [Modes for carrying out the invention]

[0042] One embodiment will be described below.

[0043] [Example of hardware configuration for an information processing system]

[0044] The following describes a game system, which is an example of the information processing system of this embodiment. An example of the game system 1 in this embodiment includes an information processing device (sometimes referred to as the "main unit") 2, a left controller 4, and a right controller 3. In this embodiment, the left controller 4 and the right controller 3 are detachable from the main unit 2.

[0045] Figure 1 shows an example of the main unit 2 with the left controller 4 and right controller 3 attached. As shown in Figure 1, the left controller 4 and right controller 3 are attached to the main unit 2 and integrated together. The main unit 2 is a device that performs various processes (e.g., game processing) in the game system 1. The main unit 2 is equipped with a display 72. The left controller 4 and right controller 3 are input devices equipped with operation parts for user input. In the following, the left controller 4 and right controller 3 may be collectively referred to as "controllers".

[0046] The main unit 2 includes a display 72. The display 72 displays images generated by the main unit 2. The display 72 is, for example, a liquid crystal display (LCD). The screen of the display 72 is equipped with a touch panel. The touch panel is, for example, a multi-touch input type (e.g., capacitive type).

[0047] Figure 2 is a hexagonal schematic diagram showing an example of the right controller 3. As shown in Figure 2, the right controller 3 is a vertically elongated plate shape, equipped with a housing 11, and has a front, rear, top, bottom, right, and left section. In the right controller 3, the rear is located opposite the front, the bottom is located opposite the top, and the left is located opposite the right section. The distance between the front and rear is greater than the distance between the top and bottom. The distance between the top and bottom is greater than the distance between the right and left sections. In other embodiments, the relative magnitudes of these distances may be different. In this embodiment, the direction connecting the bottom and top may be called the up-down direction, the direction perpendicular to the up-down direction and connecting the front and rear may be called the front-back direction, and the direction perpendicular to the up-down direction and connecting the right and left sections may be called the left-right direction. In Figure 2, the x, y, and z axes are shown with respect to a front view where the left section faces forward, indicating the coordinate system of the right controller 3. In this coordinate system, the direction from left to right is the positive z-axis direction. Also, the direction perpendicular to the z-axis and from the bottom to the top is the positive x-axis direction, and the direction perpendicular to the z-axis and x-axis and from the rear to the front is the positive y-axis direction. Note that when the bottom faces the direction of gravity, the negative x-axis direction and the direction of gravity coincide. In this embodiment, the front and bottom parts do not need to be perfectly flat and may have irregularities or slopes. For example, the bottom includes the convex part 25 described later. The directions that each part faces and the directions that connect each part are approximate directions.

[0048] The right controller 3 has a protrusion 25 that fits into a recess (not shown) of the main unit 2 when it is mounted on the main unit 2. As shown in Figure 2, the protrusion 25 is a convex shape that protrudes in the negative x-axis direction, with a width shorter in the left-right direction than the left-right direction of the right controller 3 and a width shorter in the front-back direction than the front-back direction. In this embodiment, the protrusion 25 is part of the bottom.

[0049] As will be described later, the right controller 3 can also be held in a vertical orientation when detached from the main unit 2. When held in a vertical orientation, the right controller 3 is shaped and sized to be held with one hand, especially the right hand. The right controller 3 can also be held in a horizontal orientation, and in this case, it may be held with both hands.

[0050] The right controller 3 is equipped with an analog stick (sometimes simply called a "stick") 22 on its left side, which is an example of a directional input unit. The stick 22 can be used as a directional input unit that can input directions. The user can input directions according to the direction of tilt by tilting the stick 22 in any direction, and the magnitude of the input can be according to the angle of tilt. The user can also input buttons by pressing down on the stick 22. The directional input unit may be, for example, a directional pad or a slide pad.

[0051] The right controller 3 has a set of four buttons on its left side: A button 12, B button 13, X button 14, Y button 15, a + (plus) button 16, and a home button 17. The right controller 3 has an R button 20 and a ZR button 21 extending across its front and top. The R button 20 and ZR button 21 may be located only on the front of the right controller 3, or only on the top. The right controller 3 has buttons 18 and 19 on the top surface 25a of the protrusion 25.

[0052] The right controller 3 is provided with an opening 23 for a mouse sensor on the top surface 25a of the protrusion 25. The opening 23 for the mouse sensor is an opening in the light guide path that guides light to the mouse sensor 24 located inside the right controller 3. The mouse sensor 24 is an optical mouse sensor and may include a light-emitting part and a light-receiving part. The light detected by the light-receiving part may be visible light or light of an invisible wavelength. The mouse sensor 24 may have at least a light-receiving part and may not have a light-emitting part. The mouse sensor 24 acquires data that allows for the calculation of the movement of the right controller 3 on the mounting surface, with the top surface 25a of the protrusion 25 of the bottom facing the mounting surface. In this way, the right controller 3 can also be used as a mouse. In this embodiment, when the right controller 3 is placed on the mounting surface with the top surface 25a facing the mounting surface, the direction in which the bottom extends is parallel to the direction in which the mounting surface extends.

[0053] Furthermore, in this embodiment, the right controller 3 is provided with a terminal 26 on the protrusion 25 for the right controller 3 to communicate with the main unit 2 via wired connection. As an example, the terminal 26 is provided on the inner circumferential surface of a recess provided on the top surface 25a of the protrusion 25.

[0054] Figure 3 is a hexagonal schematic diagram showing an example of the left controller 4. The same configuration as the right controller 3 will not be explained. The left controller 4 has a stick 42, a set of four buttons (right direction button 32, down direction button 33, up direction button 34, left direction button 35), a capture button 37, and a minus button 36 on its right side. Buttons 32-35 may be a single directional pad. Note that in the right controller 3, the stick 22 is located behind buttons 12-15, whereas in the left controller 4, the stick 42 is located in front of buttons 32-35. In Figure 3, the coordinate system of the left controller 4 is shown by illustrating the x, y, and z axes relative to a front view with the right side facing forward. In this coordinate system, the direction from the right side to the left side is the positive z-axis direction. The direction perpendicular to the z-axis and from the bottom to the top is the positive x-axis direction, and the direction perpendicular to the z and x axes and from the rear to the front is the positive y-axis direction. Note that when the bottom is facing the direction of gravity, the negative x-axis direction and the direction of gravity coincide.

[0055] The left controller 4 has a protrusion 45 that fits into a recess (not shown) of the main unit 2 when it is attached to the main unit 2. The protrusion 45, like the right controller 3, has buttons 38 and 39, an opening 43 for the mouse sensor, a mouse sensor 44, and a terminal 46.

[0056] When the left controller 4 is detached from the main unit 2, it can be held in either a vertical or horizontal orientation, similar to the right controller 3.

[0057] Figure 4 is a block diagram showing an example of the internal configuration of the main unit 2. The main unit 2 includes a processor 63. The processor 63 is an information processing unit that performs various information processing tasks performed in the main unit 2. The processor 63 may consist of, for example, multiple processors or cores, typically multiple CPUs (Central Processing Units) or cores, or it may consist of a SoC (System-on-a-chip) that includes multiple functions such as CPU function and GPU (Graphics Processing Unit) function. The processor 63 performs various information processing tasks by executing information processing programs (for example, game programs, etc.) stored in a storage unit (specifically, an internal storage medium such as a flash memory 68, or an external storage medium installed in a slot 51, etc.). In this embodiment, "processor" may include at least a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc.

[0058] The main unit 2 includes a flash memory 68 and a DRAM (Dynamic Random Access Memory) 69 as examples of internal storage media. The flash memory 68 is a memory mainly used to store various types of data stored in the main unit 2. The DRAM 69 is a memory mainly used to temporarily store various types of data used in information processing. The processor 63 reads and writes data to and from storage media such as the flash memory 68 and the DRAM 69 as appropriate and performs various information processing.

[0059] Furthermore, the main unit 2 has various configurations as shown in Figure 4. These are briefly described below. The recording medium slot interface (sometimes referred to as "slot I / F") 52 reads and writes data to and from the storage medium (for example, a dedicated memory card) installed in the recording medium slot 51, according to instructions from the processor 63. The second slot I / F 54 reads and writes data to and from the storage medium installed in the second slot 53, according to instructions from the processor 63.

[0060] The network communication unit 66 communicates with external devices via the network (for example, internet communication using wireless communication). The controller communication unit 67 communicates wirelessly with the left controller 4 and / or the right controller 3 (for example, communication in accordance with the Bluetooth® standard).

[0061] The left terminal 50 is a terminal for wired communication between the processor 63 and the left controller 4. The right terminal 65 is a terminal for wired communication between the processor 63 and the right controller 3. The lower terminal 64 is a terminal for communication with other devices (e.g., a stationary monitor, etc.) via the cradle when the lower terminal 64 is mounted on the cradle.

[0062] The touch panel controller 70 generates data indicating, for example, the location where a touch input occurred, based on signals from the touch panel 71 located on the display surface of the display 72, and outputs this data to the processor 63. The display 72 displays images generated by the processor 63 and / or images acquired from an external source.

[0063] The codec circuit 74 controls the input and output of audio data to the speaker 73 and the audio input / output terminal 75.

[0064] The power control unit 61 controls the power supply from the battery 62 to each part of the main unit 2 (i.e., each part that receives power from the battery 62) based on commands from the processor 63, and also starts or stops the power supply in response to the pressing of the power button 60.

[0065] The volume button 59 is used to control the volume output from the speaker 73, etc. The cooling fan 58 is a fan that cools the inside of the main unit 2.

[0066] The main unit 2 is equipped with various sensors, including a magnetic force sensor 55, an ambient light sensor 56, a temperature sensor 57, an acceleration sensor 76, and an angular velocity sensor 77. The processor 63 can perform various processes based on information from these sensors.

[0067] Figure 5 is a block diagram showing an example of the internal configuration of the main unit 2, the left controller 4, and the right controller 3. Note that the details of the internal configuration of the main unit 2 are shown in Figure 4 and are therefore omitted in Figure 5.

[0068] The left controller 4 includes a communication control unit 80 that communicates with the main unit 2. As shown in Figure 5, the communication control unit 80 is connected to each component, including terminals 88. When the left controller 4 is attached to the main unit 2, the communication control unit 80 communicates with the main unit 2 via wired communication through terminals 88. When the left controller 4 is detached from the main unit 2, the communication control unit 80 communicates with the main unit 2 wirelessly (specifically, communication in accordance with the Bluetooth® standard).

[0069] The left controller 4 includes a memory 81, such as flash memory. The communication control unit 80 is composed of a processor, such as a microcontroller (also called a microcomputer), and performs various processes by executing firmware stored in the memory 81.

[0070] The left controller 4 is equipped with buttons 82 (specifically, buttons 32-34, etc.) and a stick 42. Each button 82 and stick 42 outputs information about the operation performed on it to the communication control unit 80.

[0071] The left controller 4 is equipped with an inertial sensor. Specifically, the left controller 4 is equipped with an acceleration sensor 83 and an angular velocity sensor 84. The acceleration sensor 83 detects the magnitude of acceleration along a predetermined three-axis direction (for example, the x, y, and z axes shown in Figure 3). Note that the acceleration sensor 83 may also detect acceleration in one axis direction or two axis directions. The angular velocity sensor 84 detects angular velocity around the predetermined three axes. Note that the angular velocity sensor 84 may also detect angular velocity around one axis direction or two axis directions. The acceleration sensor 83 and the angular velocity sensor 84 are each connected to the communication control unit 80. The detection results of the acceleration sensor 83 and the angular velocity sensor 84 are repeatedly output to the communication control unit 80 at appropriate timings. In this embodiment, the inertial sensor includes an acceleration sensor and an angular velocity sensor. Note that the inertial sensor may be either an acceleration sensor or an angular velocity sensor, or it may be another sensor.

[0072] The left controller 4 is equipped with a mouse sensor 44. The mouse sensor 44 acquires data for calculating the movement of the left controller 4 placed on the mounting surface. The data acquired by the mouse sensor 44 is repeatedly output to the communication control unit 80 at appropriate intervals.

[0073] The communication control unit 80 acquires information about inputs (specifically, information about the operation of buttons and sticks, and detection results from sensors) from each input unit (specifically, each button 82, stick 42, and each sensor 83, 84, and 44). The communication control unit 80 transmits operation data, which includes the acquired information or information that has been processed in a predetermined manner, to the main unit 2. The operation data is transmitted repeatedly at a rate of once every predetermined time.

[0074] When the above operation data is transmitted to the main unit 2, the main unit 2 can obtain the input made to the left controller 4. That is, the main unit 2 can determine the operation of each button 82 and stick 42 based on the operation data. In addition, the main unit 2 can calculate information about the movement and / or posture of the left controller 4 based on the operation data (specifically, the detection results of the acceleration sensor 83 and / or angular velocity sensor 84). Furthermore, the main unit 2 can calculate information about mouse operations made to the left controller 4 based on the operation data (specifically, the detection results of the mouse sensor 44).

[0075] The left controller 4 includes an amplifier 85 and a vibrator 86. The amplifier 85 amplifies the control signal received from the communication control unit 80 and generates a drive signal. The vibrator 86 vibrates in response to the drive signal generated by the amplifier 85, causing the left controller 4 to vibrate.

[0076] The left controller 4 includes a power supply unit 87. The power supply unit 87 has a battery and a power control circuit. The power control circuit is connected to the battery and supplies power to each part of the left controller 4 (specifically, each part that receives power from the battery).

[0077] As shown in Figure 5, the right controller 3 includes a communication control unit 91, which is composed of a processor and other components and communicates with the main unit 2. The right controller 3 also includes a memory 94 connected to the communication control unit 91. The communication control unit 91 is connected to each component, including the terminal 92. The communication control unit 91 and the memory 94 have the same functions as the communication control unit 80 and memory 81 of the left controller 4. Therefore, the communication control unit 91 can communicate with the main unit 2 both by wired communication via the terminal 92 and by wireless communication without using the terminal 92, and controls the communication that the right controller 3 makes to the main unit 2.

[0078] The right controller 3 is equipped with input ports in the same way as the left controller 4. Specifically, it is equipped with buttons 95 (A button 12, B button 13, X button 14, Y button 15, etc.), a stick 22, inertial sensors (accelerometer 96 and angular velocity sensor 97), and a mouse sensor 24. Each of these input ports has the same function and operates in the same way as the input ports of the left controller 4.

[0079] The right controller 3 comprises an amplifier 98, a vibrator 99, and a power supply unit 100. The amplifier 98, vibrator 99, and power supply unit 100 have the same functions and operate in the same manner as the amplifier 85, vibrator 86, and power supply unit 87 of the left controller 4, respectively.

[0080] The right controller 3 includes a processing unit 90 and an NFC antenna 93. The processing unit 90 controls the NFC antenna 93 in response to commands from the main unit 2 via the communication control unit 91. The NFC antenna 93 performs short-range wireless communication based on the NFC (Near Field Communication) standard.

[0081] [Regarding the gripping configuration of the controller] Figure 6 is a schematic diagram showing an example of a state in which a user grasps the right controller 3 with their right hand and places it on a mounting surface to use it as a mouse. As shown in Figure 6, from the user's perspective, the front of the right controller 3 faces forward and the left side faces left. The user's right palm covers the upper side of the right controller 3. The user's right thumb is positioned on the left side of the right controller 3. The user's right thumb is placed on, for example, the A button 12. The user's right index finger is placed on, for example, the R button 20, and the user's right middle finger is placed on, for example, the ZR button 21. The user can operate the R button 20 and the ZR button 21 with their right index finger or middle finger. The user can operate each input unit located on the left side with their right thumb. Note that when the user uses the left controller 4 as a mouse with their left hand, they can grasp the left controller 4 with their left hand and use it on the mounting surface in the same manner. In this case, the right side of the left controller 4 faces to the right. The user may hold the right controller 3 with their left hand so that its front faces forward, or they may hold the left controller 4 with their right hand so that its front faces forward.

[0082] Figure 7 is a schematic diagram showing an example of a user operating the right controller 3 by holding it vertically with their right hand. As shown in Figure 7, when the right controller 3 is detached from the main unit 2, it can be used by holding it so that its longitudinal direction is in the vertical or front-to-back direction for the user. The same procedure can be used when the user uses the left controller 4, which is detached from the main unit 2, with their left hand.

[0083] Figure 8 is a schematic diagram showing an example of how a user operates the right controller 3 by holding it horizontally with both hands. As shown in Figure 8, when the right controller 3 is detached from the main unit 2, it is used so that its longitudinal direction is in the left-right direction for the user. The right controller 3 is held by one hand on one side in the longitudinal direction (one side in the vertical direction) and by the other hand on the other side. For example, the index fingers of both hands of the user are placed on the bottom side of the right controller 3. The same procedure can be used when the user uses the left controller 4, which is detached from the main unit 2, with both hands.

[0084] [Regarding controller states] Figure 9 is a diagram illustrating the states and state transitions of the controller. As shown in Figure 9, the controller states (sometimes simply referred to as "states") include the "normal state," the "mouse-ready state," and the "mouse state." The mouse-ready state is the state in which the controller is presumed to be ready to be operated as a mouse but is not yet being operated as a mouse. The mouse state is the state in which the controller is presumed to be being operated as a mouse. The normal state is the state in which the controller is neither in the mouse-ready state nor the mouse state, and is presumed to be an operation state as illustrated in Figures 7 and 8, for example.

[0085] When certain conditions are met in the Normal State, a first state transition occurs, moving to the Mouse Ready State. When certain conditions are met in the Mouse Ready State, a second state transition occurs, moving to the Mouse State. When certain conditions are met in the Mouse Ready State, a third state transition occurs, moving to the Normal State. When certain conditions are met in the Mouse State, a fourth state transition occurs, moving to the Normal State. These predetermined conditions will be described later. In addition, the mouse operation flag is turned OFF in the Normal State and Mouse Ready State, and ON in the Mouse State. The mouse operation flag is a flag that indicates that the controller is presumed to be operated as a mouse. The controller state and the mouse operation flag are controlled, for example, by the operating system (sometimes called "OS") run by processor 63.

[0086] [Details of the information processing in this embodiment] Next, the information processing of this embodiment will be described in detail with reference to Figures 10 to 16. The following explanation will use the case where the right controller 3 is used as an example. Note that the same considerations apply when the left controller 4 is used, so its explanation will be omitted.

[0087] [Examples of functional blocks in an information processing system] Figure 10 shows an example of a functional block of the information processing system 1. As shown in Figure 10, the main unit 2 has an OS execution unit 110 and an application (sometimes referred to as "AP") execution unit 113 as functional blocks. The processing of each part will be described below.

[0088] As shown in Figure 10, data detected by each sensor and input unit of the right controller 3 is transmitted from the right controller 3 to the OS execution unit 110. This data includes, for example, image clarity data, dy / dz data, inertial sensor value data, and button input data. Note that the data transmitted from the right controller 3 to the OS execution unit 110 may also include data other than those listed above.

[0089] Image clarity data is data indicating the clarity of the mouse sensor image. Image clarity data is data calculated by the mouse sensor 24. Image clarity data is calculated, for example, based on the degree of brightness of the mouse sensor image and / or the degree of the number of feature points in the mouse sensor image. Note that the degree of brightness of the mouse sensor image or the degree of the number of feature points in the mouse sensor image may be used directly as image clarity data. In addition, image clarity data may be calculated based on other factors. If the clarity indicated by the image clarity data is above a predetermined level, it can be estimated that the opening 23 of the mouse sensor 24 is blocked by the mounting surface, etc. If there is data that allows it to be estimated that the opening 23 of the mouse sensor 24 is blocked, other data may be used instead of data indicating the clarity of the mouse sensor image.

[0090] The dy / dz data is the output data of the mouse sensor 24, and when the opening 23 of the mouse sensor 24 is blocked by the mounting surface, it is data that shows the distance traveled per frame time in the y-axis and z-axis directions (i.e., the yz plane; see Figure 2) of the coordinate system of the right controller 3 relative to the mounting surface, etc.

[0091] The inertial sensor data is output from the inertial sensor of the right controller 3, and is data that allows for the calculation of, for example, the acceleration in the xyz axis direction (see Figure 2) and / or angular velocity around the xyz axis of the coordinate system of the right controller 3. Using the inertial sensor data, for example, the attitude and movement of the right controller 3 can be calculated.

[0092] The button input data represents the operations performed on each of the buttons 95 and the stick 22 of the right controller 3.

[0093] The OS execution unit 110 uses dy / dz data to calculate mouse coordinate data that shows the latest mouse coordinates on the mouse coordinate plane that define the movement of the right controller 3 as a mouse. The OS execution unit 110 uses dy / dz data and image clarity data to perform a blockage determination to determine whether or not the opening 23 of the right controller 3 is blocked, and generates blockage data that shows the result of the blockage determination. The blockage determination process corresponds to the determination processes in steps S103, S104, S110, S111, S120, and S121, which will be described later using Figures 13 to 15. Based on the blockage data and inertial sensor value data, the OS execution unit 110 sets the state of the right controller 3 (see Figure 9). The set state is stored as controller state data. Based on the state of the right controller 3, the OS execution unit 110 sets the mouse operation flag related to the right controller 3 to ON or OFF (see Figure 9).

[0094] The OS execution unit 110 sets the grip style of the right controller 3 in response to user input and generates grip style data. The user can set the grip style of the right controller 3, for example, by opening a predetermined settings menu and performing a predetermined operation. There are two grip styles: the vertical grip style exemplified in Figure 7 and the horizontal grip style exemplified in Figure 8. For example, the user may set the grip style according to their preferred operation or the game they are playing, or it may be set by the game. The AP execution unit 113 can set the function of each button 95 of the right controller 3 to a predetermined function according to the grip style. The OS execution unit 110 may also automatically determine the grip style based on the input data.

[0095] In response to a request from the AP execution unit 113, data is transmitted from the OS execution unit 110 to the AP execution unit 113. The transmitted data may include, for example, mouse coordinate data, mouse operation flag data indicating the ON / OFF state of the mouse operation flag, button input data, and grip style data. Note that only some of this data may be input to the AP execution unit 113, or other data may be input. Furthermore, the data may be transmitted continuously, or only when there is an update.

[0096] The AP execution unit 113 uses mouse operation flag data to perform a mouse mode determination to determine whether or not to run the currently executing application in mouse mode. For example, the AP execution unit 113 turns the mouse mode flag ON when the mouse operation flag data indicates ON, and turns the mouse mode flag OFF when the mouse operation flag data indicates OFF.

[0097] Figure 11 is a diagram illustrating an example of control based on the ON / OFF status of a mouse operation flag. As an example, it shows a scene where the AP execution unit 113 executes a home menu application and the home menu is displayed. As shown in Figure 11, the home menu displays button images a to f, each corresponding to a different function.

[0098] When the mouse mode flag is ON, a virtual object called a mouse cursor (sometimes simply referred to as a "cursor") 250 is displayed, which can be moved and displayed by operating the right controller 3 as a mouse (sometimes referred to as "mouse operation"), as shown in Figure 11(1). The displayed cursor 250 moves in response to the mouse operation of the right controller 3. In this embodiment, when the mouse operation flag is ON, the R button 20 and ZR button 21 of the right controller 3 function as if they were the left-click and right-click buttons of a mouse, respectively. For example, in the home menu display shown in Figure 11(1), when the tip of the cursor 250 is positioned on the button image d and the R button 20 is operated, the function associated with the button image d is executed. The functions assigned to the buttons are not particularly limited; for example, the same function may be assigned to both the R button 20 and the ZR button 21. Additionally, the select function may be assigned to buttons 14 and 12, or only to button 12.

[0099] When the mouse mode flag is OFF, the cursor 250 is not displayed, as shown in Figure 11(2), but a frame cursor (sometimes simply called a "cursor") 251 pointing to one of the button images is displayed. The frame cursor 251 moves discontinuously to a position surrounding other button images in response to operations on the direction input unit. In this embodiment, the hidden mouse cursor 250 is not controlled to move according to the output data of the mouse sensor 24, but it may be controlled to move. Also in this embodiment, when the mouse mode flag is OFF, the R button 20 and ZR button 21 of the right controller 3 do not function as the left click button and right click button of the mouse. On the other hand, for example, when button 12 is pressed, the function associated with the button image surrounded by the frame cursor 251 is executed. In this case, the R button 20 and ZR button 21 of the right controller 3 may execute other functions when operated. Thus, the AP execution unit 113 may control the button functions of the right controller 3 depending on whether or not it is in mouse mode. For example, when the AP execution unit 113 is set to mouse mode, it applies a mouse mode button operation interpretation to the button operations of the right controller 3, and when it is set not to mouse mode, it applies a non-mouse mode operation interpretation to the button operations of the right controller 3. The stick 22 may also be included as a target for the button operation interpretation. For example, when not in mouse mode, the cursor moves in response to the operation of the stick 22, and when in mouse mode, the display screen or selected items in the list may scroll in response to the operation of rotating the stick 22, similar to the operation of a mouse wheel.

[0100] The AP execution unit 113 may control the button functions of the right controller 3 based on the grip style data when the user is not in mouse mode. Specifically, if the AP execution unit 113 determines that the grip style is set to vertical grip style, it may apply a button operation interpretation for vertical grip style to the button operations of the right controller 3, and if it determines that the grip style is set to horizontal grip style, it may apply a button operation interpretation for horizontal grip style to the button operations of the right controller 3. In addition, regardless of whether the grip style is set to vertical or horizontal grip style, if the AP execution unit 113 determines that the user is in mouse mode, it may apply a button operation interpretation for mouse mode to the button operations of the right controller 3.

[0101] The OS execution unit 110 may also input controller state data to the AP execution unit 113. This allows the AP execution unit 113 to perform control according to the state, for example, to perform predetermined display processing in the mouse ready state. The OS execution unit 110 may also input occlusion data indicating the result of the occlusion determination described above to the AP execution unit 113. This allows, for example, the user's operation of blocking or opening the opening 23 with their finger to be treated as a button operation. As an example, when the AP execution unit 113 is running a game application, the AP execution unit 113 can cause a virtual object to perform a predetermined action such as an attack in response to the user's operation of holding the right controller 3 horizontally (see Figure 8) and blocking the opening 23 with their finger. Data input from the OS execution unit 110 to the AP execution unit 113 may be performed, for example, by storing the data in the DRAM 69 so that the AP execution unit 113 can acquire the data.

[0102] [About the data used] Next, we will explain the various types of data stored in the DRAM 69. Figure 12 shows an example of data stored in the DRAM 69 of the main unit 2. As shown in Figure 12, the DRAM 69 is provided with at least a program storage area 301 and a data storage area 302. The program storage area 301 stores the OS program 401 and AP programs 402, etc. The AP program 402 is an AP program that controls the home menu, settings menu, etc. Although not shown in the figure, AP programs such as game programs are also stored in the program storage area 301.

[0103] The data storage area 302 contains OS control data 405 used for control by the OS program 401 and AP control data 420 used for control by the AP program 402. In this embodiment, the OS control data 405 and the AP control data 420 are stored separately, but it is not necessary to store them separately. If this is not the case, it is not necessary to store the same data separately.

[0104] The OS control data 405 includes mouse operation flag 406, controller state data 407, image clarity data 408, dy / dz data 409, mouse coordinate data 410, inertial sensor value data 412, button input data 413, and grip style data 414. These data are as described above.

[0105] AP control data 420 includes object data 421, image data 422, virtual camera control data 423, mouse coordinate data 424, mouse operation flag data 425, mouse mode flag 426, button input data 427, and grip style data 428.

[0106] Object data 421 is data for objects placed in the virtual space, such as player characters, opponent characters, the ground, menus, and cursors.

[0107] Image data 422 consists of image data such as animation images, backgrounds, and virtual effects.

[0108] The virtual camera control data 423 is data for controlling a virtual camera that is placed in a virtual space and takes pictures of that virtual space.

[0109] The mouse coordinate data 424, button input data 427, and grip style data 428 are identical to the mouse coordinate data 410, button input data 413, and grip style data 414 described above, respectively.

[0110] The mouse operation flag data 425 indicates whether the mouse operation flag 406 described above is ON or OFF.

[0111] The mouse mode flag 426 is a flag that is turned ON when the mouse operation flag 406 described above is ON, and turned OFF when it is OFF, based on the mouse operation flag data 425.

[0112] In addition, various types of data used for drawing and other processes are stored in DRAM69 as needed.

[0113] [Details of the process according to this embodiment] Next, the processing according to this embodiment will be described with reference to the flowchart. Figures 13 to 16 are examples of flowcharts showing the processing according to this embodiment. The processing in the flowcharts of Figures 13 to 15 is executed by the OS program 401, and the processing in the flowchart of Figure 16 is executed by the AP program 402. In the following, we will mainly describe the processing that is characteristic of this embodiment, and other descriptions such as drawing processing will be omitted. Furthermore, the following processing is executed at predetermined intervals (for example, every processing frame, every 1 / 30 second).

[0114] When processing by the OS program 401 begins, in step S100 of Figure 13, the processor 63 calculates the mouse coordinates.

[0115] In step S101, the processor 63 refers to the controller state data 407 and determines whether the current state of the right controller 3 is a normal state. If the determination is YES, the process moves to step S103; otherwise, the process moves to step S102.

[0116] In step S102, the processor 63 refers to the controller state data 407 and determines whether the current state of the right controller 3 is the mouse-ready state. If the determination is YES, the process moves to step S110 in Figure 14; if NO, the process moves to step S120 in Figure 15.

[0117] In step S103, the processor 63 uses the two most recent dy / dz data 409 from the right controller 3 to determine whether consecutive dy / dz data have been acquired. In other words, the processor 63 determines whether data indicating movement in the y-axis direction or data indicating movement in the z-axis direction has been acquired by the mouse sensor 24 of the right controller 3 in the most recent consecutive processing frames (see x, y, and z axes in Figure 2). If the determination is YES, the process moves to step S105; otherwise, the process moves to step S104.

[0118] In step S104, the processor 63 determines, based on the image clarity data 408, whether the mouse sensor 24 of the right controller 3 has acquired a stable image. For example, the processor 63 determines that the mouse sensor 24 has acquired a stable image, i.e., an image with a clarity level of a predetermined or higher, if a predetermined value indicating the clarity level of the image acquired by the mouse sensor 24 is greater than or equal to a predetermined threshold. If this determination is YES, the process moves to step S105; otherwise, the process returns to step S100. Here, the condition described above, "the predetermined value is greater than or equal to a predetermined threshold," is a condition that is satisfied, for example, when the brightness level of the image acquired by the mouse sensor 24 (i.e., the amount of light received by the mouse sensor 24) is greater than or equal to a predetermined level (or within a predetermined range), and / or when the number of feature points in the image acquired by the mouse sensor 24 (e.g., the degree of image coarseness) is greater than or equal to a predetermined level.

[0119] Here, when the aperture 23 of the mouse sensor 24 is blocked and the right controller 3 is moved, the dy / dz data is expected to output a value indicating the movement of the controller. By using dy / dz data from multiple frames rather than just one frame, it is possible to accurately estimate that the aperture 23 is blocked. Also, even if the right controller 3 is not moved, if the mouse sensor 24 is acquiring a stable image, it can be inferred that the aperture 23 is blocked. Therefore, if the determination in step S103 is YES, or if the determination in step S104 is YES, it can be said that the aperture 23 of the mouse sensor 24 is in a state where it can be inferred that it is blocked.

[0120] In step S105, the processor 63 determines whether the average acceleration in the positive x-axis direction of the coordinate system of the right controller 3 (see Figure 2) is near -1.0, that is, near the acceleration due to gravity. Specifically, the processor 63 determines, based on the inertial sensor data 412 of the right controller 3, whether the average acceleration in the positive x-axis direction over a predetermined time (for example, the last 5 processing frames) is greater than, for example, -1.15 and less than -0.85. This determination allows the processor 63 to estimate that the bottom of the right controller 3 is facing downwards. This "bottom facing downwards" means that the bottom is facing the direction of gravity, and is not limited to the bottom facing directly downwards (i.e., the direction of gravity), but also includes the state in which it is tilted to a predetermined degree relative to the direction directly downwards. Alternatively, in step S105, the processor 63 may also determine whether the x-axis of the coordinate system of the right controller 3 is tilted within 45 degrees relative to the direction of gravity. If the result is YES, the process moves to step S106; if it is NO, the process returns to step S100.

[0121] In step S106, the processor 63 determines whether or not it has detected an impact in the positive x-axis direction of the coordinate system of the right controller 3. Specifically, the processor 63 determines, for example, whether or not it has detected an acceleration of a predetermined value or higher (for example, 2.0G or higher) in the positive x-axis direction based on the inertial sensor value data 412 of the right controller 3. This determination allows the processor 63 to estimate that the right controller 3 has been placed on a mounting surface or the like. If the determination is YES, the process moves to step S108; if NO, the process moves to step S107.

[0122] In step S107, the processor 63 determines whether the rotational speed of the right controller 3 around the z-axis of the coordinate system is stable for a predetermined period. Specifically, based on the inertial sensor data 412 of the right controller 3, the processor 63 determines whether the average rotational speed around the z-axis over a predetermined period (for example, the last 5 processing frames) (sometimes referred to as the "first parameter") is below a first threshold. This determination allows the processor 63 to determine whether the right controller 3 is stationary around the z-axis. This "stationary around the z-axis" means not rotating around the z-axis, and includes not only a state of being completely stationary around the z-axis, but also a state of slight rotation. If the determination is YES, the process moves to step S108; if NO, the process returns to step S100.

[0123] In other embodiments, in step S107, the processor 63 may determine whether both the rotational speed around the z axis and the rotational acceleration around the y axis of the coordinate system of the right controller 3 are stable for a predetermined period. Specifically, the processor 63 may, based on the inertial sensor data 412 of the right controller 3, determine, for example, whether the average rotational speed around the z axis over a predetermined period (e.g., the last 5 processing frames) is less than or equal to a first threshold, and whether the average rotational speed around the y axis over the same predetermined period (sometimes referred to as the "second parameter") is less than or equal to a second threshold (the second threshold is greater than the first threshold). This determination allows the processor 63 to determine whether the right controller 3 is stationary around the z axis and the y axis. This "stationary state around the z axis and the y axis" means a state in which there is no rotation around the z axis and the y axis, and is not limited to a state in which there is complete stationary around the z axis and the y axis, but also includes a state in which there is some rotation around at least one of the axes. In step S107, the processor 63 may determine whether only the rotational acceleration of the right controller 3 around the y-axis of the coordinate system is stable for a predetermined period of time. Compared to when the right controller 3 is placed on a wide, stable surface such as a desk, if the opening 23 of the mouse sensor 24 is covered by a finger or the like, rotation around an axis parallel to the bottom is more likely to occur. By monitoring rotation around an axis parallel to the bottom, it is possible to estimate that the controller is placed on a stable surface. When the controller is being operated like a mouse, it is more likely to tip over in the left-right direction than in the front-back direction. In particular, the controller in this embodiment has a length in the left-right direction that is shorter than the length in the front-back direction, and a length in the up-down direction that is longer than the length in the left-right direction. Therefore, the controller in this embodiment is more likely to tip over in the left-right direction. Therefore, by prioritizing the z-axis among the axes parallel to the bottom, or by monitoring whether it is stationary only around the z-axis, it is possible to suppress the assumption that the controller has moved away from the mounting surface contrary to the user's intention when the user unintentionally rotates the controller around the y-axis.Furthermore, this could also have the effect of preventing the system from mistakenly assuming that the controller has moved away from a curved surface, such as the thigh, when the user operates the controller with the mouse on such a surface.

[0124] In step S108, the processor 63 sets the state of the right controller 3 to the mouse-ready state. Specifically, the processor 63 performs a first state transition (see Figure 9) and updates the state of the right controller 3, indicated by the controller state data 407, to the mouse-ready state. After that, the process returns to step S100.

[0125] Next, let's explain the flowchart in Figure 14. The processes in steps S110 to S112 in Figure 14 are the same as the processes in steps S103 to S105 in Figure 13.

[0126] In step S113, the processor 63 determines, based on the inertial sensor value data 412, whether or not it detected rotation of the right controller 3 around the z-axis of the coordinate system in the current processing frame. For example, the processor 63 determines whether or not it detected a rotational speed of the right controller 3 around the z-axis that is greater than or equal to a predetermined value in the most recent processing frame. Furthermore, since this determination is based on the duration of one processing frame, it is a period shorter than the predetermined period used in the determination in step S107 (for example, the most recent five processing frames). From this, it can be said that this determination determines whether or not the right controller 3 is stationary around the z-axis for each processing frame, i.e., in short cycles. The determination in step S113 allows for instantaneous determination of whether the right controller 3, which is placed on the mounting surface, is not on a stable mounting surface, for example, whether or not it has been lifted from the mounting surface. If this determination is YES, the process moves to step S117; if NO, the process moves to step S114.

[0127] In step S114, the processor 63 determines, based on the dy / dz data 409, whether or not it has detected movement of the right controller 3 by a predetermined distance (e.g., 3 mm) or more. In other words, the processor 63 determines whether or not the right controller 3 is moving on the mounting surface. If this determination is YES, the process moves to step S115; otherwise, the process returns to step S100 in Figure 13.

[0128] In step S115, the processor 63 sets the state of the right controller 3 to the mouse state. Specifically, the processor 63 performs a second state transition (see Figure 9) and updates the state of the right controller 3, indicated by the controller state data 407, to the mouse state. After that, the process moves to step S116.

[0129] In step S116, the processor 63 sets the mouse operation flag 406 of the right controller 3 to ON. Then, the process returns to step S100 in Figure 13.

[0130] In step S117, the processor 63 sets the state of the right controller 3 to the normal state. Specifically, the processor 63 performs a third state transition (see Figure 9) and updates the state of the right controller 3, indicated by the controller state data 407, to the normal state. After that, the process returns to step S100 in Figure 13.

[0131] Next, let's explain the flowchart in Figure 15. The processes in steps S120 to S123 in Figure 15 are the same as the processes in steps S110 to S113 in Figure 14.

[0132] In step S124, the processor 63 sets the state of the right controller 3 to the normal state. Specifically, the processor 63 performs a fourth state transition (see Figure 9) and updates the state of the right controller 3, indicated by the controller state data 407, to the normal state. After that, the process moves to step S125.

[0133] In step S125, the processor 63 sets the mouse operation flag 406 of the right controller 3 to OFF. Then, the process returns to step S100 in Figure 13.

[0134] Next, the flowchart in Figure 16 will be explained. When processing by the AP program 402 begins, in step S130, the processor 63 refers to the mouse operation flag data 425 and determines whether the mouse operation flag 406 is ON or not. If this determination is YES, the process moves to step S131; if it is NO, the process moves to step S132.

[0135] In step S131, the processor 63 sets the mouse mode flag 426 of the right controller 3 to ON. Then, the process returns to step S100 in Figure 13. An example of the process when the mouse mode flag 426 is ON is as described above.

[0136] In step S132, the processor 63 sets the mouse mode flag 426 of the right controller 3 to OFF. Then, the process returns to step S100 in Figure 13. An example of the process when the mouse mode flag 426 is OFF is as described above.

[0137] In other embodiments, control of the controller's button functions may be switched, for example, based on controller state data 407 or mouse operation flag 406, by control of the OS program 401. In other embodiments, the controller may generate controller state data and mouse operation flags by a communication control unit or the like (see Figure 5) and transmit this data to the main unit.

[0138] As described above, according to this embodiment, when movement is detected in the controller while the mouse sensor opening is closed and the bottom is facing downwards and not rotated (see Figures 13 and 14), the mouse operation flag is turned ON. Therefore, according to this embodiment, it is possible to accurately estimate that the controller is being operated as a mouse, and the user experience can be improved. Furthermore, according to this embodiment, the OS program can determine whether the controller is estimated to be used as a mouse, and the AP program can switch, for example, the control of the cursor 250 or the control of the controller's button functions according to this estimation.

[0139] [Differentiation] In the embodiment described above, two examples were given for determining whether or not the controller is stationary around the z-axis parallel to the bottom of the controller: the determination process in step S107 in Figure 13 and the determination process in step S113 in Figure 14 (the same process as the determination process in step S123 in Figure 15). However, for example, the processes executed in these steps may be the same process (i.e., one type), or there may be three or more types of processes.

[0140] Furthermore, in the embodiment described above, examples were given in which the determination process in step S107 of Figure 13 and the determination process in step S113 of Figure 14 (the same process as the determination process in step S123 of Figure 15) are performed by comparison with a predetermined value that serves as the basis for determination. However, in these determination processes, the determination may be performed by methods other than comparison with a predetermined value. Also, in these determination processes, the determination may be performed using rotational acceleration in addition to rotational speed, or using rotational acceleration alone. Furthermore, the determination may be performed using other parameters.

[0141] Furthermore, in the embodiment described above, the AP execution unit 113 may execute some or all of the processes explained using Figures 13 to 15, or the controller side, i.e., the controller's communication control unit and memory (see Figure 5), may execute them.

[0142] Furthermore, in the embodiment described above, if the condition in steps S120 and S121 remains NO for a predetermined time (for example, 0.5 seconds) in the flowchart of Figure 15, the control may proceed to step S124 to set the controller to the normal state. By controlling in this way, for example, when a user moves the controller on the mounting surface using a mouse, then lifts it slightly from the mounting surface, returns it to its original position, and moves it again, it is possible to prevent the controller from immediately returning to the normal state the moment it is lifted.

[0143] Furthermore, in the embodiment described above, when the mouse mode flag is OFF, a frame cursor 251 is displayed instead of the mouse cursor 250 (see Figures 11 and 16). However, the mouse cursor 250 may be displayed without the frame cursor 251, and the mouse cursor 250 may be moved by operation of the direction input unit. Also, when the mouse mode flag is ON, the mouse cursor 250 may be moved by either mouse operation or operation of the direction input unit.

[0144] Furthermore, in the embodiment described above, the cursor display and button function control were switched between two states depending on whether the mouse mode flag was ON or OFF, but they may be switched between three or more states. A mode that is neither of the above states may also be included. For example, the home menu application may be operable by touch operation on a touch panel provided on the display or controller. For example, items such as button images may be selectable by touch operation. In this case, for example, the mouse mode (see Figure 11(1)), the directional input mode (see Figure 11(2)), and the touch mode may be switched. Also, the cursor does not have to be displayed in touch mode. As an example, when the directional input mode is in and the frame cursor is displayed, if a touch is made on the touch panel, the system may switch to touch mode and the frame cursor may disappear. A cursor for touch operation may be displayed instead of the cursor for directional input mode. For example, even if the mouse operation flag data indicates ON, the system may switch to touch mode when a touch is detected. The mouse mode, directional input mode, and touch mode may be switched from one mode to another based on at least one of the following data: mouse operation flag data, directional input or button operation data, or touch operation data. In this case, the data used when switching from the first mode to the second mode may be the same as or different from the data used when switching from the second mode to the first mode, or when switching modes in at least one direction between the third mode and the first or second mode. As already mentioned, the conditions under which the mouse operation flag data indicates ON or OFF in this embodiment are just examples. These conditions may be different, and the mouse operation flag data may be any data used to estimate whether or not a mouse operation is being performed. Also, the data format does not have to be a flag.

[0145] Furthermore, in the embodiment described above, a configuration without a mouse-ready state is also possible. Alternatively, a configuration without any states is also possible.

[0146] Furthermore, the virtual object controlled by mouse operations is not limited to the mouse cursor 250; it may be any other object, such as a crosshair, bullets, or player character. Note that a single AP program may have multiple virtual objects that can be controlled by mouse operations, and the virtual object controlled by the mouse operations may be switched between.

[0147] Furthermore, in the above-described embodiment, mouse coordinate data may be used in the process where dy / dz data is used, and dz / dy data may be used in the process where mouse coordinate data is used. In addition, the various data in this embodiment are examples, and other data converted to other formats may be used as appropriate in each process.

[0148] Furthermore, the game system is an example of an information processing system, and the information processing system may be a system that does not run games. Also, the main unit may be a general-purpose personal computer. Also, the controller may be a general-purpose mouse.

[0149] At least a portion of the series of processes described above may be executed by the server-side device in an information processing system that includes a terminal-side device and a server-side device that can communicate via a network. The server may consist of multiple information processing devices, and the processing may be divided and executed by these multiple devices.

[0150] Although this embodiment and its variations have been described above, these descriptions are merely illustrative in every respect and are not intended to limit its scope. Furthermore, it goes without saying that various improvements and modifications can be made to this embodiment and its variations. [Explanation of Symbols]

[0151] 1. Information Processing System 2. Main unit 3, 4 Controllers 24, 44 Mouse Sensors 63 processors 68, 69, 81, 94 Memory 72 displays 76, 77, 83, 84, 96, 97 Inertial Sensors Buttons 82 and 95 250, 251 Cursors

Claims

1. An information processing method using an input device comprising an inertial sensor, an optical mouse sensor, and a housing having a bottom opening through which a light guide path for guiding light to the mouse sensor is opened, Based on the output of the mouse sensor, a first state in which the opening is closed is determined. Based on the output of the inertial sensor, a second state is determined in which the bottom is facing downwards. Based on the output of the inertial sensor, a third state is determined in which the input device is stationary around at least a first axis among the axes parallel to the bottom. An information processing method for controlling a displayed virtual object based on the output of the mouse sensor when at least the first state, the second state, and the third state are determined.

2. The aforementioned input device is The upper part opposite to the bottom, With respect to the front-to-back direction which is parallel to the bottom and is in the longitudinal direction, it has a front part that connects the bottom and the top at the front and a rear part that connects them at the rear, The upper part is provided with a first button on the front side or at least one of the front parts, The information processing method according to claim 1, wherein the third state is at least a state in which the device is stationary around the first axis which is parallel to the bottom and extends in a left-right direction perpendicular to the front-rear direction.

3. The information processing method according to claim 2, wherein the third state is a state in which a first parameter indicating the degree of rotation around the first axis is less than or equal to a first threshold.

4. The information processing method according to claim 3, wherein the third state is further a state in which a second parameter indicating the degree of rotation around the second axis extending in the front-rear direction is less than or equal to a second threshold greater than the first threshold.

5. The information processing method according to claim 3, wherein the third state is a state in which the first parameter is less than or equal to the first threshold, regardless of the value of the second parameter which indicates the degree of rotation around the second axis extending in the front-rear direction.

6. The information processing method according to claim 4 or 5, wherein the input device is longer in the vertical direction perpendicular to the front-to-back direction and the left-to-right direction than in the left-to-right direction.

7. The information processing method according to claim 1, wherein the second state is a state in which the third axis perpendicular to the bottom is inclined at an angle of 45 degrees or less with respect to the direction of gravity.

8. The information processing method according to claim 1, which, based on the determination of at least the first state, the second state, and the third state, switches from a first information processing state in which the virtual object is not displayed to a second information processing state in which the virtual object is displayed.

9. Based on the output of the mouse sensor, the fourth state in which the input device is moving on the mounting surface is determined. The information processing method according to claim 1, wherein when the fourth state is determined in addition to the determination of the first state, the second state, and the third state, the displayed virtual object is controlled based on the output of the mouse sensor.

10. The aforementioned input device is The upper part opposite to the bottom, With respect to the front-to-back direction which is parallel to the bottom and is in the longitudinal direction, it has a front part that connects the bottom and the top at the front and a rear part that connects them at the rear, The upper part is provided with a first button on the front side or at least one of the front parts, The information processing method according to claim 1, wherein when at least the first state, the second state, and the third state are determined, a predetermined process is executed by operating the first button, and when the determination is not made, the predetermined process is not executed by operating the button.

11. The aforementioned input device is It has a second button, If at least one of the first state, the second state, and the third state has not been determined, the predetermined process is executed by operating the second button. The information processing method according to claim 10.

12. The information processing method according to claim 1, wherein the third state includes a state in which the input device is stationary around the first axis for a first period of time.

13. The input device has an upper part opposite to the bottom, Based on the output of the inertial sensor, a fifth state is determined in which an acceleration greater than a predetermined value is generated in the direction from the bottom to the top. The information processing method according to claim 12, wherein when at least the first state, the second state, and the fifth state are determined, the displayed virtual object is operated based on the output of the mouse sensor.

14. Based on the output of the inertial sensor, a sixth state is determined in which the input device is stationary around the first axis for a second period shorter than the first period. The information processing method according to claim 12, wherein when the sixth state is not determined while the displayed virtual object is being controlled based on the output of the mouse sensor, the control of the displayed virtual object based on the output of the mouse sensor is stopped.

15. The input device has an upper part opposite to the bottom, When the virtual object is not displayed in the first information processing state, the system switches to a third information processing state in which the virtual object is not displayed, based on the determination of at least the first state, the second state, and the third state. Based on the output of the mouse sensor, a fourth state is determined in which at least the input device is moving on the mounting surface. Based on the output of the inertial sensor, a sixth state is determined in which the input device is stationary around the first axis for a second period shorter than the first period. When the third information processing state is in place, the system switches to a second information processing state in which the virtual object is displayed, based on the determination of at least the first state, the second state, the fourth state, and the sixth state. The information processing method according to claim 12.

16. The input device performs at least operation system processing and application processing. The information processing method according to claim 1, wherein in the operation system processing, information relating to the determination result of the first state and information relating to the determination result that at least the first state, the second state and the third state have been determined are stored in a manner that can be used in the application processing.

17. The input device performs at least the operation system processing, The information processing method according to claim 1, wherein the operation system processing determines at least whether the input device is set to vertical orientation or horizontal orientation, and determines the first state, the second state and the third state in either case.

18. An information processing system comprising an input device having an inertial sensor, an optical mouse sensor, and a housing having a bottom opening through which a light guide path for guiding light to the mouse sensor is opened, Based on the output of the mouse sensor, a first state in which the opening is closed is determined. Based on the output of the inertial sensor, a second state is determined in which the bottom is facing downwards. Based on the output of the inertial sensor, a third state is determined in which the input device is stationary around at least a first axis among the axes parallel to the bottom. An information processing system that controls a displayed virtual object based on the output of the mouse sensor when at least the first state, the second state, and the third state are determined.

19. A processor in an information processing system comprising an input device having an inertial sensor, an optical mouse sensor, and a housing having a bottom opening through which a light guide path for guiding light to the mouse sensor is provided. Based on the output of the mouse sensor, the first state in which the opening is closed is determined. Based on the output of the inertial sensor, a second state in which the bottom is facing downwards is determined. Based on the output of the inertial sensor, a third state is determined in which the input device is stationary around at least one axis parallel to the bottom. An information processing program that controls the displayed virtual object based on the output of the mouse sensor when at least the first state, the second state, and the third state are determined.

Images