Program, information processing system, and information processing method

Abstract

To provide an appropriate operation mode according to an operation mode of a controller.SOLUTION: A program causes an information processing system to acquire first input device data including inertial sensor data and mouse sensor data of a first input device, acquire second input device data including inertial sensor data and mouse sensor data of a second input device, execute processing according to a set operation mode on the basis of the first input device data and the second input device data, set a first operation mode on the basis of the orientation of the first input device satisfying a first condition, set the first operation mode on the basis of the orientation of the second input device satisfying a second condition, and set a second operation mode on the basis of the orientation of the first input device satisfying a third condition and the orientation of the second input device satisfying a fourth condition. Processing according to the first operation mode includes processing for operating an object on the basis of at least one of the first mouse data and the second mouse data.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present disclosure relates to processing data based on operations on an input device. [Background technology]

[0002] BACKGROUND ART Input devices such as game controllers have been known for some time (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-757 Summary of the Invention [Problem to be solved by the invention]

[0004] When the controller is operated in various manners with each hand, there is room for providing an appropriate operation mode according to the manner of operation. [Means for solving the problem]

[0005] For example, the following configuration example can be given.

[0006] A first configuration example is a program that causes a computer to acquire first input device data including first inertial data corresponding to an output of an inertial sensor of a first input device and first mouse data corresponding to an output of a mouse sensor of the first input device, acquire second input device data including second inertial data corresponding to an output of an inertial sensor of a second input device and second mouse data corresponding to an output of a mouse sensor of the second input device, execute processing corresponding to a set operation mode based on the first input device data and the second input device data, set the first operation mode based on an attitude of the first input device satisfying a first condition, set the first operation mode based on an attitude of the second input device satisfying a second condition, and set the second operation mode based on an attitude of the first input device satisfying a third condition and an attitude of the second input device satisfying a fourth condition, wherein the processing corresponding to the first operation mode includes processing for operating an object based on at least one of the first mouse data or the second mouse data.

[0007] As a second configuration example, in the first configuration example, the first condition may be a condition that is met when the mouse sensor opening of the first input device faces at least in the direction of gravity, and the second condition may be a condition that is met when the mouse sensor opening of the second input device faces at least in the direction of gravity.

[0008] As a third configuration example, in the first or second configuration example, the third condition may be a condition that is met when the mouse sensor opening of the first input device faces a direction at least perpendicular to the direction of gravity, and the fourth condition may be a condition that is met when the mouse sensor opening of the second input device faces a direction at least perpendicular to the direction of gravity.

[0009] As a fourth configuration example, in the first to third configuration examples, the first mode may be set based on the attitude of the first input device satisfying a first condition and satisfying a condition that is met when the mouse sensor opening of the first input device is blocked, and the first mode may be set based on the attitude of the second input device satisfying a second condition and satisfying a condition that is met when the mouse sensor opening of the second input device is blocked.

[0010] As a fifth configuration example, in the first to fourth configuration examples, regardless of whether the condition that is met when the mouse sensor opening of the first input device is blocked and the condition that is met when the mouse sensor opening of the second input device is blocked are met, the second operation mode may be set based on the attitude of the first input device satisfying the third condition and the attitude of the second input device satisfying the fourth condition.

[0011] As a sixth configuration example, in the first to fifth configuration examples, the processing according to the second operation mode may be processing in which the object is not operated based on the first mouse data and the second mouse data.

[0012] As a seventh configuration example, in the sixth configuration example, the processing according to the second operation mode may be processing in which no object is operated based on the first mouse data and the second mouse data.

[0013] As an eighth configuration example, in the first to seventh configuration examples, the object may be a cursor, and the first input device data may include first operation unit data corresponding to a user operation on an operation unit of the first input device, and the processing corresponding to the second operation mode may include processing for moving the cursor based on the first operation unit data.

[0014] As a ninth configuration example, in the eighth configuration example, the processing according to the first operation mode may include processing in which the cursor is moved continuously based on at least one of the first mouse data or the second mouse data, and the processing according to the second operation mode may include processing in which the cursor is moved discontinuously based on the first operation unit data.

[0015] As a tenth configuration example, in the ninth configuration example, the cursor displayed when the first operation mode is set may be the same as the cursor displayed when the second operation mode is set.

[0016] As an eleventh configuration example, in the tenth configuration example, when the first input device and the second input device are not operated, the cursor displayed when the first operation mode is set may be stopped, and the cursor displayed when the second operation mode is set may be moved.

[0017] As a twelfth configuration example, in the first to eleventh configuration examples, the processing according to the first operation mode may include processing in which the object is operated based on both the first mouse data and the second mouse data.

[0018] As a thirteenth configuration example, in the first to twelfth configuration examples, the computer may be caused to execute game processing, and the game processing may include a game portion in which both the first mouse data and the second mouse data are used, and a menu portion in which one of a plurality of operation modes including the first operation mode and the second operation mode is set.

[0019] Each of the configuration examples described above may be applied to an information processing method or an information processing system. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 shows an example of a state in which the right controller 3 and the left controller 4 are attached to the main unit 2. [Figure 2] Six-sided diagram showing an example of the right controller 3 [Figure 3] Six-sided diagram showing an example of the left controller 4 [Figure 4] A block diagram showing an example of the configuration of the main unit 2. [Figure 5] A block diagram showing an example of the configuration of the main unit 2, the right controller 3, and the left controller 4. [Figure 6] FIG. 10 is a diagram showing an example of a state in which a controller is held and operated with both hands. [Figure 7] A diagram showing an example of a state in which the controller is operated with both hands as a mouse. [Figure 8]FIG. 10 shows an example of a state in which the right controller 3 is operated as a mouse with the right hand, and the left controller 4 is held and operated with the left hand. [Figure 9] FIG. 10 shows an example of a state in which the right controller 3 is held horizontally with both hands and operated. [Figure 10] FIG. 10 is a diagram for explaining a first operation method; [Figure 11] FIG. 10 is a diagram for explaining a second operation method; [Figure 12] Diagram showing an example of various data [Figure 13] An example of a sequence DETAILED DESCRIPTION OF THE INVENTION

[0021] An embodiment will be described below.

[0022] [Example of hardware configuration of information processing system]

[0023] A game system, which is an example of an information processing system, will be described below. The game system 1 in this embodiment includes a main unit 2, a left controller 4, and a right controller 3. The main unit 2 is an example of an information processing device. The information processing device may be, for example, a personal computer, a tablet device, a smartphone, a wearable device, a server, etc. The left and right controllers 3 and 4 are an example of an input device. The input device does not have to be detachable from the main unit 2. The input device is also an example of an information processing device. The information processing device and the input device are examples of a computer. Note that the term "computer" does not necessarily refer to a single device, but can refer to a system in which multiple devices are connected by wire or wirelessly. Furthermore, each of the information processing device and input device can be an example of an information processing system.

[0024] The main unit 2 is a device that executes various processes (for example, game processes) in the game system 1. In this embodiment, the main unit 2 includes a display 72.

[0025] The left controller 4 and the right controller 3 are equipped with operation units and the like that allow the user to perform inputs. Note that hereinafter, the left controller 4 and the right controller 3 may be collectively referred to as the "controller." FIG. 2 is a six-sided schematic diagram showing an example of the right controller 3. As shown in FIG. 2, the right controller 3 is shaped like a vertically long plate, includes a housing 11, and has a front section, a rear section, a top section, a bottom section, a right section, and a left section. As will be described later, an opening for a mouse sensor (hereinafter also referred to as a mouse sensor opening) is provided in the bottom section. In the right controller 3, the rear section is located opposite the front section, the bottom section is located opposite the top section, and the left section is located opposite the right section. The distance between the front and rear sections is greater than the distance between the top and bottom sections. The distance between the top and bottom sections is greater than the distance between the right and left sections. Note that in other embodiments, the magnitude relationship between these distances may be other magnitude relationships. In this embodiment, the direction connecting the bottom and top is referred to as the up-down direction, the direction connecting the front and rear, which is perpendicular to the up-down direction, is referred to as the front-rear direction, and the direction connecting the right and left, which is perpendicular to the up-down direction, is referred to as the left-right direction. In FIG. 2, the coordinate system of the right controller 3 is shown with x, y, and z axes illustrated in a front view in which the left side faces forward. In this coordinate system, the direction from the left side to the right side is the positive z-axis. The direction from the bottom to the top, which is perpendicular to the z-axis, is the positive x-axis, and the direction from the rear to the front, which is perpendicular to the z-axis and x-axis, is the positive y-axis. When the bottom faces the direction of gravity, the negative x-axis and the direction of gravity coincide. In this embodiment, each part, such as the front and bottom, does not need to be completely flat and may have irregularities or slopes. For example, the bottom includes a convex portion 25, which will be described later. The directions of each part and the directions connecting each part are shown as rough directions.

[0026] The right controller 3 has a protrusion 25 that fits into a recess (not shown) of the main unit 2 when attached to the main unit 2. As shown in Fig. 2, the protrusion 25 has a convex shape that protrudes in the negative x-axis direction, with a width that is shorter than the left-right direction of the right controller 3 and a front-to-back width that is shorter than the front-to-back direction of the right controller 3. In this embodiment, the protrusion 25 is part of the bottom.

[0027] As will be described later, the right controller 3 can also be held in a portrait orientation when detached from the main unit 2. The right controller 3 has a shape and size that allows it to be held in one hand, particularly the right hand, when held in a portrait orientation. The right controller 3 can also be held in a landscape orientation, and when held in a landscape orientation, it may be held with both hands.

[0028] The right controller 3 has, on its left side, an analog stick (sometimes simply referred to as a "stick") 22, which is an example of a direction input unit. The stick 22 can be used as a direction input unit that can input directions. The user can tilt the stick 22 in any direction through 360 degrees to input a direction according to the tilt direction, and the input size can be determined according to the tilt angle. The user can also perform button input by pressing the stick 22. The stick may be slidable instead of tiltable. The direction input unit may be, for example, a cross key.

[0029] The right controller 3 has a set of four buttons on its left side: an A button 12, a B button 13, an X button 14, a Y button 15, a + (plus) button 16, and a home button 17. The right controller 3 has an R button 20 and a ZR button 21 on its front and top. The R button 20 and ZR button 21 may be provided only on the front or only on the top of the right controller 3. The right controller 3 has buttons 18 and 19 on the top surface 25a of the convex portion 25. The right controller 3 does not have an operation unit on its rear.

[0030] The right controller 3 has a mouse sensor opening 23 on the top surface 25a of the convex portion 25. The mouse sensor opening 23 is an opening in a light guide path that guides light to the mouse sensor 24 located inside the right controller 3. The mouse sensor 24 is an optical mouse sensor and includes at least a light receiving unit. The light detected by the light receiving unit may be visible light or light of an invisible wavelength. The mouse sensor 24 may also include a light emitting unit. Light from the light emitting unit may be emitted to the outside through the opening 23. The mouse sensor 24 acquires data that enables calculation of the movement of the right controller 3 on a mounting surface when the right controller 3 is placed with the top surface 25a of the convex portion 25 of the bottom facing the mounting surface. This allows the right controller 3 to be used as a mouse. In this embodiment, when the right controller 3 is placed on a mounting surface with the top surface 25a facing the mounting surface, the direction in which the bottom extends is parallel to the direction in which the mounting surface extends.

[0031] In this embodiment, the right controller 3 is provided with a terminal 26 on the protrusion 25, which enables the right controller 3 to perform wired communication with the main unit 2. As an example, the terminal 26 is provided on the inner circumferential surface of a recess provided in the top surface 25a of the protrusion 25.

[0032] FIG. 3 is a six-sided schematic diagram showing an example of the left controller 4. Description of components similar to those of the right controller 3 will be omitted. The left controller 4 is equipped with a stick 42, a set of four buttons on the right side: a right button 32, a down button 33, an up button 34, a left button 35, a capture button 37, and a - (minus) button 36. The buttons 32 to 35 may be a single cross button. The left controller 4 does not have an operation unit on the rear. Note that the stick 22 on the right controller 3 is located behind the buttons 12 to 15, whereas the stick 42 on the left controller 4 is located in front of the buttons 32 to 35. In FIG. 3, the coordinate system of the left controller 4 is shown with the x, y, and z axes illustrated in a front view in which the right side faces forward. In this coordinate system, the direction from the right side to the left side is the positive z-axis direction. The direction perpendicular to the z-axis, from the bottom to the top, is the positive x-axis direction, and the direction perpendicular to the z-axis and x-axis, from the back to the front, is the positive y-axis direction. When the bottom faces the direction of gravity, the negative x-axis direction and the direction of gravity coincide.

[0033] The left controller 4 has a protrusion 45 that fits into a recess (not shown) of the main unit 2 when attached to the main unit 2. Similar to the right controller 3, the protrusion 45 has buttons 38 and 39, a mouse sensor 44, and a terminal 46.

[0034] When the left controller 4 is detached from the main unit 2, it can be held in a portrait or landscape orientation, similar to the right controller 3.

[0035] FIG. 4 is a block diagram showing an example of the configuration of the main unit 2. The main unit 2 includes a processor 63. The processor 63 is an information processing unit that executes various types of information processing executed in the main unit 2. The processor 63 may be composed of, for example, multiple processors or cores, typically multiple central processing units (CPUs) or cores, or may be composed of a system-on-a-chip (SoC) that includes multiple functions such as a CPU function and a graphics processing unit (GPU) function. In this embodiment, the "processor" may include at least a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The processor 63 executes various types of information processing by executing a program (e.g., a program that runs a game application) stored in a storage unit (e.g., an internal storage medium such as flash memory 68, or an external storage medium attached to slot 51, etc.).

[0036] The main unit 2 includes a flash memory 68 and a dynamic random access memory (DRAM) 69 as examples of internal storage media. The flash memory 68 is a memory used primarily for storing various types of data saved in the main unit 2. The DRAM 69 is a memory used primarily for temporarily storing various types of data used in information processing. The processor 63 executes various types of information processing by appropriately reading and writing data from and to storage media such as the flash memory 68 and the DRAM 69. In this embodiment, the "memory" may include at least the flash memory and the DRAM, and may also include other storage media.

[0037] Furthermore, the main device 2 has various components as shown in Fig. 4. A brief explanation is given below. A recording medium slot interface (sometimes called a "slot I / F") 52 reads and writes data from a storage medium (e.g., a dedicated memory card) inserted into the recording medium slot 51 in response to instructions from the processor 63. A second slot I / F 54 reads and writes data from a storage medium inserted into the second slot 53 in response to instructions from the processor 63.

[0038] The network communication unit 66 communicates with external devices via a network (for example, internet communication using wireless communication). The controller communication unit 67 communicates wirelessly with the left controller 4 and / or right controller 3 (for example, communication in accordance with the Bluetooth (registered trademark) standard).

[0039] The left-side terminal 50 is a terminal for wired communication between the processor 63 and the left controller 4. The right-side terminal 65 is a terminal for wired communication between the processor 63 and the right controller 3. The lower terminal 64 is a terminal for communicating with other devices (such as a stationary monitor) via the cradle when the lower terminal 64 is attached to the cradle, for example.

[0040] The codec circuit 74 controls the input and output of audio data to the speaker 73 and the audio input / output terminal 75 .

[0041] The power control unit 61 controls the power supply from the battery 62 to each part of the main unit 2 (i.e., each part that receives power from the battery 62) based on instructions from the processor 63, and also starts or stops the power supply in response to pressing the power button 60.

[0042] The main body device 2 includes various sensors such as an acceleration sensor 76 and an angular velocity sensor 77. The processor 63 can execute various processes based on information from these sensors.

[0043] Figure 5 is a block diagram showing an example of the configuration of the main unit 2, left controller 4, and right controller 3. Note that details of the configuration of the main unit 2 are omitted in Figure 5 because they are shown in Figure 4.

[0044] The left controller 4 is equipped with a communication control unit 80 that communicates with the main unit 2. As shown in FIG. 5, the communication control unit 80 is connected to various components, including a terminal 88. When the left controller 4 is attached to the main unit 2, the communication control unit 80 performs wired communication with the main unit 2 via the terminal 88, and when the left controller 4 is detached from the main unit 2, the communication control unit 80 performs wireless communication with the main unit 2 (for example, communication in accordance with the Bluetooth (registered trademark) standard).

[0045] The left controller 4 includes a memory 81, such as a flash memory. The communication control unit 80 is configured with a processor, such as a microcomputer (also called a microprocessor), and executes firmware stored in the memory 81 to perform various processes.

[0046] The communication control unit 80 acquires information relating to operations performed on each of the buttons 82 and the stick 42 .

[0047] The left controller 4 is equipped with an inertial sensor. Specifically, the left controller 4 is equipped with an acceleration sensor 83 and an angular velocity sensor 84. The acceleration sensor 83 and the angular velocity sensor 84 are each connected to the communication control unit 80. The communication control unit 80 repeatedly acquires the detection results of the acceleration sensor 83 and the angular velocity sensor 84 at appropriate timing. Note that the inertial sensor may be either the acceleration sensor or the angular velocity sensor, or another sensor.

[0048] The left controller 4 is equipped with a mouse sensor 44. The mouse sensor 44 acquires data for calculating the movement of the left controller 4 placed on the placement surface. The communication control unit 80 repeatedly acquires the data acquired by the mouse sensor 44 at appropriate timing.

[0049] The communications control unit 80 transmits left controller data including the acquired information or information obtained by performing predetermined processing on the acquired information to the main unit 2. The left controller data may be transmitted repeatedly at predetermined times. For example, the communications control unit 80 may transmit left controller data in response to data requests received repeatedly from the main unit 2 at predetermined times.

[0050] The left controller 4 is equipped with a power supply unit 87. The power supply unit 87 has a battery and a power control circuit. The power control circuit is connected to the battery and supplies power to each part of the left controller 4 (specifically, each part that receives power from the battery).

[0051] As shown in FIG. 5 , the right controller 3 has a similar configuration to the left controller 4. For example, the right controller 3 has a communication control unit 91, which is composed of a processor and the like and communicates with the main unit 2. The right controller 3 has a memory 94 connected to the communication control unit 91. The communication control unit 91 is connected to each component, including a terminal 92. The communication control unit 91 and memory 94 have the same functions as the communication control unit 80 and memory 81 of the left controller 4. Therefore, the communication control unit 91 can communicate with the main unit 2 both via wired communication via the terminal 92 and via wireless communication not via the terminal 92, and controls the communication between the right controller 3 and the main unit 2. For example, the communication control unit 91 obtains information regarding operations performed on each button 95 and stick 22 and sends the information to the main unit 2.

[0052] The right controller 3 includes a processing unit 90 and an NFC antenna 93. The processing unit 90 controls the NFC antenna 93 in response to commands from the main unit 2 via a communication control unit 91. The NFC antenna 93 performs short-range wireless communication based on the NFC (Near Field Communication) standard.

[0053] [Controller operation] Next, the operation modes of the controllers will be described. In this embodiment, it is assumed that one user uses both the right controller 3 and the left controller 4.

[0054] FIG. 6 is a schematic diagram showing an example of a state in which a user holds the right controller 3 vertically in their right hand and the left controller 4 vertically in their left hand and operates them. As shown in FIG. 7, when the right controller 3 is detached from the main unit 2, it can be held and used so that the longitudinal direction of the right controller 3 is the up-down direction or the front-back direction for the user. The left controller 4 can be used in the same manner. In the following explanation, the operating mode shown in FIG. 6 is also referred to as a two-handed vertical holding mode.

[0055] FIG. 7 is a schematic diagram showing an example of a state in which a user places their right hand on the right controller 3 and their left hand on the left controller 4, placing them on a placement surface and using them as a mouse. As shown in FIG. 7, when viewed from the user, the front of the right controller 3 faces forward and the left part faces left. The palm of the user's right hand covers the upper side of the right controller 3. The user's right thumb is placed on the left side of the right controller 3. The user's right thumb is placed on, for example, the A button 12. The user's right index finger is placed on, for example, the R button 20, and the user's right middle finger is placed on, for example, the ZR button 21. The user can operate the R button 20 and the ZR button 21 with their right index finger or middle finger. The user can operate each input unit placed on the left part with their right thumb. Furthermore, when viewed from the user, the front of the left controller 4 faces forward and the right part faces right. The palm of the user's left hand covers the upper side of the left controller 4. The user's left thumb is placed on the right side of the left controller 4. The user's left thumb is placed on, for example, the left stick 42. Alternatively, the left thumb may be placed on the left direction button 35. The user's left index finger is placed on, for example, the L button 40, and the user's left middle finger is placed on, for example, the ZL button 41. The user can operate the L button 40 and the ZL button 41 with the left index finger or middle finger. The user can operate each input unit placed on the right side with the left thumb. In the following description, the operation mode shown in FIG. 7 may also be referred to as a two-handed mouse mode.

[0056] 8 is a schematic diagram showing an example of a state in which a user operates the right controller 3 as a mouse and operates the left controller 4 by holding it vertically in the left hand. In the following description, the operation mode shown in FIG. 8 may also be referred to as a mixed operation mode.

[0057] In addition to the above-described operation modes, the user can also operate the right controller 3 by holding it horizontally with both hands, as shown in FIG. 9. As shown in FIG. 9, when the right controller 3 is detached from the main unit 2, the longitudinal direction of the right controller 3 is oriented left-right for the user. The right controller 3 is held by one hand at one end in the longitudinal direction (one end in the front-to-back direction), and by the other hand at the other end. For example, the index fingers of both hands of the user are placed on the bottom side of the right controller 3. The same procedure can also be used when the user uses both hands to use the left controller 4 detached from the main unit 2.

[0058] Next, the information processing of this embodiment will be described.

[0059] 10 and 11 are examples of menus displayed on the display of the main unit 2 or an external display. The contents of the menu are not limited, and may be, for example, a menu for selecting an application to be executed on the main unit 2, a menu for changing the settings of the main unit 2, or a menu for selecting some content. Alternatively, the menu may be some menu for a running application.

[0060] In this embodiment, there are two methods for operating a menu using a controller. Note that there may be three or more operation methods. The method for switching between operation methods will be described later. The first operation method is an operation method using a mouse. As shown in FIG. 10, in the first operation method, a cursor 250 included in the menu is continuously moved by operating the mouse. When a predetermined button on the controller, such as the R button 20, is pressed while the cursor 250 is superimposed on one of the items (shown as a to f), a process corresponding to that item is executed. As an example, when the user is operating in the two-handed mouse mode (FIG. 7) or mixed operation mode (FIG. 8), the menu is operated using the first operation method. Note that the cursor 250 may be moved by operating the mouse of either the right controller 3 or the left controller 4, or by both.

[0061] The second operation method is an operation method that does not rely on mouse operation. As shown in Fig. 11, in the second operation method, a frame 251 surrounding an item is moved discontinuously in response to an operation on the direction input unit. For example, when frame 251 surrounds item d, in response to a right operation on the direction input unit, frame 251 instantaneously moves to a position surrounding item e, which is immediately to the right of item d. Then, by pressing a predetermined button on the controller, for example, button A 12, a process corresponding to the item surrounded by frame 251 is executed. As an example, when the user is operating the device in the above-mentioned two-handed vertical holding mode (Fig. 6), the menu is operated by the second operation method.

[0062] In this way, the menu operation method changes depending on how the user holds the controller, improving the operability of the menu operation by the user.

[0063] Note that the processing executed in response to operations on the operation units of the controller may be the same or different between the first and second operation methods. For example, in the first operation method, pressing the R button 20 executes a processing corresponding to an item, while in the second operation method, pressing the R button 20 may not execute a processing corresponding to the item, or may execute another processing. Also, for example, in the second operation method, processing is executed to move the frame 251 and the cursor 250 in response to operations on the stick 42, while in the first operation method, the cursor 250 may not move in response to operations on the sticks 42 and 22.

[0064] The method for changing the process executed in response to an operation on the operation unit of the controller based on the operation method is not limited. For example, in a menu within an application executed on the main unit 2, the process corresponding to the operation on the operation unit may be changed within the application based on the operation method, or the system software of the main unit 2 may change the interpretation of the operation on the operation unit based on the operation method and then pass the operation information to the application.

[0065] 11, a cursor 250 may also be displayed in the second operation method. The cursor 250 is displayed at a position corresponding to the frame 251 or the position and size of an item enclosed by the frame 251. The cursor 250 moves discontinuously together with the frame 251. In the first operation method, the cursor 250 is displayed stationary unless a mouse operation is performed, but in the second operation method, the cursor 250 may be constantly moving. Note that the movement here does not mean, for example, moving to the position of another item, but rather means small vibrations or circular motions based on a predetermined position. This makes it easier for the user to recognize that the current operation method is, for example, the second operation method.

[0066] In the second operation method, the cursor 250 does not have to be displayed. In the second operation method, the cursor is also displayed as described above, but the cursor may be different in color, size, or shape from the cursor displayed in the first operation method. In the second operation method, the frame 251 may not be displayed, and only the cursor may be displayed. In the second operation method, the display means for pointing to an item is not limited, and for example, the color of the item itself may change. In the first operation method and the second operation method, the type, position, or size of the displayed item may be different.

[0067] In the following description, an operation mode using the first operation method may be referred to as a "first operation mode," and an operation mode using the second operation method may be referred to as a "second operation mode."

[0068] Next, various types of data stored in the memory will be described. Figure 12 shows an example of data stored in the memory of the main unit 2. In Figure 12, the program storage area 301 and the data storage area 303 are shown separately for convenience. These do not necessarily have to be separated in substance. Furthermore, smaller storage areas or other storage areas may be conceptually provided.

[0069] The program storage area 301 contains application programs (hereinafter referred to as AP programs) 302. The AP programs 302 may include game application programs, communication application programs, and the like.

[0070] The data storage area 303 includes object data 304 , image data 305 , virtual camera control data 306 , controller data 307 , and a first operation mode flag 318 .

[0071] The object data 304 is data of an object to be placed in the virtual space, such as a menu or a cursor.

[0072] The image data 305 is image data such as animation images, backgrounds, virtual effects, and the like.

[0073] The virtual camera control data 306 is data for controlling a virtual camera that is placed in the virtual space and captures images of the virtual space.

[0074] The controller data 307 is various data output from the controller and is used to determine the operation content. The controller data 307 includes right controller data 308 and left controller data 313. The right controller data 308 includes right button input data 309, right stick input data 310, right inertial sensor value data 311, and right mouse sensor data 312. The left controller data 313 includes left button input data 314, left stick input data 315, left inertial sensor value data 316, and left mouse sensor data 317.

[0075] The right button input data 309 is data indicating operations performed on each button 95 of the right controller 3. The right stick input data 310 is data indicating operations performed on the stick 22 of the right controller 3. The right stick input data 310 indicates the two-dimensional position of the stick. Note that the content of the right stick input data 310 is not limited as long as it can indicate the two-dimensional position, and may, for example, indicate the amount of change in each of the x-axis and y-axis from the initial position, or may indicate the distance and angle from the initial position.

[0076] The right inertial sensor value data 311 is data output from the inertial sensor of the right controller 3, and is data that enables calculation of, for example, the acceleration in the x, y, and z-axis directions (see FIG. 2) and / or the angular velocity around the x, y, and z-axis of the coordinate system of the right controller 3. Using the right inertial sensor value data 311, for example, the attitude, movement, etc. of the right controller 3 can be calculated.

[0077] The right mouse sensor data 312 is output data of the mouse sensor 24 of the right controller 3, and is data that indicates, for example, the movement distance per frame time in the y-axis direction and z-axis direction of the coordinate system of the right controller 3 relative to the placing surface, etc. (i.e., the yz plane; see Figure 2) when the opening 23 of the mouse sensor 24 is blocked by the placing surface, etc.

[0078] The left button input data 314 is data that indicates operations performed on the buttons 82 of the left controller 4. The left stick input data 315 is data that indicates operations performed on the stick 42 of the left controller 4.

[0079] The left inertial sensor value data 316 is data output from the inertial sensor of the left controller 4. As with the right controller 3, the left inertial sensor value data 316 can be used to calculate, for example, the attitude and movement of the left controller 4.

[0080] The left mouse sensor data 317 is output data of the mouse sensor 44 of the left controller 4, and is data that indicates, for example, the movement distance per frame time in the y-axis direction and z-axis direction of the coordinate system of the left controller 4 relative to the placement surface or the like when the opening 43 of the mouse sensor 44 is blocked by the placement surface or the like (i.e., the yz plane; see Figure 3).

[0081] The timing at which the controller data 307 is acquired from the controller is not limited. For example, the data described above may be acquired continuously. Also, for example, in a situation where the running application does not support mouse operation, mouse sensor data may not be acquired. Also, for example, in a situation where the running application supports mouse operation, mouse sensor data may be acquired, but button input data, stick input data, and inertial sensor value data may not be acquired.

[0082] The first operation mode flag 318 is data for determining the current operation mode. When it is on, it indicates the first operation mode, and when it is off, it indicates the second operation mode.

[0083] The above various data are merely examples. Other data may be included, or some data may not be included. The various data do not need to be always present, and may be added, deleted, changed, etc.

[0084] FIG. 13 is an example of a sequence diagram showing menu processing according to this embodiment. Note that only the processing related to switching of the operation mode will be described here, and details of other processing will be omitted. The order of each processing is an example, and for example, each processing may be executed simultaneously or in the reverse order. Also, each processing is described separately for convenience, but may be an integrated processing. Furthermore, the following processing may be executed, for example, at predetermined intervals (for example, every processing frame, every 1 / 30 seconds).

[0085] 13, the process determines whether the direction of each mouse sensor opening, i.e., the negative x-axis direction, is within a predetermined downward range (first range) that includes the direction of gravity (directly downward). If the direction of the mouse sensor opening of one controller is within the first range, the first operation method is used, and if the directions of both mouse sensor openings are outside the first range, the second operation method is used.

[0086] A specific processing example will be described. First, processor 63 determines whether the current operation mode is the first operation mode based on first operation mode flag 318 (S1). If it is the first operation mode (YES in S1), processor 63 then determines whether mouse sensor opening 23 of right controller 3 and mouse sensor opening 43 of left controller 4 are facing sideways or downward (facing perpendicular to the direction of gravity or facing opposite to the direction of gravity). For example, the attitude of each controller is calculated based on the output of the inertial sensor, and if the negative x-axis direction of each controller is outside the first range, it is determined that the controller is facing sideways or upward.

[0087] If the result of the above determination is that both the mouse sensor opening 23 of the right controller 3 and the mouse sensor opening 43 of the left controller 4 are facing sideways or upward (YES in S2), the processor 63 sets the first operation mode flag 318 to OFF (S3). This switches the operation mode from the first operation mode to the second operation mode.

[0088] On the other hand, if at least one of the mouse sensor opening 23 of the right controller 3 and the mouse sensor opening 43 of the left controller 4 is not facing sideways or upwards (NO in S2), the processing in S3 above is skipped.

[0089] On the other hand, if the result of the determination in S1 above is that the current operation mode is the second operation mode (NO in S1), then processor 63 determines whether mouse sensor opening 23 of right controller 3 or mouse sensor opening 43 of left controller 4 is facing downward (S4). For example, the orientation of each controller is calculated, and it is determined whether the negative x-axis direction is within the first range. If the result of this determination is that the mouse sensor opening of either controller is facing downward (YES in S4), processor 63 sets first operation mode flag 318 to ON (S5). This switches the operation mode from the second operation mode to the first operation mode.

[0090] On the other hand, if neither the mouse sensor opening 23 of the right controller 3 nor the mouse sensor opening 43 of the left controller 4 is facing downward (NO in S4), the processing of S5 above is skipped.

[0091] This concludes the description of the sequence diagram showing the menu processing according to this embodiment.

[0092] As described above, in this embodiment, the operation mode is switched based on whether the mouse sensor opening faces downward. If the mouse sensor opening of at least one controller faces downward, it is possible that mouse operation is intended, and therefore switching to the first operation mode is likely to meet the user's intention. On the other hand, when operating the mouse, the user may lift the controller to adjust the mouse position, and at this time, the mouse sensor opening may unintentionally no longer face downward. In light of this, the second operation mode is selected when the mouse sensor openings of both controllers, not just one, are not facing downward. In such cases, it can be assumed that the user actively intends to operate the controllers in the above-mentioned two-handed vertical holding mode, and therefore switching to the second operation mode is likely to meet the user's intention.

[0093] In the above-described process, the "range (threshold)" determined when switching from the second operation mode to the first operation mode may be different from the "range" determined when switching from the first operation mode to the second operation mode. For example, whether to switch from the first operation mode to the second operation mode may be determined based on whether the negative x-axis direction is within a second range that includes a direction perpendicular to the direction of gravity but does not include the direction of gravity and a direction opposite to the direction of gravity. In this case, it can be said that the determination is whether the mouse sensor openings of the → controller and the left controller are facing sideways. Note that the second range and the first range in the above-described process may partially overlap. Alternatively, these ranges may not overlap, and there may be a direction between these ranges that does not belong to either range.

[0094] 13 is an example and can be modified as appropriate. For example, regardless of the current operation mode, it is possible to determine whether "the mouse sensor openings of the right controller 3 and left controller 4 are facing sideways" and whether "the mouse sensor openings of the right controller 3 or left controller 4 are facing downwards." If either determination is YES, the corresponding operation mode is set, and if both determinations are NO, the current operation mode may be maintained.

[0095] Furthermore, for example, when the orientation of the mouse sensor opening of one controller is outside the first range, it may be determined that the orientation of the mouse sensor opening of the other controller is outside the third range in order to switch to the second operation mode. That is, for example, when determining whether both controllers are facing sideways or upwards, the threshold for determining whether one controller is facing sideways or upwards may be different from the threshold for determining whether the other controller is facing sideways or upwards.

[0096] Furthermore, when switching from the second operation mode to the first operation mode, the conditions may also include the mouse sensor opening facing downward and being blocked. For example, if the condition is determined solely by "facing downward," the mouse sensor opening may (temporarily) be facing downward while the device is being held vertically with both hands as shown in Figure 6 above. On the other hand, if the condition is determined solely by "blocking," the mouse sensor opening may be unintentionally blocked by a finger while the device is being held vertically. Therefore, by using both conditions of "facing downward" and "blocking," it becomes easier to infer that operation as a mouse is intended.

[0097] The method for determining the "occlusion" is not limited. For example, a switch or a distance sensor may be provided in the mouse sensor opening. Occlusion may also be determined based on whether the mouse sensor outputs data indicating mouse movement. Alternatively, the determination may be made as follows. For example, the controller data 307 may include "image clarity data," which indicates the clarity of the mouse sensor image. The image clarity data is calculated by the mouse sensor 24. The image clarity data is calculated based on, for example, the brightness of the mouse sensor image and / or the number of feature points in the mouse sensor image. The brightness of the mouse sensor image or the number of feature points in the mouse sensor image may be used as the image clarity data. The image clarity data may also be calculated based on other factors. If the clarity indicated by the image clarity data is equal to or greater than a predetermined value, it can be assumed that the mouse sensor opening 23 is blocked by a placement surface or the like. Using such image clarity data, it may be possible to determine whether the mouse sensor opening is blocked.

[0098] Note that while "being blocked" is a condition for switching from the second operation mode to the first operation mode, when determining whether to switch from the first operation mode to the second operation mode, it is possible to determine that both controllers are facing sideways, and not to require that "not being blocked." Note that "not being blocked" may also be a condition. Furthermore, while "being blocked" is not a condition for determining whether to switch from the second operation mode to the first operation mode, "not being blocked" may be a condition for determining whether to switch from the first operation mode to the second operation mode.

[0099] In the second operation method, mouse operations may not be accepted at all, or, for example, predetermined processes other than the movement of the cursor may be executed based on mouse operations.

[0100] Furthermore, in the second operation method, the cursor 250 or the frame 251 may be moved continuously in response to an operation on the stick 42. For example, in the case where the frame 251 is not present and the cursor 250 is moved, similarly to the first operation method, when a predetermined button is pressed with the cursor 250 superimposed on an item, a corresponding process may be executed.

[0101] The menu may also be a menu in a game application using mouse operation. The type of game is not limited to this, but it may be a menu that is opened by performing a predetermined operation in a game in which the right leg is moved by operating the mouse on the right controller 3 and the left leg by operating the mouse on the left controller 4. Alternatively, it may be a menu that is opened by performing a predetermined operation in a game in which the right wheel is moved by operating the mouse on the right controller 3 and the left wheel is moved by operating the mouse on the left controller 4. Because this is a menu in a game in which the left and right controllers are operated with the mouse, it is possible that the user intends to operate the mouse on the controller. Therefore, switching to the first operation mode based on the fact that the mouse sensor opening of the right or left controller is facing downward is likely to be in line with the user's intention.

[0102] Furthermore, the application examples of the first and second operation methods are not limited to menus, and may be applied to games, for example. For example, in the first operation method, the player character may be moved by operating the mouse on the right controller 3 to move the right leg of the player character and the mouse on the left controller 4 to move the left and right leg of the player character, and in the second operation method, the player character may be moved by operating the left and right sticks to move the left and right legs. Alternatively, in the second operation method, the player character may move in the input direction by operating a single stick rather than operating the left and right legs individually. Alternatively, in the second operation method, instead of operating the player character, the player may operate items or terrain, or open a map to operate the map. Alternatively, a menu may be opened when the second operation method is used, and the game may be returned to when the first operation method is used.

[0103] Furthermore, the conditions used to determine whether to switch the operation mode may include other conditions in addition to those described above. For example, the conditions for determining whether to switch the operation mode may include whether the controller is in a stable position, whether a specific button has been pressed, whether the mouse is being moved (whether the mouse sensor output indicates mouse movement), etc.

[0104] Furthermore, the various determinations described above may be made within the controller, by the operating system of the main unit 2, or by an application executed by the main unit 2.

[0105] Furthermore, a game system is an example of an information processing system, and the information processing system may be a system in which a game is not executed.

[0106] For example, when we say "a processor in an information processing system," the processor may mean one or more processors in one device, such as a main unit, or it may mean some or all of one or more processors in each of multiple devices, such as a main unit and a controller, or a main unit, a controller, and a server. The same applies to "memory in an information processing system" and other configurations.

[0107] The program that causes a computer to execute each process may be a single program, or a group of programs including multiple programs. "A certain program" does not necessarily mean a single program, but may include a group of programs. Furthermore, the programs do not necessarily need to be stored in a single device. "A certain program" may refer to the entire set of programs stored in multiple devices included in an information processing system, for example.

[0108] At least a part of the series of processes may be executed by a server-side device in an information processing system including a terminal-side device and a server-side device capable of communicating with the terminal-side device via a network. Note that the server may be composed of multiple information processing devices, and the processes may be shared and executed by the multiple information processing devices.

[0109] Other embodiments are also described below. For example, the condition for switching from the first operation mode to the second operation mode may be configured so that the mouse sensor opening of at least one of the controllers faces sideways, and the condition for switching from the second operation mode to the first operation mode may be configured so that the mouse sensor openings of both controllers face downward.

[0110] In the above processing example, the controller's orientation (inertial sensor output) is used to determine whether to switch between operation modes. In other embodiments, the controller's orientation is used to determine whether to switch from one operation mode to the other, but the orientation does not need to be used to determine whether to switch from the other to the one. When determining whether to switch from the other to the one, for example, the occlusion status of the mouse sensor opening or input to a button / stick may be used.

[0111] Alternatively, the operation mode may be switched based on the occlusion status of the mouse sensor opening, without using the controller's orientation. For example, switching from the first operation mode to the second operation mode may be performed on the condition that the mouse sensor openings of both controllers are not occluded. Switching from the second operation mode to the first operation mode may be performed on the condition that only one of the mouse sensor openings is occluded. Switching from the second operation mode to the first operation mode may be performed on the condition that the mouse sensor openings of both controllers are occluded. In this case, if only one of the mouse sensor openings is occluded, the current operation mode may be maintained.

[0112] Alternatively, the operation mode may be switched from one mode to another based on whether a mouse movement operation is being performed or not, without using the controller orientation, or based on whether a stick input is being performed or not.

[0113] Alternatively, only one of the controllers may have a mouse function, while the other controller does not. In this case, the operation mode may be switched depending on at least one of the attitude of the controller with the mouse function and the occlusion status of the mouse sensor opening. The operation mode may also affect the operation and interpretation of the controller without the mouse function.

[0114] Each controller may also be provided with a physical switch that switches whether it is used as a mouse or not. When either of the physical switches is on, the first operation mode is selected, and when both physical switches are off, the second operation mode is selected.

[0115] Although the present embodiment, its modifications, and other embodiments have been described above, these descriptions are merely examples in all respects and are not intended to limit the scope of the present embodiment, its modifications, and other embodiments. It goes without saying that various improvements and modifications can be made to the present embodiment, its modifications, and other embodiments. [Explanation of symbols]

[0116] 1. Information Processing Systems 2 Main unit 3, 4 Controller 24, 44 mouse sensor

Claims

1. On the computer, acquiring first input device data including first inertial data according to an output of an inertial sensor of the first input device and first mouse data according to an output of a mouse sensor of the first input device; acquiring second input device data including second inertial data corresponding to an output of an inertial sensor of the second input device and second mouse data corresponding to an output of a mouse sensor of the second input device; Executing a process according to a set operation mode based on the first input device data and the second input device data; setting a first operation mode based on the orientation of the first input device satisfying a first condition; setting the first operation mode based on the orientation of the second input device satisfying a second condition; setting a second operation mode based on the orientation of the first input device satisfying a third condition and the orientation of the second input device satisfying a fourth condition; the processing according to the first operation mode includes processing in which an object is operated based on at least one of the first mouse data and the second mouse data; program.

2. the first condition is a condition that is satisfied when a mouse sensor opening of the first input device faces at least in the direction of gravity; the second condition is satisfied when a mouse sensor opening of the second input device faces at least in the direction of gravity; The program according to claim 1.

3. the third condition is a condition that is satisfied when a mouse sensor opening of the first input device faces at least a direction perpendicular to the direction of gravity, the fourth condition is satisfied when a mouse sensor opening of the second input device faces at least a direction perpendicular to the direction of gravity; The program according to claim 2.

4. setting the first mode based on the attitude of the first input device satisfying the first condition and a condition that is satisfied when a mouse sensor opening of the first input device is blocked; setting the first mode based on whether the attitude of the second input device satisfies the second condition and whether a condition that is satisfied when a mouse sensor opening of the second input device is blocked is satisfied; The program according to claim 3.

5. Regardless of whether a condition that is satisfied when the mouse sensor opening of the first input device is blocked and a condition that is satisfied when the mouse sensor opening of the second input device is blocked are satisfied, setting a second operation mode based on the orientation of the first input device satisfying a third condition and the orientation of the second input device satisfying a fourth condition; The program according to claim 4.

6. the processing according to the second operation mode is processing in which the object is not operated based on the first mouse data and the second mouse data; The program according to claim 1.

7. the processing according to the second operation mode is processing in which no object is operated based on the first mouse data and the second mouse data; The program according to claim 4.

8. the object is a cursor; the first input device data includes first operation unit data corresponding to a user operation on an operation unit of the first input device; the processing according to the second operation mode includes processing of moving the cursor based on the first operation unit data. The program according to claim 1.

9. the processing according to the first operation mode includes processing of continuously moving the cursor based on at least one of the first mouse data and the second mouse data; the processing according to the second operation mode includes processing for discontinuously moving the cursor based on the first operation unit data. The program according to claim 8.

10. the cursor displayed when the first operation mode is set is the same as the cursor displayed when the second operation mode is set; The program according to claim 9.

11. When the first input device and the second input device are not operated, the cursor displayed when the first operation mode is set is stopped, and the cursor displayed when the second operation mode is set is moved; The program according to claim 10.

12. the processing according to the first operation mode includes processing in which an object is operated based on both the first mouse data and the second mouse data; The program according to claim 1.

13. causing the computer to execute a game process; the game processing includes a game portion in which both the first mouse data and the second mouse data are used, and a menu portion in which one of a plurality of operation modes including the first operation mode and the second operation mode is set. The program according to any one of claims 1 to 12.

14. acquiring first input device data including first inertial data according to an output of an inertial sensor of a first input device and first mouse data according to an output of a mouse sensor of the first input device; acquiring second input device data including second inertial data corresponding to an output of an inertial sensor of the second input device and second mouse data corresponding to an output of a mouse sensor of the second input device; Executing a process according to the set operation mode based on the first input device data and the second input device data; setting a first operation mode based on the orientation of the first input device satisfying a first condition; setting the first operation mode based on the orientation of the second input device satisfying a second condition; setting a second operation mode based on the orientation of the first input device satisfying a third condition and the orientation of the second input device satisfying a fourth condition; the processing according to the first operation mode includes processing in which an object is operated based on at least one of the first mouse data and the second mouse data; Information processing methods.

15. 1. An information processing system including at least one processor, The processor: acquiring first input device data including first inertial data according to an output of an inertial sensor of the first input device and first mouse data according to an output of a mouse sensor of the first input device; acquiring second input device data including second inertial data corresponding to an output of an inertial sensor of the second input device and second mouse data corresponding to an output of a mouse sensor of the second input device; Execute a process according to the set operation mode based on the first input device data and the second input device data; setting a first operation mode based on the orientation of the first input device satisfying a first condition; setting the first operation mode based on whether the orientation of the second input device satisfies a second condition; setting a second operation mode based on the orientation of the first input device satisfying a third condition and the orientation of the second input device satisfying a fourth condition; the processing according to the first operation mode includes processing in which an object is operated based on at least one of the first mouse data and the second mouse data; Information processing system.

Images