Program, information processing system, and information processing method

Abstract

The preferred data may vary depending on the application and its state. [Solution] Based on first input data corresponding to an operation on the input device's control unit and second input data corresponding to the output of the mouse sensor, a first response having a first data structure including first data based on the first input data and a second response having a second data structure including second data based on the first input data and third data based on the second input data are provided in response to the first and second requests of the application.

Description

Technical Field

[0001] This disclosure relates to data processing based on operations on an input device.

Background Art

[0002] Conventionally, input devices such as game controllers have been known. (For example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The application execution unit may execute certain processing based on an operation on the input device. Depending on the application executed by the application execution unit or the state of the application, even for the same operation on the input device, the preferred data for the application based on the same operation may be different.

Means for Solving the Problems

[0005] For example, the following configuration examples can be cited.

[0006] The first configuration example is a program that causes a computer to acquire first input data corresponding to an operation on a first operation unit of a first input device and second input data corresponding to the output of the mouse sensor of the first input device, to generate a first response having a first data structure in response to a first request from a game application, and to generate a second response having a second data structure in response to a second request from a game application, wherein the first response includes first data generated based on the first input data, and the second response includes second data generated based on the first input data and third data generated based on the second input data.

[0007] As a second example of configuration, in the first example of configuration, the first data may indicate that the first item is at least one of two values, and the second data may indicate that the second item is at least one of two values.

[0008] As a third configuration example, in the first or second configuration example, the first data and the second data may be generated based on the same first input data.

[0009] As a fourth configuration example, in the first to third configuration examples, the computer may be made to acquire third input data corresponding to an operation on the second operation unit of the first input device, to make a first response in response to a first request that includes fourth data generated based on the first data and the third input data, and to make a second response in response to a second request that includes second data generated based on the first input data or the third input data.

[0010] As a fifth configuration example, in the first to fourth configuration examples, the computer may be instructed to modify the second data generated based on the first input data in response to a third request from the game application.

[0011] As a sixth configuration example, in the first to fourth configuration examples, the second data may indicate that at least one item is one of at least two values, and the computer may be instructed to change the value of at least one of the items based on the input data corresponding to the operation on any of the control units in response to a third request from the game application.

[0012] As a seventh configuration example, in the sixth configuration example, the computer may be instructed to acquire fourth input data corresponding to an operation on the third operation unit of the first input device, the first response may include fifth data generated based on the fourth input data, and the third request may be instructed to set which of at least one item's values ​​to change based on input data corresponding to an operation on any operation unit other than the third operation unit.

[0013] As an eighth example of configuration, in at least the first example of configuration, the first operating unit is a stick, the first input data indicates the two-dimensional position of the stick, the first data indicates the two-dimensional position, and the second data may indicate at least one of two values.

[0014] As a ninth configuration example, in the first to eighth configuration examples, the computer may be instructed to provide a third response having a third data structure that includes a sixth data generated based on the second input data, in response to a fourth request from a game application.

[0015] As a tenth example of the configuration, in the ninth example, the sixth data point may represent a two-dimensional change, and the third data point may represent a two-dimensional coordinate.

[0016] As a 11th configuration example, in the 1st to 10th configuration examples, the computer may be caused to acquire fifth input data corresponding to an operation on a fourth operation unit of a second input device and sixth input data corresponding to an output of a mouse sensor of the second input device, and in response to a fifth request from a game application, cause a fourth response having a fourth data structure including seventh data generated based on the first input data and the fifth input data and eighth data generated based on the second input data and the sixth input data.

[0017] As a 12th configuration example, in the 1st to 11th configuration examples, the second response may include ninth data indicating whether a game application, which is a requester of the second request, is a target of a mouse operation.

[0018] As a 13th configuration example, in the 1st to 12th configuration examples, the second response may include tenth data indicating that the first input device is being mouse-operated with an opening exposing the mouse sensor facing the ground direction.

[0019] Each of the configuration examples described above may be applied to an information processing method or an information processing system.

Brief Description of Drawings

[0020] [Figure 1] A diagram showing an example of a state where a right controller 3 and a left controller 4 are attached to a main body device 2 [Figure 2] A six-sided view showing an example of a right controller 3 [Figure 3] A six-sided view showing an example of a left controller 4 [Figure 4] A block diagram showing an example of the configuration of a main body device 2 [Figure 5] A block diagram showing an example of the configuration of a main body device 2, a right controller 3, and a left controller 4 [Figure 6] A diagram showing an example of a state where a right controller 3 is operated with a right hand placed on it [Figure 7] A diagram showing an example of a state where a right controller 3 is gripped and operated with a right hand [Figure 8] A diagram showing an example of a state in which the right controller 3 is held and operated with both hands [Figure 9] Functional block diagram showing an example of a function [Figure 10A] A diagram for explaining an example of wheel data [Figure 10B] A diagram for explaining an example of wheel data [Figure 11A] A diagram showing an example of mouse button data [Figure 11B] A diagram showing an example of mouse button data [Figure 12] A diagram showing an example of various data [Figure 13] An example of a sequence [Figure 14] Functional block diagram showing an example of a function [Figure 15] A diagram for explaining an example of wheel data [Figure 16] A diagram for explaining an example of wheel data [Figure 17] A diagram for explaining an example of wheel data [Figure 18] A diagram for explaining an example of wheel data [Figure 19] A diagram for explaining an example of wheel data [Examples of the hardware configuration of the information processing system]

[0021] Hereinafter, an embodiment will be described

[0022] [Examples of the hardware configuration of the information processing system]

[0023] The following describes a game system, which is an example of an information processing system. In this embodiment, the game system 1 includes a main unit 2, a left controller 4, and a right controller 3. The main unit 2 is an example of an information processing device. The information processing device may be, for example, a personal computer, a tablet terminal, a smartphone, a wearable terminal, a server, etc. The left and right controllers 3 and 4 are examples of input devices. The input device may be, for example, a general-purpose mouse. The input device is also an example of an information processing device. The information processing device and input device are examples of a computer. Note that "computer" does not necessarily mean a single device, but may refer to an entire system in which multiple devices are connected by wires or wirelessly. Furthermore, each information processing device and input device may be an example of an information processing system.

[0024] The main unit 2 is a device that performs various processes (for example, game processing) in the game system 1. The main unit 2 is equipped with a display 72.

[0025] The left controller 4 and the right controller 3 are equipped with operating parts for user input. In the following, the left controller 4 and the right controller 3 may be collectively referred to as "controllers". Figure 2 is a six-sided schematic diagram showing an example of the right controller 3. As shown in Figure 2, the right controller 3 is a vertically elongated plate shape, equipped with a housing 11, and has a front, rear, top, bottom, right, and left section. As will be described later, an opening for a mouse sensor is provided in the bottom section. In the right controller 3, the rear section is located opposite the front section, the bottom section is located opposite the top section, and the left section is located opposite the right section. The distance between the front section and the rear section is greater than the distance between the top section and the bottom section. The distance between the top section and the bottom section is greater than the distance between the right section and the left section. In other embodiments, the relative sizes of these distances may be different. In this embodiment, the direction connecting the bottom and top is called the vertical direction, the direction perpendicular to the vertical direction and connecting the front and rear is called the front-to-back direction, and the direction perpendicular to the vertical direction and connecting the right and left is called the left-to-right direction. In Figure 2, the coordinate system of the right controller 3 is shown by illustrating the x, y, and z axes with respect to a front view where the left side faces forward. In this coordinate system, the direction from the left side to the right side is the positive z-axis direction. Also, the direction perpendicular to the z-axis and going from the bottom to the top is the positive x-axis direction, and the direction perpendicular to the z and x axes and going from the rear to the front is the positive y-axis direction. When the bottom faces the direction of gravity, the negative x-axis direction and the direction of gravity coincide. In this embodiment, the front and bottom parts do not need to be perfectly flat and may have irregularities or slopes. For example, the bottom includes the convex part 25 described later. The directions of each part and the directions connecting each part are approximate directions.

[0026] The right controller 3 has a protrusion 25 that fits into a recess (not shown) of the main unit 2 when it is mounted on the main unit 2. As shown in Figure 2, the protrusion 25 is a convex shape that protrudes in the negative x-axis direction, with a width shorter in the left-right direction than the left-right direction of the right controller 3 and a width shorter in the front-back direction than the front-back direction. In this embodiment, the protrusion 25 is part of the bottom.

[0027] As will be described later, the right controller 3 can also be held in a vertical orientation when detached from the main unit 2. When held in a vertical orientation, the right controller 3 is shaped and sized to be held with one hand, especially the right hand. The right controller 3 can also be held in a horizontal orientation, and in this case, it may be held with both hands.

[0028] The right controller 3 is equipped with an analog stick (sometimes simply called a "stick") 22 on its left side, which is an example of a directional input unit. The stick 22 can be used as a directional input unit that can input directions. The user can input directions according to the direction of tilt by tilting the stick 22 in any direction within 360 degrees, and the magnitude of the input can be according to the angle of tilt. The user can also input buttons by pressing down on the stick 22. The stick may be slideable instead of tiltable. The directional input unit may be, for example, a directional pad.

[0029] The right controller 3 has a set of four buttons on its left side: A button 12, B button 13, X button 14, Y button 15, a + (plus) button 16, and a home button 17. The right controller 3 has an R button 20 and a ZR button 21 extending across its front and top. The R button 20 and ZR button 21 may be located only on the front of the right controller 3, or only on the top. The right controller 3 has buttons 18 and 19 on the top surface 25a of the protrusion 25. The right controller 3 does not have an operating section on its rear.

[0030] The right controller 3 is provided with an opening 23 for a mouse sensor on the top surface 25a of the protrusion 25. The opening 23 for the mouse sensor is an opening in the light guide path that guides light to the mouse sensor 24 located inside the right controller 3. The mouse sensor 24 is an optical mouse sensor and includes at least a light receiving unit. The light detected by the light receiving unit may be visible light or light of an invisible wavelength. The mouse sensor 24 may also include a light emitting unit. Light from the light emitting unit may be radiated to the outside through the opening 23. The mouse sensor 24 acquires data that allows for the calculation of the movement of the right controller 3 on the mounting surface, where the top surface 25a of the protrusion 25 of the bottom is facing the mounting surface. As a result, the right controller 3 can also be used as a mouse. In this embodiment, when the right controller 3 is placed on the mounting surface with its top surface 25a facing the mounting surface, the direction in which the bottom extends is parallel to the direction in which the mounting surface extends.

[0031] Furthermore, in this embodiment, the right controller 3 is provided with a terminal 26 on the protrusion 25 for the right controller 3 to communicate with the main unit 2 via wired connection. As an example, the terminal 26 is provided on the inner circumferential surface of a recess provided on the top surface 25a of the protrusion 25.

[0032] Figure 3 is a hexagonal schematic diagram showing an example of the left controller 4. The same configuration as the right controller 3 will not be explained. The left controller 4 has a stick 42, a set of four buttons (right direction button 32, down direction button 33, up direction button 34, left direction button 35), a capture button 37, and a minus button 36 on its right side. Buttons 32-35 may be a single directional pad. The left controller 4 does not have an operating section on its rear. Note that in the right controller 3, the stick 22 is located behind buttons 12-15, whereas in the left controller 4, the stick 42 is located in front of buttons 32-35. In Figure 3, the coordinate system of the left controller 4 is shown by illustrating the x, y, and z axes relative to a front view with the right side facing forward. In this coordinate system, the direction from the right side to the left side is the positive z-axis direction. Furthermore, the direction perpendicular to the z-axis and moving from the bottom to the top is the positive x-axis direction, and the direction perpendicular to both the z-axis and x-axis and moving from the rear to the front is the positive y-axis direction. Note that when the bottom is facing the direction of gravity, the negative x-axis direction and the direction of gravity coincide.

[0033] The left controller 4 has a protrusion 45 that fits into a recess (not shown) of the main unit 2 when it is attached to the main unit 2. The protrusion 45, like the right controller 3, has buttons 38 and 39, a mouse sensor 44, and a terminal 46.

[0034] When the left controller 4 is detached from the main unit 2, it can be held in either a vertical or horizontal orientation, similar to the right controller 3.

[0035] Figure 4 is a block diagram showing an example of the configuration of the main unit 2. The main unit 2 includes a processor 63. The processor 63 is an information processing unit that performs various information processing tasks performed in the main unit 2. The processor 63 may consist of, for example, multiple processors or cores, typically multiple CPUs (Central Processing Units) or cores, or it may consist of a SoC (System-on-a-chip) that includes multiple functions such as CPU function and GPU (Graphics Processing Unit) function. In this embodiment, the "processor" may include at least a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), etc. The processor 63 performs various information processing tasks by executing a program (for example, a program that runs a game application) stored in a storage unit (for example, an internal storage medium such as a flash memory 68, or an external storage medium installed in a slot 51, etc.).

[0036] The main unit 2 includes, as an example of an internal storage medium, a flash memory 68 and a DRAM (Dynamic Random Access Memory) 69. The flash memory 68 is a memory mainly used to store various types of data stored in the main unit 2. The DRAM 69 is a memory mainly used to temporarily store various types of data used in information processing. The processor 63 performs various information processing by appropriately reading and writing data to and from storage mediums such as the flash memory 68 and the DRAM 69. In this embodiment, "memory" may include at least flash memory and DRAM, or it may include other storage mediums.

[0037] Furthermore, the main unit 2 has various configurations as shown in Figure 4. These are briefly described below. The recording medium slot interface (sometimes referred to as "slot I / F") 52 reads and writes data to and from the storage medium (for example, a dedicated memory card) installed in the recording medium slot 51, according to instructions from the processor 63. The second slot I / F 54 reads and writes data to and from the storage medium installed in the second slot 53, according to instructions from the processor 63.

[0038] The network communication unit 66 communicates with external devices via the network (for example, internet communication using wireless communication). The controller communication unit 67 communicates wirelessly with the left controller 4 and / or the right controller 3 (for example, communication in accordance with the Bluetooth® standard).

[0039] The left terminal 50 is a terminal for wired communication between the processor 63 and the left controller 4. The right terminal 65 is a terminal for wired communication between the processor 63 and the right controller 3. The lower terminal 64 is a terminal for communication with other devices (e.g., a stationary monitor, etc.) via the cradle when the lower terminal 64 is mounted on the cradle.

[0040] The codec circuit 74 controls the input and output of audio data to the speaker 73 and the audio input / output terminal 75.

[0041] The power control unit 61 controls the power supply from the battery 62 to each part of the main unit 2 (i.e., each part that receives power from the battery 62) based on commands from the processor 63, and also starts or stops the power supply in response to the pressing of the power button 60.

[0042] The main unit 2 is equipped with various sensors, such as an acceleration sensor 76 and an angular velocity sensor 77. The processor 63 can perform various processes based on the information from these sensors.

[0043] Figure 5 is a block diagram showing an example configuration of the main unit 2, left controller 4, and right controller 3. Note that the details of the main unit 2 configuration are shown in Figure 4 and are therefore omitted in Figure 5.

[0044] The left controller 4 includes a communication control unit 80 that communicates with the main unit 2. As shown in Figure 5, the communication control unit 80 is connected to each component, including terminals 88. When the left controller 4 is attached to the main unit 2, the communication control unit 80 communicates with the main unit 2 via wired communication through terminals 88. When the left controller 4 is detached from the main unit 2, the communication control unit 80 communicates with the main unit 2 wirelessly (for example, communication in accordance with the Bluetooth® standard).

[0045] The left controller 4 includes a memory 81, such as flash memory. The communication control unit 80 is composed of a processor, such as a microcontroller (also called a microcomputer), and performs various processes by executing firmware stored in the memory 81.

[0046] The communication control unit 80 acquires information regarding the operations performed on each button 82 and stick 42.

[0047] The left controller 4 is equipped with an inertial sensor. Specifically, the left controller 4 is equipped with an acceleration sensor 83 and an angular velocity sensor 84. The acceleration sensor 83 and the angular velocity sensor 84 are each connected to the communication control unit 80. The communication control unit 80 repeatedly acquires the detection results of the acceleration sensor 83 and the angular velocity sensor 84 at appropriate intervals. Note that the inertial sensor may be either the acceleration sensor or the angular velocity sensor, or it may be another type of sensor.

[0048] The left controller 4 is equipped with a mouse sensor 44. The mouse sensor 44 acquires data for calculating the movement of the left controller 4 placed on the mounting surface. The communication control unit 80 repeatedly acquires the data acquired by the mouse sensor 44 at appropriate intervals.

[0049] The communication control unit 80 transmits left controller data, which includes the acquired information or information that has undergone predetermined processing, to the main unit 2. The left controller data may be transmitted repeatedly at predetermined intervals. For example, the communication control unit 80 may transmit left controller data in response to data requests repeatedly received from the main unit 2 at predetermined intervals.

[0050] The left controller 4 includes a power supply unit 87. The power supply unit 87 has a battery and a power control circuit. The power control circuit is connected to the battery and supplies power to each part of the left controller 4 (specifically, each part that receives power from the battery).

[0051] As shown in Figure 5, the right controller 3 has the same configuration as the left controller 4. For example, the right controller 3 includes a communication control unit 91, which is composed of a processor and the like, and communicates with the main unit 2. The right controller 3 also includes a memory 94 connected to the communication control unit 91. The communication control unit 91 is connected to each component, including the terminals 92. The communication control unit 91 and the memory 94 have the same functions as the communication control unit 80 and memory 81 of the left controller 4. Therefore, the communication control unit 91 can communicate with the main unit 2 both by wired communication via the terminals 92 and by wireless communication without the terminals 92, and controls the communication that the right controller 3 makes to the main unit 2. For example, the communication control unit 91 acquires information about operations performed on each button 95 and stick 22 and transmits it to the main unit 2.

[0052] The right controller 3 includes a processing unit 90 and an NFC antenna 93. The processing unit 90 controls the NFC antenna 93 in response to commands from the main unit 2 via the communication control unit 91. The NFC antenna 93 performs short-range wireless communication based on the NFC (Near Field Communication) standard.

[0053] Furthermore, the main unit 2 may have pre-existing data relating to at least the right controller 3 and the left controller 4. For example, controller type information may be assigned to the right controller 3 and the left controller 4 during manufacturing, and the main unit 2 may have various other information along with this controller type information.

[0054] [Regarding the gripping configuration of the controller] Figure 6 is a schematic diagram showing an example of a state in which a user places their right hand on the right controller 3 and places it on a mounting surface to use it as a mouse. As shown in Figure 6, from the user's perspective, the front of the right controller 3 faces forward and the left side faces left. The user's right palm covers the upper side of the right controller 3. The user's right thumb is positioned on the left side of the right controller 3. The user's right thumb is placed on, for example, the A button 12. The user's right index finger is placed on, for example, the R button 20, and the user's right middle finger is placed on, for example, the ZR button 21. The user can operate the R button 20 and the ZR button 21 with their right index finger or middle finger. The user can operate each input unit located on the left side with their right thumb. Note that when the user uses the left controller 4 as a mouse with their left hand, they can use it in the same manner by placing their left hand on the left controller 4 and placing it on the mounting surface. In this case, the right side of the left controller 4 faces to the right. The user may place their left hand on the right controller 3 so that the front faces forward, or they may place their right hand on the left controller 4 so that the front faces forward.

[0055] Figure 7 is a schematic diagram showing an example of a user operating the right controller 3 by holding it vertically with their right hand. As shown in Figure 7, when the right controller 3 is detached from the main unit 2, it can be used by holding it so that its longitudinal direction is in the vertical or front-to-back direction for the user. The same procedure can be used when the user uses the left controller 4, which is detached from the main unit 2, with their left hand.

[0056] Figure 8 is a schematic diagram showing an example of how a user operates the right controller 3 by holding it horizontally with both hands. As shown in Figure 8, when the right controller 3 is detached from the main unit 2, it is used so that its longitudinal direction is in the left-right direction for the user. The right controller 3 is held by one hand on one side in the longitudinal direction (one side in the front-to-back direction) and by the other hand on the other side. For example, the index fingers of both hands of the user are placed on the bottom side of the right controller 3. The same procedure can be used when the user uses the left controller 4, which is detached from the main unit 2, with both hands.

[0057] Furthermore, one user may use the right controller 3 while another user uses the left controller 4. Alternatively, one user may simultaneously use the right controller 3 with their right hand and the left controller 4 with their left hand. Also, two users may each use both the right controller 3 and the left controller 4. In other words, more than two controllers may be connected to the main unit 2. For example, four users may each use one or two controllers.

[0058] Next, the information processing of this embodiment will be described. The following explanation will use the case where the right controller 3 is used as an example. Note that the same considerations apply when the left controller 4 is used, so its explanation will be omitted except for a few parts.

[0059] [Examples of functional blocks in an information processing system] Figure 9 shows an example of a functional block of the information processing system 1. As shown in Figure 9, the main unit 2 has an OS execution unit 110 and an application (sometimes referred to as "AP") execution unit 113 as functional blocks. The application is, for example, a game application. When the game application is executed, game processing is realized, for example. The game application may also include a game menu application that executes a menu for the user to select the game processing to be executed. The application may also be a communication application, a drafting application, a map application, etc. The processing of each part will be described below.

[0060] As shown in Figure 9, right controller data is transmitted from the right controller 3 to the OS execution unit 110. This data includes, as an example, image clarity data, dy / dz data, inertial sensor value data, button input data, and stick input data. Note that the data transmitted from the right controller 3 to the OS execution unit 110 may include other data, or may not include any of these data.

[0061] Image clarity data is data indicating the clarity of the mouse sensor image. Image clarity data is data calculated by the mouse sensor 24. Image clarity data is calculated, for example, based on the degree of brightness of the mouse sensor image and / or the degree of the number of feature points in the mouse sensor image. Note that the degree of brightness of the mouse sensor image or the degree of the number of feature points in the mouse sensor image may be used directly as image clarity data. In addition, image clarity data may be calculated based on other factors. If the clarity indicated by the image clarity data is above a predetermined level, it can be estimated that the opening 23 of the mouse sensor 24 is blocked by the mounting surface, etc. If there is data that allows it to be estimated that the opening 23 of the mouse sensor 24 is blocked, other data may be used instead of data indicating the clarity of the mouse sensor image.

[0062] The dy / dz data is the output data of the mouse sensor 24, and when the opening 23 of the mouse sensor 24 is blocked by the mounting surface, it is data that indicates the distance traveled per frame time in the y-axis and z-axis directions (i.e., the yz plane; see Figure 2) of the coordinate system of the right controller 3 relative to the mounting surface, etc. Note that "dy / dz data" is simply a name for identification purposes, and it may also be called, for example, "dx / dy data". The same applies to other data.

[0063] The inertial sensor data is output from the inertial sensor of the right controller 3, and is data that allows for the calculation of, for example, the acceleration in the xyz axis direction (see Figure 2) and / or angular velocity around the xyz axis of the coordinate system of the right controller 3. Using the inertial sensor data, for example, the attitude and movement of the right controller 3 can be calculated.

[0064] Button input data indicates the operation performed on each of the buttons 95 of the right controller 3. Stick input data indicates the operation performed on the stick 22 of the right controller 3. Stick input data indicates the two-dimensional position of the stick. Note that the content of the stick input data is not limited as long as it indicates the two-dimensional position; for example, it may show the change in the x and y axes from the initial position, or it may show the distance and angle from the initial position.

[0065] The OS execution unit 110 generates pad data and mouse data based on the right controller data. The OS execution unit 110 also generates mouse operation data based on the pad data and mouse data. Some or all of this data generation may be performed continuously, or it may be performed, for example, in response to a request from the AP execution unit 113.

[0066] The AP execution unit 113 executes an application. The AP execution unit 113 may execute multiple applications simultaneously. Depending on the application being executed, the AP execution unit 113 may request one or more data from the OS execution unit 110, for example, from pad data, mouse data, and mouse operation data. For example, the AP execution unit 113 may individually request each data from the OS execution unit 110 using the API (Application Programming Interface) corresponding to each data. The AP execution unit 113 may change the data it requests depending on the application being executed and the state of that application, or it may choose not to request any data at all. For example, the AP execution unit 113 may issue a single request to the OS execution unit 110, and the AP execution unit 113 may provide a response that includes multiple data from the pad data, mouse data, and mouse operation data. The AP execution unit 113 may also request data other than these from the OS execution unit 110, or it may choose not to request any of these data.

[0067] The pad data has a data structure that includes information indicating operations on each button and stick. For example, the pad data has a data structure that includes information indicating operations on the A button 12, B button 13, X button 14, Y button 15, + (plus) button 16, R button 20, ZR button 21, R stick 22 (pressed), button 18 and button 19. For example, the pad data has a data structure that includes information indicating whether or not the R button 20 was pressed. For example, the pad data has a data structure that includes information indicating operations on the 2D position of the stick 22. Note that information regarding operations on the home button 17 and information regarding the inertial sensor are not included in the pad data and may be included in a separate response. Similarly, information indicating operations on the 2D position of the stick 22 is not included in the pad data and may be included in a separate response. Similarly, for the left controller 4, the pad data has a data structure that includes information indicating operations on each button and the stick. Note that information regarding operations on the capture button 37 is not included in the pad data.

[0068] The mouse data has a data structure that includes, for example, dy / dz data and occlusion data. The dy / dz data has a data structure that includes information indicating the distance traveled per frame time in the y-axis and z-axis directions (i.e., the yz plane; see Figure 2) of the coordinate system of the right controller 3. The dy / dz data is generated based on the dy / dz data included in the right controller data. These may be the same data. That is, even if one piece of data and another piece of data are the same data, it can be said that the other piece of data is generated from the other piece of data. The occlusion data has a data structure that includes information indicating whether or not the opening 23 of the right controller 3 is closed. As an example, the occlusion data may include a value that indicates the occlusion of the opening 23 when either the dy / dz data indicates the movement of the controller, or the image clarity data indicates that the image is clear.

[0069] The processing that the AP execution unit 113 performs based on the acquired data is not limited. For example, the AP execution unit 113 may control the position of the pointer based on the acquired dy / dz data, depending on the application being executed. The AP execution unit 113 may also switch the display of the pointer on or off based on the acquired blocking data, for example.

[0070] Mouse operation data has a data structure that includes, for example, x / y coordinate data, x / y coordinate range data, wheel data, mouse button data, and attribute data.

[0071] The x / y coordinate data is generated based on the dy / dz data included in the mouse data. Alternatively, the x / y coordinate data may be generated based on the dy / dz data included in the right controller data. The x / y coordinate data has a data structure that includes information indicating the two-dimensional coordinates of the pointer. For example, the x / y coordinate data may be generated by adding a value based on newly acquired dy data to the previous x coordinate data, and adding a value based on newly acquired dz data to the previous y coordinate data.

[0072] The x / y coordinate range data has a data structure that includes information indicating the range of values ​​that the x and y coordinates can each take. The x / y coordinate range data may be set based on a request from the AP execution unit 113.

[0073] The AP execution unit 113 may, for example, control the position of the pointer based on the acquired x / y coordinate data.

[0074] Wheel data is generated based on stick input data. Figure 10A is a schematic diagram showing the state when the right controller 3 is operated by a mouse. Depending on the application, operations on the stick 22 may include scrolling pages or bars, or moving the selection of items up and down, similar to the operation of a typical mouse wheel. In this case, when a user attempts an operation equivalent to a backward rotation of the wheel, that is, an operation to scroll a page downward, the direction in which the stick 22 is tilted may differ depending on the user. For example, one user may tilt the stick 22 backward (negative y-axis direction), while another user may tilt the stick 22 downward (negative x-axis direction). The reverse is also true. Therefore, in this embodiment, the OS execution unit 110 generates wheel data based on the stick input data included in the right controller data, indicating whether the position of the stick 22 is within a first region including backward and downward from the initial position, or within a second region including forward and upward. The wheel data has a data structure that includes information indicating which region the two-dimensional position of the stick 22 is located in. The first and second regions may be set symmetrically with respect to the initial position. The first and second regions may also be set with respect to a boundary. For example, the boundary may be a straight line passing through the center of the stick 22 and extending from a direction 45 degrees forward and downward to a direction 45 degrees backward and upward. When the position of the stick 22 is on the boundary line, it can be appropriately decided whether to treat that position as being within the first region, within the second region, or neither. In this embodiment, the first and second regions are subcircles centered on the initial position of the stick 22.

[0075] For example, the wheel data may indicate a negative value when the position of stick 22 is within the first region and a positive value when it is within the second region. That is, the wheel data may indicate the direction of wheel operation in a typical mouse wheel. The wheel data may have a data structure that includes information indicating a binary value (which may also include cases where neither value is indicated). As an example, the positive or negative corresponding to the wheel operation direction may be indicated by a sign, by strings such as positive and negative, or by an appropriate flag.

[0076] The wheel data has a data structure that includes information indicating the two-dimensional position of the stick 22 from its initial position, i.e., the amount of movement. The wheel data may also have a data structure that includes information indicating a value equivalent to the amount of rotation of the wheel in a typical mouse. In a typical mouse, for example, a change of 120 is often transmitted for every 15 degrees of rotation. If the device to which the mouse is connected acquires the change amount every frame, the mouse wheel operation speed is slow relative to the time of one frame, so the device may acquire values ​​such as 0, 120, 0, 0, 0, ... On the other hand, if the value indicated by the wheel data when the stick 22 is tilted to its limit is 120, the AP execution unit 113 may acquire wheel data showing values ​​such as 120, 120, 120, ... in consecutive frames. Such values ​​may result in an excessive amount of wheel rotation (or wheel rotation speed) for the application executed by the AP execution unit 113. Therefore, in this embodiment, the value indicated by the wheel data when the stick 22 is tilted to its limit is set to a value less than 120, for example, a value between 1 and 50, a value between 5 and 30, or a value between 15 and 20. Even if the main unit 2 acquires the tilt of the stick 22 every frame or less, the generated wheel data may reflect that value only once every few frames. In other words, by reducing the frequency of reflection in the wheel data, the value corresponding to the amount of tilt of the stick 22 may be made closer to, for example, the value in a typical mouse. Furthermore, the wheel data does not necessarily have to include information indicating the amount of change in the wheel.

[0077] Figure 10B is a schematic diagram showing the state when the left controller 4 is being operated with a mouse. Similar to the right controller 3, wheel data is generated based on the stick data contained in the left controller data. The wheel data may, for example, be negative when the position of the stick 42 is within a first region including behind and below the initial position, and positive when the position of the stick 42 is within a second region including in front and above the initial position.

[0078] The AP execution unit 113 may switch selections, increase or decrease numbers, or scroll pages or bars based on the acquired wheel data.

[0079] Mouse button data is generated based on button input data. The mouse button data has a data structure that includes information indicating the operation for each button type. The button type is an item to which each button can be associated. Figure 11A shows the button type item and its corresponding button included in the button input data based on the right controller data. For example, button type = PrimaryButton is associated with the R button 20 and the ZR button 21. When either the R button 20 or the ZR button 21 is pressed, the mouse button data includes data with a data structure that includes information indicating that PrimaryButton is ON. Note that no corresponding buttons are set for other button types, so no matter which button is pressed, none of the button types included in the mouse button data will indicate ON. Figure 11B shows an example where different corresponding buttons are set for different button types. In this example, for example, when the ZR button 21, X button 14, and B button 13 are pressed, mouse button data is generated that includes information indicating that SecondaryButton, Button 3, and Button 5 are ON. Thus, it may be possible to change which button corresponds to each button type. Changes may be performed based on a request from the AP execution unit 113, or they may be performed spontaneously by the OS execution unit 110 or based on user input. The default correspondence between each button type and the button is not limited and may be, for example, the correspondence shown in Figure 11A.

[0080] For each button type, all buttons may be mappable, or some buttons may not be mappable. For example, with respect to the mouse button data of the right controller 3, the buttons that can be mappable to each button type may be A button 12, B button 13, X button 14, Y button 15, + (plus) button 16, R button 20, ZR button 21, and R stick 22 (press). In other words, buttons 18 and 19 do not need to be mappable. This is because these buttons are in close proximity to the desk or other surface when the mouse is being used, and are not generally expected to be operated by the user. Therefore, in this case, AP execution 113 can recognize when buttons 18 or 19 are pressed by acquiring the pad data, but it cannot recognize this from the mouse operation data. The same may apply to the left controller 4.

[0081] The AP execution unit 113 may perform various processes such as confirming, canceling, selecting, attacking, and jumping based on the acquired mouse button data.

[0082] Attribute data has a data structure that includes, for example, data indicating the target state and data indicating whether or not the mouse is operating on a horizontal plane.

[0083] The data indicating the target status has a data structure that includes information indicating whether the application that was executed by the AP execution unit 113 and requested mouse operation data including attribute data is the target of the operation. For example, the data indicating the target status will show False if the application is running in the background or if a menu that is more user-operated on the display (e.g., the quick settings menu) is displayed.

[0084] The AP execution unit 113 may, for example, switch the display of the pointer based on the acquired attribute data.

[0085] The data indicating whether or not the mouse is operating on a horizontal plane has a data structure that includes information such as the bottom of the controller facing the direction of gravity and the dy / dz data indicating the movement of the controller.

[0086] The AP execution unit 113 may, for example, choose not to move or display the pointer when the horizontal mouse operation state is False, even if the dy / dz data indicates controller movement, because it is possible that the user simply covered the opening 23 with their finger while holding the controller.

[0087] The AP execution unit 113 can use pad data, mouse data, and mouse operation data in the running application. Furthermore, the timing at which the AP execution unit 113 acquires each of these data is not limited. For example, the AP execution unit 113 may acquire these three data at all times. Also, for example, in situations where the running application does not support mouse operation, the AP execution unit 113 may acquire pad data but not mouse operation data or mouse data. Also, for example, in situations where the running application supports mouse operation, the AP execution unit 113 may acquire mouse operation data but not pad data or mouse data. The AP execution unit 113 may, for example, execute application processing based on x / y coordinate data and mouse button data included in the mouse operation data.

[0088] Furthermore, the AP execution unit 113 may acquire pad data in addition to mouse operation data when the running application responds to mouse operations. For example, suppose that the mouse button data has the correspondence between button type and corresponding button as shown in Figure 11B. In this case, when button A of the right controller 3 is pressed, the AP execution unit 113 cannot obtain the press from the mouse button data, but can obtain the press from the pad data. In this way, the AP execution unit 113 may use each data complementaryly. Also, for example, when button R 20 is pressed, the AP execution unit 113 can obtain the press in the format Rbutton=ON based on the data format of the pad data, and can also obtain it in the format PrimaryButton=ON based on the data format of the mouse operation data. Similarly, the AP execution unit 113 may acquire mouse operation data and mouse data, in which case x / y coordinate data can be obtained from the mouse operation data, while dy / dz data can be obtained from the mouse data. Thus, whether the AP execution unit 113 acquires pad data, mouse data, or mouse operation data, or all of them, and how it uses the acquired data, may be determined based on the developer's intentions, the application status, etc.

[0089] In this embodiment, as shown in Figures 6 to 8, the controller can be used in a variety of ways. Therefore, by enabling the application to acquire the necessary data as needed, smoother application behavior, appropriate behavior, and easier application design can be achieved. For example, as shown in Figures 7 and 8, in applications and application situations where the controller is grasped and operated, the AP execution unit 113 does not need to acquire mouse operation data or mouse data. On the other hand, even in similar situations, the AP execution unit 113 may acquire this data. For example, the AP execution unit 113 may acquire mouse operation data and monitor the horizontal mouse operation state included in the attribute data. When the application executed by the AP execution unit 113 determines that the mouse is in a horizontal mouse operation state, it may display a pointer or perform operations on objects, etc., based on mouse button data rather than pad data.

[0090] Next, we will explain the various types of data stored in memory. Figure 12 shows an example of data stored in the memory of the main unit 2. In Figure 12, the program storage area 301 and the data storage area 302 are shown separately for convenience. These do not need to be actually separated. Furthermore, more detailed storage areas or other storage areas may be conceptually provided. The program storage area 301 includes the OS program 401 and the AP program 402. The AP program 402 may include game application programs, communication application programs, etc.

[0091] The data storage area 302 includes OS data 405 generated and used by the OS program 401 and AP data 420 generated and used by the AP program 402. In this embodiment, the OS data 405 and AP data 420 are stored separately, but it is not necessary to store the OS data 405 and AP data 420 separately.

[0092] OS data 405 includes gamepad data 406, mouse data 407, and mouse operation data 408. These data are as described above.

[0093] AP data 420 includes object data 421, image data 422, virtual camera control data 423, pad data 424, mouse data 425, and mouse operation data 426.

[0094] Object data 421 is data for objects placed in the virtual space, such as player characters, opponent characters, the ground, menus, and cursors.

[0095] Image data 422 consists of image data such as animation images, backgrounds, and virtual effects.

[0096] The virtual camera control data 423 is data for controlling a virtual camera that is placed in a virtual space and takes pictures of that virtual space.

[0097] The pad data 424, mouse data 425, and mouse operation data 426 are generated by the AP execution unit 113 based on the pad data 406, mouse data 407, and mouse operation data 408 acquired by the OS execution unit. These do not need to be strictly identical.

[0098] The above data is merely an example. Other data may be included, and some data may be omitted. Also, for example, if the AP execution unit 113 generates new data based on the mouse operation data 426, that data may be included in the AP data 420. Thus, the various data do not need to be present at all times and may be added, deleted, modified, etc.

[0099] Figure 13 is an example of a sequence diagram showing the processing according to this embodiment. Some of the processing according to this embodiment will be described below. Note that the processing may include other processing, or some of the processing may be omitted. Also, the order of each processing is just an example, and for example, each processing may be executed simultaneously or in reverse order. In addition, each processing is described separately for convenience, but it may be an integrated processing. Furthermore, the following processing may be executed at predetermined intervals (for example, every processing frame, every 1 / 30 second).

[0100] First, the OS execution unit 110 assigns a handle value to each controller connected to the main unit 2 (S100). One handle value may be assigned to each controller, or, for example, a single controller may be assigned separate handle values ​​for a bundle corresponding to mouse operation data and mouse data. Also, for example, a controller without mouse functionality may be assigned a handle value corresponding to pad data, but not handle values ​​corresponding to mouse operation data or mouse data. The handle values ​​may be numbers that increase in the order in which they are connected to the main unit 2. Even for the same controller, if it is disconnected from the main unit 2 and then reconnected, a different handle value may be assigned.

[0101] The AP execution unit 113 requests the OS execution unit 110 to initialize the mouse operation data (S110). At this time, the AP execution unit 113 may provide arguments to specify the range of x and y coordinates. For example, if the AP execution unit 113 is drawing an image of the running application at a resolution of 1080p, the width of the x coordinate may be specified as 1920 and the width of the y coordinate as 1080. This allows the mouse operation data to specify x and y coordinates in the same 1080p coordinate system. Note that initialization of pad data and mouse data may also be requested.

[0102] The AP execution unit 113 requests a handle value from the OS execution unit 110 (S120), and the OS execution unit 110 passes the handle value to the AP execution unit 113 (S130). The handle values ​​may be requested together or individually. For example, the handle value used when requesting mouse operation data and the handle value used when requesting mouse data may be requested separately.

[0103] The AP execution unit 113 requests the OS execution unit 110 to set mouse operation data (S140), and the OS execution unit 110 performs the setting (S150). As an example, the mapping between each button type and the button may be set in the mouse button data. As another example, the width of the x and y coordinates may be set. Note that the setting may be performed for each type of controller. For example, the setting of mouse button data for the right controller 3 and the setting of mouse button data for the left controller 4 may be performed separately. In this case, even if the handle values ​​are different, if the controller type is the same, the same mouse operation button data setting may be applied. Note that the setting of mouse button data may be performed for each handle value.

[0104] The AP execution unit 113 requests mouse operation data from the OS execution unit 110 (S160). At this time, the acquired handle value may be used to specify which handle value corresponds to which mouse operation data to request from the OS execution unit 110.

[0105] The OS execution unit 110 generates mouse operation data (S170). Note that, as with other processes, this generation may be performed at a different time. For example, a continuous generation process may be performed. Alternatively, for example, individual data included in the mouse operation data may be generated continuously, and then combined into a single data set in response to a request for mouse operation data from the AP execution unit 113.

[0106] The OS execution unit 110 passes mouse operation data to the AP execution unit 113.

[0107] The AP control unit 113 can also request and obtain pad data and mouse data from the OS execution unit 110 (S200, 210, 220, 300, 310, 320). The timing of generating each data is not limited, similar to the generation process for mouse operation data and other processes.

[0108] As already mentioned, for each piece of data, separate APIs may be used for processing such as data acquisition. For example, initialization, handle requests, settings, and data requests related to mouse operation data may be processed via an API related to mouse operation data. Note that the term API may at least include, for example, the execution of processing that includes a predetermined response to a predetermined request in accordance with predetermined rules, but does not exclude other meanings. The term API is not required to be used in the system or design.

[0109] Figure 14 shows a functional block diagram in an example of another information processing system. The main unit 102 is, for example, a general-purpose personal computer. The main unit 102 includes an OS execution unit 110 that runs the OS. The main unit 102 also includes a data generation AP execution unit 1111 and an AP execution unit 1113. The OS execution unit 1110 acquires right controller data from, for example, a right controller 3 connected to the main unit. The data generation AP execution unit 1111 executes a data generation application that generates pad data, mouse operation data, and mouse data based on some or all of the right controller data acquired from the OS execution unit 1110. Note that the mouse operation data may be generated directly from the right controller data as shown in the figure, or it may be generated based on pad data and mouse data as in the embodiment described above. The AP execution unit 1113 then acquires some or all of this data. Thus, the data generation AP execution unit 1111 may be provided separately from the application execution unit 1113 that runs, for example, a game application. The application such as a game application and the data generation application may be independent applications. Furthermore, a data generation application or data generation function may be installed on the platform application used to run applications such as game applications. Also, each type of data may be generated by a different application, or each of the multiple steps in the generation process may be executed by a different application. In addition, the AP execution unit may directly acquire the data output from the controller without going through the OS execution unit. Furthermore, a game application executed by the AP execution unit may generate data equivalent to, for example, mouse operation data. As an example, a game application executed by the AP execution unit may generate wheel data based on the position of the stick.

[0110] Figure 15 shows another example of the first region where the wheel data is negative, or the second region where it is positive. In the following examples, the right controller is used as an example, but the same applies to the left controller. In this example, the first region includes the backward and upward directions from the initial position, and the second region includes the forward and downward directions from the initial position. In this case, the region for the left controller may be set as shown in Figure 10B. That is, the positive or negative values ​​may differ for the left and right controllers when the 2D position of the stick is forward or backward. The left controller may also be set so that, similar to the right controller in Figure 15, the first region includes the backward and forward directions from the initial position, and the second region includes the forward and downward directions from the initial position.

[0111] Figure 16 shows another example. As shown in Figure 16, a fifth region that does not belong to either the first or second region may be set between the first and second regions. When the two-dimensional position of the stick is within the fifth region, the wheel data does not need to indicate whether it is positive or negative. In this case, the wheel data does not need to indicate the amount of wheel rotation. The fifth region is set symmetrically around the initial position of the stick. The central angle of each fifth region may be, for example, 5 to 60 degrees or 20 to 50 degrees.

[0112] Figure 17 shows another example. As shown in Figure 17, multiple first and second regions may be provided. In Figure 17, one first region includes the direction downward from the initial position, and the other first region includes the direction backward from the initial position. A fifth region may be set between these first regions. The same applies to the second region. A fifth region may or may not be set between the first and second regions.

[0113] Figure 18 shows another example. As shown in Figure 18, the wheel data has a data structure that includes information indicating whether the stick's 2D position is within a first region or a second region, and information indicating whether it is within a third region or a fourth region. As an example, the wheel data has a data structure that includes wheel data 1 and wheel data 2. Wheel data 1 may indicate a negative value if the stick is in the first region, a positive value if it is in the second region, and a positive or negative value if it is neither. Wheel data 2 may indicate a negative value if the stick is in the third region, a positive value if it is in the fourth region, and a positive or negative value if it is neither. The first region includes positions below the initial position but does not include positions in the front-to-back direction, the second region includes positions above the initial position but does not include positions in the front-to-back direction, the third region includes positions behind the initial position but does not include positions in the up-and-down direction, and the fourth region includes positions in front of the initial position but does not include positions in the up-and-down direction. Each region may be, for example, a fan shape with a central angle of 90 degrees, extending 45 degrees to the left and right from each central direction. In this case, the boundaries of each region may or may not overlap. The AP execution unit can decide which region's wheel data to use depending on the processing of the application being executed. For example, the AP execution unit may use wheel data 1 in the above example when moving an object such as a bar that moves vertically on the screen, and use wheel data 2 in the above example when moving an object that moves horizontally. Note that whether these objects are moved vertically or horizontally is not limited by whether the wheel data indicates a positive or negative value.

[0114] Figure 19 shows an example where different regions are defined than in the example shown in Figure 18. Each region is semicircular. For example, the first region includes the area below the initial position. Specifically, it includes the area below the boundary extending in the front-to-back direction through the initial position. The second region is on the opposite side of the first region. The third region includes the area behind the initial position. Specifically, it includes the area behind the boundary extending in the up-and-down direction through the initial position. The fourth region is on the opposite side of the third region. As shown in Figure 19, the third region overlaps with the first and second regions, and the fourth region overlaps with the first and second regions. For example, when the 2D position of the stick is 45 degrees forward and downward from the initial position, wheel data 1 may show a negative value and wheel data 2 may show a positive value. In this way, regions may overlap between the data included in the wheel data.

[0115] In the example above, the wheel data includes wheel data 1 and wheel data 2, but these may be independent data. Furthermore, these may be independent of the mouse operation data. The AP execution unit may request wheel data independently of the mouse operation data, or it may request wheel data 1 and wheel data 2 independently, for example. Also, the wheel data may include, for example, two or more sets of wheel data, such as the wheel data corresponding to the example shown in Figure 19 and the wheel data corresponding to the example shown in Figure 10.

[0116] The regions shown in Figures 10 and 15-19 are examples, and the definition of the regions is not limited. For example, the shape of the region is not limited to a partial circle. For example, the sizes of the first and second regions may be different. For example, the sizes of the first and third regions may be different.

[0117] The main unit (for example, main unit 2, 102) may be able to connect to a general-purpose mouse. Even when a general-purpose mouse is connected, mouse operation data may be generated based on the output data from that mouse. For example, the AP execution unit may use a mouse operation data acquisition API. For example, the AP execution unit may acquire mouse operation data based on the output data from the connected mouse by specifying the handle assigned to the mouse or a controller type indicating "general-purpose mouse".

[0118] When a general-purpose mouse is connected to the main unit, the PrimaryButton in the mouse button data may be assigned by default to buttons 1 and 2 (generally corresponding to the left-click and right-click buttons) of the buttons 1 to 5 that a general-purpose mouse typically has. In the above embodiment, an example was shown in which the R button 20 and ZR button 21 of the right controller 3 are assigned to PrimaryButton by default. However, since the arrangement of these buttons when operating the right controller 3 with a mouse is similar to the arrangement of buttons 1 and 2 on a general-purpose mouse, such a default setting can result in a user experience similar to that when operating with the right controller 3 or left controller 4, even when using a general-purpose mouse. Note that buttons 1 to 5 on a general-purpose mouse may also be assigned by default to button types PrimaryButton to Button5, respectively.

[0119] Furthermore, for example, wheel data may correspond to operations performed on the mouse wheel of the mouse in question. Also, for example, with respect to attribute data, it may indicate that the mouse operation state on the horizontal plane is not always the case, and data indicating such a state may not be included. In any case, the content of each data may be changed as appropriate.

[0120] In this way, the AP execution unit can acquire data as the same mouse operation data not only when a right controller or left controller is connected, but also when a general-purpose mouse is connected, making it easier to perform appropriate processing and reducing the development burden on application developers. Note that information indicating the controller type may be included when requesting mouse operation data, or information indicating the controller type may be included in the response.

[0121] Furthermore, if a general-purpose mouse is connected to the main unit, the OS execution unit may generate general-purpose mouse data based on the output data from the mouse, and the AP execution unit may request this data. For example, the AP execution unit may use a general-purpose mouse data acquisition API. The general-purpose mouse data may have a data structure that includes information such as dx / dy data, x / y coordinate data, and button data. The dx / dy data may correspond to, for example, the dy / dz data in the mouse data. The AP execution unit may acquire either the mouse operation data or the general-purpose mouse data, or it may acquire both. As an example, if the AP execution unit requests mouse operation data settings and sets a range for x / y coordinates, the x / y coordinate data included in the general-purpose mouse data and the x / y coordinate data included in the mouse operation data may represent different coordinates. That is, the range of the x / y coordinate data included in the general-purpose mouse data may not be set by the AP execution unit, for example. Also, for example, if the speed of the mouse pointer can be set as a main unit setting, the x / y coordinate data included in the mouse operation data may reflect that speed. Furthermore, the button data included in the general-purpose mouse data may indicate buttons 1 to 5. In this way, the AP execution unit can acquire mouse operations and button operations of the general-purpose mouse based on the data format of the general-purpose mouse data, or it may acquire them based on the data format of the mouse operation data.

[0122] The AP execution unit may be capable of acquiring merged mouse operation data based on input data from multiple controllers. For example, the AP execution unit may specify "right controller" and "left controller" as controller types. The main unit may then generate mouse operation data based on the right controller data from the right controller and the left controller data from the left controller. For example, x / y coordinate data corresponding to the dx / dy data obtained by adding the dx / dy data of the right controller and the dx / dy data of the left controller may be calculated. Also, for example, if mouse button data is set for both the right controller and the left controller, the merged mouse operation data may indicate that the PrimaryButton of either controller has been pressed.

[0123] Merged mouse operation data may be generated based on the input data of all controllers of the specified controller type. That is, if two right controllers and two left controllers are connected to the main unit, and the right controllers and left controllers are specified as controller types, the data may be generated based on the input data of these four controllers. Alternatively, if only right controllers are specified as types, the data may be generated based on the input data of the two right controllers. The controller type may include a type indicating a general-purpose mouse. Regarding the generation of merged mouse operation data, instead of specifying the controller type, a handle value may be specified, other values ​​may be specified, or nothing may be specified, and the data may be generated based on the input data of all connected controllers.

[0124] Merged mouse operation data can be generated in any way; for example, it may be generated directly from the input data from each controller, or it may be generated using the mouse operation data corresponding to each controller after that data has been generated separately.

[0125] As repeatedly stated, each data point is merely an example, its content is not limited, and other data may be generated or acquired. For example, mouse operation data may include the elapsed time since the last sampling. For example, the elapsed time may be longer if packet loss occurs between the controller and the main unit. Also, for example, mouse operation data may include the sampling count. For example, if the update frequency of mouse operation data in the main unit is greater than the frequency with which the AP execution unit requests mouse operation data from the OS execution unit, the AP execution unit can determine how many times the mouse operation data has been updated between the last acquisition and the current acquisition by checking this sampling count. Note that this data may also be included in mouse data and pad data.

[0126] The right controller 3, the left controller 4, and a general-purpose mouse are examples of input devices. The input devices do not necessarily have to be detachable from the main unit 2, do not necessarily have a mouse sensor, do not necessarily have a stick, and may be connectable to devices other than the main unit 2. For example, the right controller 3 may have a wheel. The wheel may be located, for example, between the R button 20 and the ZR button 21. The R button 20 may also have a wheel function. The same applies to the left controller 4. When a controller has both a wheel and a stick, the wheel data included in the mouse operation data may be based on operations on either one or both.

[0127] For example, while mouse operation data such as mouse button data and state data were shown, the OS execution unit does not have to return all of this data in response to a request for mouse operation data from the AP execution unit. For example, it may return only the data that it was able to acquire or generate.

[0128] Even when using the same terminology to describe data, each piece of data does not need to be exactly the same. At the very least, if one piece of data and another substantially convey a certain piece of information, they can be considered the same piece of data.

[0129] The names of the data do not limit the scope of what the data represents. For example, mouse operation data does not mean that it is data acquired only when the AP execution unit runs an application that uses mouse operations. Also, for example, mouse operation data may be generated based on input data from a controller that does not have mouse functionality. The same applies to the names of other elements.

[0130] The generation of each data point may be performed by the input device.

[0131] Furthermore, a game system is an example of an information processing system, and an information processing system may be a system in which a game is not executed.

[0132] For example, when we say "processor of an information processing system," the term "processor" could mean one or more processors within a single device, such as the main unit, or it could mean some or all of the one or more processors present in each of multiple devices, such as the main unit and a controller, or the main unit, a controller, and a server. The same applies when we say "memory of an information processing system" or other configurations.

[0133] The program that causes a computer to perform each process may be a single program or a group of programs. "A certain program" does not necessarily mean a single program; it can include a group of programs. Furthermore, programs do not necessarily have to be stored in a single device. "A certain program" may, for example, refer to the totality of the individual programs stored in multiple devices within an information processing system.

[0134] At least a portion of the series of processes described above may be executed by the server-side device in an information processing system that includes a terminal-side device and a server-side device that can communicate via a network. The server may consist of multiple information processing devices, and the processing may be divided and executed by these multiple devices.

[0135] Although this embodiment and its variations have been described above, these descriptions are merely illustrative in every respect and are not intended to limit its scope. Furthermore, it goes without saying that various improvements and modifications can be made to this embodiment and its variations. [Explanation of Symbols]

[0136] 1. Information Processing System 2.102 Main Unit 3, 4 Controllers 24, 44 Mouse Sensors

Claims

1. On the computer, The system acquires first input data corresponding to the operation on the first operation unit of the first input device and second input data corresponding to the output of the mouse sensor of the first input device. In response to a first request from a game application, a first response having a first data structure is generated. In response to a second request from the game application, a second response having a second data structure is generated. The first response includes first data generated based on the first input data, The second response is a program that includes a second data generated based on the first input data and a third data generated based on the second input data.

2. The first data indicates that the first item is one of at least two values. The program according to claim 1, wherein the second data indicates that the second item is one of at least two values.

3. The program according to claim 2, wherein the first data and the second data are generated based on the same first input data.

4. The system acquires third input data corresponding to the operation of the second operation unit of the first input device. In response to the first request, the system provides a first response including the first data and a fourth data generated based on the third input data. The program according to claim 1, which, in response to the second request, causes a second response including the second data generated based on the first input data or the third input data.

5. The program according to claim 1, which modifies second data generated based on the first input data in response to a third request from a game application.

6. The second data described above indicates that at least one item is one of at least two values. The program according to claim 4, which, in response to a third request from a game application, sets which of the at least one of the items will have its value changed based on input data corresponding to an operation on any of the operation units.

7. The system acquires fourth input data corresponding to the operation of the third operation unit of the first input device. The first response includes a fifth data generated based on the fourth input data, The program according to claim 6, wherein the third request causes the program to set which of the at least one of the items to change the value of based on input data corresponding to an operation on any of the operation units except the third operation unit.

8. The first operating unit is a stick, and the first input data indicates the two-dimensional position of the stick. The first data indicates the two-dimensional position, The program according to claim 1, wherein the second data represents at least one of two values.

9. The program according to claim 1, which, in response to a fourth request from a game application, causes a third response having a third data structure including a sixth data generated based on the second input data.

10. The program according to claim 9, wherein the sixth data represents a two-dimensional change and the third data represents two-dimensional coordinates.

11. The system acquires fifth input data corresponding to the operation of the fourth control unit of the second input device and sixth input data corresponding to the output of the mouse sensor of the second input device. The program according to claim 1, which, in response to a fifth request from a game application, causes a fourth response having a fourth data structure including a seventh data generated based on the first input data and the fifth input data, and an eighth data generated based on the second input data and the sixth input data.

12. The program according to claim 1, wherein the second response includes ninth data indicating whether the game application that made the request for the second request is the target of the mouse operation.

13. The program according to claim 1, wherein the second response includes a tenth data indicating that the first input device is being operated with the opening that exposes the mouse sensor facing the ground.

14. First input data corresponding to the operation on the first operation unit of the first input device and second input data corresponding to the output of the mouse sensor of the first input device are acquired. In response to a first request from a game application, a first response having a first data structure is provided. In response to a second request from the game application, a second response having a second data structure is provided. The first response includes first data generated based on the first input data, The second response is an information processing method comprising a second data generated based on the first input data and a third data generated based on the second input data.

15. An information processing system including at least one processor, First input data corresponding to the operation on the first operation unit of the first input device and second input data corresponding to the output of the mouse sensor of the first input device are acquired. In response to a first request from a game application, a first response having a first data structure is provided. In response to a second request from the game application, a second response having a second data structure is provided. The first response includes first data generated based on the first input data, The second response is an information processing system comprising a second data generated based on the first input data and a third data generated based on the second input data.

Images