Information processing device, information processing method, and program

Abstract

To provide a technique that enables more appropriate control of objects displayed on a display device using a controller.SOLUTION: An information processing device disclosed herein comprises acquisition means for acquiring operation information of a two-dimensional operation on a controller, and control means configured to control objects displayed on a display device based on the operation information and a correspondence relationship between an operation coordinate system defining a direction of the two-dimensional operation on the controller and a display coordinate system defining the display position on the display devoice.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is an xR system that can draw a virtual object on a display device and control the virtual object using a controller. Such an xR system may move a virtual object in applications such as games and simulations. In this case, the xR system detects a user's operation using a device provided in the controller and controls the virtual object in response to the user's operation.

[0003] In order to reflect a user's operation in the control of a virtual object as intended by the user, it may be necessary to adjust the controller to a specific orientation before operating the controller. However, depending on the shape of the controller, it may be difficult for the user to recognize the orientation of the controller. In addition, when the display device is attached to the user's head and the user's field of vision is obstructed, it becomes even more difficult for the user to recognize the orientation of the controller.

[0004] Therefore, Patent Document 1 describes a technology in which the position of the controller is identified by detecting an externally installed marker with the controller's camera, and a virtual object is moved based on the position information and the user's operation. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2007-80002 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, in order to identify the position of the controller, the marker must be detected by the camera of the controller, and the controller must be pointed in the direction of the marker. Therefore, if the controller and the marker are blocked by an obstacle, the position of the controller cannot be identified, and the virtual object on the display device cannot be appropriately controlled.

[0007] An object of the present invention is to provide a technique that enables more appropriate control of an object displayed on a display device using a controller. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: An acquisition means for acquiring operation information of a two-dimensional operation on a controller; a control means for controlling an object displayed on the display device based on a correspondence relationship between an operation coordinate system that defines an operation direction of the two-dimensional operation on the controller and a display coordinate system that defines a display position of the display device, and based on the operation information; The information processing device is characterized by having:

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: An acquisition means for acquiring operation information of a two-dimensional operation on a controller; A difference between movement information of the controller from a reference state of the controller and movement information of the display device from a reference state of the display device and the operation information are used to display the difference. control means for controlling the presented object; The information processing device is characterized by having:

[0010] One aspect of the present invention is a method for producing a composition comprising the steps of: An acquisition step of acquiring operation information of a two-dimensional operation on a controller; a control step of controlling an object displayed on the display device based on a correspondence relationship between an operation coordinate system that defines an operation direction of the two-dimensional operation on the controller and a display coordinate system that defines a display position of the display device, and based on the operation information; The information processing method is characterized by having the following features.

[0011] One aspect of the present invention is a method for producing a composition comprising the steps of: An acquisition step of acquiring operation information of a two-dimensional operation on a controller; a control step of controlling an object displayed on the display device based on a difference between movement information of the controller from a reference state of the controller and movement information of the display device from a reference state of the display device, and based on the operation information; The information processing method is characterized by having the following features. Effect of the Invention

[0012] According to the present invention, it becomes possible to more appropriately control objects displayed on a display device using a controller. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a diagram illustrating the configuration of a display device. [Diagram 2] FIG. 2 is a diagram illustrating the configuration of a controller. [Diagram 3] FIG. [Figure 4] 3A and 3B are diagrams illustrating an operation coordinate system and a display coordinate system. [Diagram 5] 3A and 3B are diagrams illustrating an operation coordinate system and a display coordinate system. [Figure 6] 13 is a flowchart showing a process of a controller. [Figure 7] 13 is a flowchart showing a process of the display device. [Figure 8] 11A and 11B are diagrams illustrating an operation coordinate system and a display coordinate system when operation information is corrected. [Figure 9] FIG. 13 is a diagram illustrating a correction angle. [Figure 10] FIG. 13 is a diagram illustrating an operation coordinate change angle. [Figure 11] FIG. 13 is a diagram illustrating a display coordinate change angle. [Figure 12] FIG. 2 is a diagram illustrating an example of a basic coordinate system. [Figure 13] FIG. 2 is a diagram illustrating an example of a basic coordinate system. [Figure 14] FIG. 2 is a diagram illustrating an example of a basic coordinate system. [Figure 15] FIG. 2 is a diagram illustrating an example of a basic coordinate system. [Figure 16] FIG. 2 is a diagram illustrating an example of a basic coordinate system. [Figure 17] FIG. 2 is a diagram illustrating an example of a basic coordinate system. [Figure 18] FIG. 2 is a diagram illustrating an example of a basic coordinate system. [Figure 19] FIG. 2 is a diagram illustrating an example of a basic coordinate system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] 1 shows an example of the configuration of a display device 100. The display device 100 is, for example, a head mounted display (HMD) worn on the head of a user. The display device 100 includes a CPU 101, a GPU 102, a ROM 103, a RAM 104, a display unit 105, a communication unit 106, an inertial measurement unit 107, a geomagnetic sensor 108, and a system bus 109.

[0016] The CPU 101 is a control unit that controls each component connected to a system bus 109. The CPU 101 uses a RAM 104 as a work memory and executes a program stored in a ROM 103. In this way, the CPU 101 controls each component.

[0017] The GPU 102 is a graphics processing device that draws virtual objects in a virtual space. The programs stored in the ROM 103 include a program for implementing a correction process for operation information of the controller 200, which will be described later.

[0018] The display unit 105 is an output device (such as a liquid crystal panel or an organic EL panel) that displays rendered graphics.

[0019] The communication unit 106 is capable of performing communication via wired or wireless communication. The communication unit 106 communicates with a communication unit 205 of the controller 200 to transmit and receive data.

[0020] The inertial measurement unit 107 is a sensor that detects the attitude and movement of the display device 100. The geomagnetic sensor 108 is a sensor that detects the direction (orientation) of the display device 100.

[0021] The display device 100 may be composed of the display unit 105 and an information processing device (display control device) that controls the display unit 105. In this case, the information processing device has components other than the display unit 105 of the display device 100 (such as the CPU 101, the inertial measurement device 107, and the geomagnetic sensor 108).

[0022] Fig. 2 shows an example of the configuration of the controller 200. The controller 200 is, for example, a ring-shaped controller that can be worn on a user's finger as shown in Fig. 3. The controller 200 includes a CPU 201, a ROM 202, a RAM 203, an input device 204, a communication unit 205, an inertial measurement unit 206, a geomagnetic sensor 207, and a system bus 208.

[0023] The CPU 201 is a control unit that controls each component connected to a system bus 208. The CPU 201 uses a RAM 203 as a work memory and executes a program stored in a ROM 202. In this way, the CPU 201 controls each component.

[0024] The programs stored in the ROM 202 include a program for implementing a process of transmitting operation information and status information (information on posture, movement, or direction) of the controller 200 to the display device 100, which will be described later.

[0025] The input device 204 is a device that detects a user's operation. The input device 204 is provided, for example, on a side surface of the ring-shaped controller 200 as shown in FIG. 3. In the first embodiment, the input device 204 is a touchpad sensor (optical trackpad) that can sense a planar movement amount (a two-dimensional operation by the user). However, the input device 204 may be, for example, any operation member (such as a stick or a touch panel) that can receive a planar movement operation.

[0026] The communication unit 205 is capable of performing communication via wired or wireless communication. The communication unit 205 communicates with the communication unit 106 of the display device 100 to transmit and receive data.

[0027] The inertial measurement unit 206 is a sensor that detects the attitude and movement of the controller 200. The geomagnetic sensor 207 is a sensor that detects the direction (orientation) of the controller 200.

[0028] (Regarding operation coordinate system and display coordinate system) 4 and 5, the operation coordinate system C of the controller 200 and the display coordinate system D of the display device 100 will be described. The operation coordinate system C of the controller 200 is a two-dimensional coordinate system of the touch pad sensor of the controller 200, and is a coordinate system for determining the operation direction of the two-dimensional operation. The display coordinate system D of the display device 100 is a two-dimensional coordinate system for determining the display position on the display device 100 (display unit 105).

[0029] In the following, the orientation of the controller 200 is defined as an orientation 210 that is one of the directions parallel to the surface of the touch pad sensor of the input device 204, as shown in Fig. 3. The orientation of the display device 100 is defined as an orientation 211 toward the front surface (front direction) of the display device 100 (display unit 105), as shown in Fig. 4. In this embodiment, the orientation 210 corresponds to the Y-axis direction Cy of the operation coordinate system C, and the orientation 211 corresponds to the Y-axis direction Dy of the display coordinate system D.

[0030] 4 and 5 are diagrams expressing a composite space in which a virtual space represented by an image displayed on the display device 100 and a real space are combined. An example in which a user moves a virtual object displayed (drawn) on the display device 100 while wearing the display device 100 on his / her head and the controller 200 on his / her finger will be described with reference to FIGS. 4 and 5.

[0031] 4 shows a composite space in which both the display device 100 and the controller 200 face forward of the user. In the composite space (virtual space), a virtual object 300 exists at a position (in the air) at a moderate distance from the user's eyes.

[0032] 4, orientation 210 and orientation 211 are aligned, and the two axis directions (X-axis and Y-axis directions) of the operation coordinate system C are aligned with the two axis directions of the display coordinate system D. That is, the X-axis direction Cx of the operation coordinate system C is aligned with the X-axis direction Dx of the display coordinate system D, and the Y-axis direction Cy of the operation coordinate system C is aligned with the Y-axis direction Dy of the display coordinate system D.

[0033] The input device 204 of the controller 200 detects a user operation 400 (a two-dimensional operation in an operation coordinate system) for moving the virtual object 300 displayed on the display device 100. Then, the controller 200 transmits information on the user operation 400 (operation information) corresponding to the user sliding his / her finger forward to the display device 100.

[0034] The display device 100 calculates movement information 401 indicating a movement vector (movement amount and movement direction) of the virtual object 300 according to the received operation information of the user operation 400. For example, the display device 100 multiplies the X-axis direction component and the Y-axis direction component of the user operation 400 by a constant. Then, the display device 100 calculates movement information 401 indicating a vector in which the X-axis direction component and the Y-axis direction component multiplied by the constant are treated as the X-axis direction component and the Y-axis direction component in the display coordinate system. After that, the display device 100 moves the virtual object 300 to the movement target position 301 according to the movement information 401. In this case, as described above, since the two axial directions in the operation coordinate system C and the display coordinate system D are the same, the virtual object 300 moves in the direction in which the user actually performs the sliding operation.

[0035] FIG. 5 shows a composite space in a case where the controller 200 faces in a different direction from the display device 100 because the user unconsciously grips his / her hand tightly, for example.

[0036] 5, a virtual object 302 exists at a position (in the air) at a moderate distance from the user's eyes. At this time, the orientation 210 and the orientation 211 do not match, and the two axis directions (X-axis direction and Y-axis direction) of the operation coordinate system C of the controller 200 and the two axis directions of the display coordinate system D of the display device 100 do not match.

[0037] The input device 204 of the controller 200 detects a user operation 400, which is a two-dimensional operation for moving a virtual object 300 displayed on the display device 100. Then, the controller 200 transmits operation information of a user operation 402 corresponding to the user sliding his / her finger forward to the display device 100. The display device 100 calculates movement information 403 of the virtual object in accordance with the operation information of the user operation 402. Here, since the two axial directions of the operation coordinate system C and the two axial directions of the display coordinate system D do not match, the movement information 403 is calculated to execute a movement different from the user's intention (intention to move the virtual object 300 forward). As a result, the virtual object 302 moves to a movement target position 303 different from the movement target position 304 intended by the user.

[0038] Therefore, in the first embodiment, the display device 100 corrects operation information of the user operation 402 in accordance with the correspondence between the operation coordinate system C and the display coordinate system D, and calculates movement information 404 based on the corrected operation information. Then, the display device 100 moves the virtual object 300 to an appropriate movement target position 304 in accordance with the movement information 404.

[0039] Therefore, hereinafter, a process of correcting information on a user operation according to the correspondence between the operation coordinate system C and the display coordinate system D will be described. The correspondence between the operation coordinate system C and the display coordinate system D is a relationship indicated by the difference between the orientation of the operation coordinate system C (the orientation of the reference direction (e.g., the Y-axis direction) of the operation coordinate system C) and the orientation of the display coordinate system D (the orientation of the reference direction (e.g., the Y-axis direction) of the display coordinate system D).

[0040] (Controller processing) FIG. 6 is a flowchart showing the process of the controller 200 for moving a virtual object to a position that matches the user's intention.

[0041] In step S600, the CPU 201 determines whether or not a two-dimensional operation (e.g., a slide operation) by the user is detected as a user operation by the input device 204. If it is determined that a user operation is detected, the process proceeds to step S601. If it is determined that a user operation is not detected, the process of step S600 is repeated.

[0042] In step S601, the CPU 201 obtains operation information of a user operation in the operation coordinate system C (operation vector; operation amount and operation direction).

[0043] In step S602, the CPU 201 acquires information on the orientation of the controller 200 in the reference coordinate system G (hereinafter referred to as "controller orientation information"). The reference coordinate system G will be described in detail later, but for example, the reference coordinate system G is a coordinate system based on the measurement values ​​of the geomagnetic sensors 108 and 207. Specifically, the reference coordinate system G is a coordinate system in which the east-west direction is the X-axis direction and the north-south direction is the Y-axis direction, for example.

[0044] In step S603, the CPU 201 controls the communication unit 205 to transmit the controller direction information and the operation information to the display device 100.

[0045] (Display Device Processing) The process of the display device 100 for moving a virtual object to a position that matches the user's intention will be described with reference to the flowchart in FIG. 7 and FIGS.

[0046] In step S700, the CPU 101 displays on the display unit 105 a virtual object that is to be operated by the user.

[0047] In step S701, the CPU 101 receives the command from the controller 200 via the communication unit 106. It is determined whether data has been received from the controller 200. If it is determined that data has been received from the controller 200, the process proceeds to step S702. If it is determined that data has not been received from the controller 200, the process of step S701 is repeated.

[0048] In step S702, the CPU 101 acquires operation information and controller direction information from the controller 200 from the data received in step S701.

[0049] In step S703, the CPU 101 acquires information on the orientation of the display device 100 in the reference coordinate system G (hereinafter, referred to as "display orientation information").

[0050] In step S704, the CPU 101 calculates (determines) a correction angle θ indicating the difference between the controller direction information and the display direction information. The correction angle θ is an angle indicating the difference between the orientation of the operation coordinate system C and the orientation of the display coordinate system D (i.e., the correspondence between the operation coordinate system C and the display coordinate system D). The CPU 101 then corrects the operation information based on the correction angle θ. Specifically, the CPU 101 corrects the operation information by adjusting the direction of the vector indicated by the operation information by the correction angle θ. The CPU 101 then determines the movement information based on the corrected operation information.

[0051] In step S705, CPU 101 moves the virtual object displayed on display unit 105 in accordance with the movement information. In addition to moving the virtual object, CPU 101 may also perform other control of the display of display device 100 in accordance with the movement information (for example, control the size of the virtual object, or draw a new virtual object in a virtual space).

[0052] Fig. 8 is a diagram for explaining a case where the virtual object 305 is moved according to the flowchart of Fig. 7. The virtual object 305 exists at a position (in the air) an appropriate distance away from the user's eyes. At this time, the display device 100 and the controller 200 face in different directions. In other words, the two axis directions (orientations) of the operation coordinate system C and the two axis directions (orientations) of the display coordinate system D do not match.

[0053] The input device 204 detects a user operation 405 for moving a virtual object 305 displayed on the display device 100. The controller 200 transmits operation information of the user operation 405 to the display device 100 via the communication unit 205. The display device 100 corrects the operation information based on a correction angle θ indicating the correspondence between the operation coordinate system C and the display coordinate system D, and calculates movement information 406 of the virtual object based on the corrected operation information. This allows the display device 100 to move the virtual object 305 to an appropriate movement target position 306 based on the movement information 406.

[0054] (Another method for calculating the correction angle) Although the method for calculating the correction angle θ indicating the difference between the controller direction information and the display direction information has been described above, other methods can be used, and these methods will be described below.

[0055] FIG. 9 shows a correction angle θ indicating the difference between the orientation of the operation coordinate system C and the orientation of the display coordinate system D (the difference between the X-axis directions or the Y-axis directions). FIG. 10 shows an operation coordinate change angle θ' indicating the difference between the orientation of the operation coordinate system C and the orientation of the reference coordinate system G. FIG. 11 shows a display coordinate change angle θ'', which is the difference between the orientation of the display coordinate system D and the orientation of the reference coordinate system G.

[0056] Therefore, the correction angle θ is calculated by subtracting the operation coordinate change angle θ' from the display coordinate change angle θ' as shown in the following formula 1. 9, the orientation of the operation coordinate system C and the orientation of the display coordinate system D are tilted in the same direction as seen from the orientation of the reference coordinate system G, so the correction angle θ can be calculated using Equation 1. On the other hand, if the orientation of the operation coordinate system C and the orientation of the display coordinate system D are tilted in different directions as seen from the orientation of the reference coordinate system G, the correction angle θ can be calculated by adding the operation coordinate change angle θ' and the display coordinate change angle θ''. θ=θ'-θ'' (1)

[0057] Therefore, in S704 of the flowchart in FIG. 7, the CPU 101 may calculate an operation coordinate change angle θ' (a difference between the orientation of the reference coordinate system G and the orientation of the operation coordinate system C) and a display coordinate change angle θ'' (a difference between the orientation of the reference coordinate system G and the orientation of the display coordinate system D). Then, the CPU 101 may calculate a correction angle θ indicating a difference between the orientation of the operation coordinate system C and the orientation of the display coordinate system D from the operation coordinate change angle θ' and the display coordinate change angle θ''. The CPU 101 corrects operation information of the user operation in accordance with the correction angle θ, and determines movement information of the virtual object.

[0058] CPU 101 may determine a correction angle θ indicating a difference between the orientation of display device 100 and the orientation of controller 200 based on a captured image of controller 200 captured by a camera of display device 100.

[0059] (Basic coordinate system example 1) 12 and 13, an example of the reference coordinate system G according to the measurement values ​​from the geomagnetic sensor 207 and the geomagnetic sensor 108 will be described. For example, as shown in Fig. 12, the CPU 201 sets the east direction (east-west direction) as the X-axis direction Gx of the reference coordinate system G and the north direction (north-south direction) as the Y-axis direction Gy of the reference coordinate system G based on the direction value (measurement value) obtained from the geomagnetic sensor 207. This allows the CPU 201 to calculate controller direction information that indicates the orientation of the controller 200 in the reference coordinate system G.

[0060] 13, based on the direction value obtained from the geomagnetic sensor 108, the CPU 101 sets the east direction as the X-axis direction Gx of the reference coordinate system G and the north direction as the Y-axis direction Gy of the reference coordinate system G. This allows the CPU 101 to calculate display direction information indicating the orientation of the display device 100 in the reference coordinate system G.

[0061] (Basic coordinate system example 2) 14 and 15, an example of the reference coordinate system G according to the measurement values ​​by the inertial measurement unit 206 and the inertial measurement unit 107 will be described. For example, as shown in FIG. 14, the CPU 101 displays UIs (UI501 and UI502) for placing the display device 100 and the controller 200 in a specific positional relationship on the display surface 500 of the display unit 105. This causes the CPU 101 to prompt the user to move the controller 200 to the position indicated by the UI 501. The user moves the position of the controller 200 by a movement operation 407 of the controller 200. The position indicated by the UI 501 is, for example, a position where it is assumed that the user can easily operate the controller 200.

[0062] When the controller 200 reaches the position of the UI 501, the CPU 101 determines that the display device 100 and the controller 200 are in a specific positional relationship, and issues a notification 503 as shown in FIG. 15 to the user. The notification 503 indicates to the user that the display device 100 and the controller 200 have reached a specific positional relationship. The CPU 101 sets the display coordinate system D of the display device 100 as a reference coordinate system G' and the operation coordinate system C of the controller 200 as a reference coordinate system G'', in a state in which the display device 100 and the controller 200 are in the specific positional relationship. Information on the set reference coordinate system G' and information on the reference coordinate system G'' are stored in the RAM 104.

[0063] Thereafter, the CPU 101 calculates the cumulative movement information (movement angle measurable using the inertial measurement unit 107) of the display device 100 from this state (reference state) as a display coordinate change angle θ''. Then, the CPU 201 calculates the cumulative movement information (movement angle measurable using the inertial measurement unit 206) of the controller 200 from this state (reference state) as an operation coordinate change angle θ'. Thereafter, the CPU 101 calculates a correction angle θ that indicates the correspondence between the operation coordinate system C and the display coordinate system D from the difference between the operation coordinate change angle θ' and the display coordinate change angle θ''.

[0064] (Basic coordinate system example 3) 16 to 19, an example of the reference coordinate system G according to the measurement values ​​by the inertial measurement unit 206 and the inertial measurement unit 107 will be described. FIG. 16 shows an example of moving a virtual object in a direction intended by a user according to a two-dimensional operation on the controller 200. The input device 204 detects a user operation 408 for moving a virtual object 307 displayed on the display device 100. The CPU 101 calculates movement information 409 of the virtual object 307 based on operation information of the user operation 408, and moves the virtual object 307 to a movement target position 308 of the virtual object.

[0065] Fig. 17 shows the correction angle θ in the state shown in Fig. 16. Fig. 18 shows the operation coordinate change angle θ' indicating the difference between the orientation of the operation coordinate system C and the orientation of the reference coordinate system G. At this time, for example, the X-axis direction Cx of the operation coordinate system C is parallel to the surface of the touchpad sensor and the horizontal direction is the Y-axis direction Cy of the operation coordinate system C. Then, by using the value of the inertial measurement device 206, the CPU 201 can determine the horizontal direction (such as the horizontal direction and the direction in front of the user) as the Y-axis direction Gy of the reference coordinate system G and the vertical direction as the Z-axis direction Gz of the reference coordinate system G.

[0066] FIG. 19 shows a display coordinate change angle θ'' indicating the difference between the orientation of the display coordinate system D and the orientation of the reference coordinate system G. At this time, the CPU 101 can set the horizontal direction as the Y-axis direction Gy of the reference coordinate system G and the vertical direction as the Z-axis direction Gz of the reference coordinate system G by using the value of the inertial measurement device 107. Even in such a case, the CPU 101 calculates a correction angle θ indicating the correspondence between the operation coordinate system C and the display coordinate system D from the difference between the operation coordinate change angle θ' and the display coordinate change angle θ'', and corrects the operation information based on the correction angle θ. For example, the CPU 101 rotates a vector indicated by the operation information around the X-axis (X-axis direction Cx) by the correction angle θ.

[0067] According to this embodiment, the movement of the virtual object is controlled according to the correspondence between the operation coordinate system C and the display coordinate system D. Therefore, the virtual object can be moved in a direction intended by the user (for example, the same direction as the direction actually operated by the user). In other words, the display of the display device can be more appropriately controlled using the controller.

[0068] Note that the display device 100 corrects the operation information based on the correction angle θ and calculates the movement information according to the corrected operation information. On the other hand, the display device 100 may calculate the movement information according to uncorrected operation information and then correct the movement information based on the correction angle θ.

[0069] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." For this reason, unless a contradiction arises, the expression "equal to or greater than A" may be read as "A or greater (high; long; many)," or may be read as "greater than A (high; long; many)." On the other hand, the expression "equal to or less than A" may be read as "A or smaller (low; short; few) than A," or may be read as "smaller than A (low; short; few)." They may be substituted or read differently. In addition, "bigger (higher; longer; more) than A" may be read as "A or more," and "smaller (lower; shorter; less) than A" may be read as "A or less."

[0070] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0071] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0072] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0073] The disclosure of the above embodiments includes the following configurations, methods, and programs. [Configuration 1] An acquisition means for acquiring operation information of a two-dimensional operation on a controller; a control means for controlling an object displayed on the display device based on a correspondence relationship between an operation coordinate system that defines an operation direction of the two-dimensional operation on the controller and a display coordinate system that defines a display position of the display device, and based on the operation information; 13. An information processing device comprising: [Configuration 2] the control means corrects the operation information based on a correspondence relationship between the operation coordinate system and the display coordinate system, and controls the object based on the corrected operation information. 2. The information processing device according to configuration 1. [Configuration 3] the correspondence relationship between the operation coordinate system and the display coordinate system is a relationship represented by a difference between an orientation of the operation coordinate system and an orientation of the display coordinate system; 3. The information processing device according to configuration 1 or 2. [Configuration 4] the control means determines a difference between an orientation of the operation coordinate system and an orientation of the display coordinate system from a difference between an orientation of the controller in a reference coordinate system and an orientation of the display device in the reference coordinate system. 4. The information processing device according to configuration 3. [Configuration 5] the control means determines a difference between the orientation of the operation coordinate system and the orientation of the display coordinate system from a difference between an orientation of a reference coordinate system and an orientation of the operation coordinate system and a difference between the orientation of the reference coordinate system and the orientation of the display coordinate system. 4. The information processing device according to configuration 3. [Configuration 6] The reference coordinate system is a coordinate system according to a measurement value of a geomagnetic sensor. 6. The information processing device according to configuration 4 or 5. [Configuration 7] The reference coordinate system is a coordinate system according to measurements of an inertial measurement unit. 6. The information processing device according to configuration 4 or 5. [Configuration 8] An acquisition means for acquiring operation information of a two-dimensional operation on a controller; a control means for controlling an object displayed on the display device based on a difference between movement information of the controller from a reference state of the controller and movement information of the display device from a reference state of the display device, and based on the operation information; 13. An information processing device comprising: [Configuration 9] the control means controls the object by moving the object; 9. The information processing device according to any one of configurations 1 to 8. [Method 1] An acquisition step of acquiring operation information of a two-dimensional operation on a controller; a control step of controlling an object displayed on the display device based on a correspondence relationship between an operation coordinate system that defines an operation direction of the two-dimensional operation on the controller and a display coordinate system that defines a display position of the display device, and based on the operation information; 13. An information processing method comprising: [Method 2] An acquisition step of acquiring operation information of a two-dimensional operation on a controller; a control step of controlling an object displayed on the display device based on a difference between movement information of the controller from a reference state of the controller and movement information of the display device from a reference state of the display device, and based on the operation information; 13. An information processing method comprising: [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 9. [Explanation of symbols]

[0074] 100: display device, 200: controller, 101: CPU, 105: Display section

Claims

1. an acquisition means for acquiring operation information of a two-dimensional operation on a controller; a control means for controlling an object displayed on the display device based on a correspondence relationship between an operation coordinate system that determines the operation direction of the two-dimensional operation on the controller and a display coordinate system that determines the display position of the display device, and based on the operation information; An information processing device comprising:

2. The controller has an operation member capable of receiving a planar movement operation, the operation information is information about the two-dimensional operation performed on the operation member; 2. The information processing apparatus according to claim 1, wherein:

3. the control means corrects the operation information based on the correspondence between the operation coordinate system and the display coordinate system, and controls the object based on the corrected operation information.

2. The information processing apparatus according to claim 1, wherein:

4. the correspondence relationship between the operation coordinate system and the display coordinate system is a relationship indicated by a difference between the orientation of the operation coordinate system and the orientation of the display coordinate system; 2. The information processing apparatus according to claim 1, wherein:

5. the control means determines a difference between an orientation of the operation coordinate system and an orientation of the display coordinate system in accordance with a difference between an orientation of the controller in a reference coordinate system and an orientation of the display device in the reference coordinate system.

5. The information processing apparatus according to claim 4,

6. the control means determines a difference between the orientation of the operation coordinate system and the orientation of the display coordinate system in accordance with 1) a difference between the orientation of a reference coordinate system and the orientation of the operation coordinate system, and 2) a difference between the orientation of the reference coordinate system and the orientation of the display coordinate system.

5. The information processing apparatus according to claim 4,

7. the reference coordinate system is a coordinate system according to the measurement value of a geomagnetic sensor; 7. The information processing apparatus according to claim 5, wherein the information processing apparatus is a computer.

8. The reference coordinate system is a coordinate system according to measurements of an inertial measurement unit.

7. The information processing apparatus according to claim 5, wherein the information processing apparatus is a computer.

9. an acquisition means for acquiring operation information of a two-dimensional operation on a controller; a control means for controlling an object displayed on the display device based on a difference between movement information of the controller from a reference state of the controller and movement information of the display device from a reference state of the display device, and based on the operation information; An information processing device comprising:

10. The controller has an operation member capable of receiving a planar movement operation, the operation information is information about the two-dimensional operation performed on the operation member; 10. The information processing apparatus according to claim 9,

11. the control means controls the object by moving the object; 10. The information processing apparatus according to claim 1 or 9.

12. an acquisition step of acquiring operation information of a two-dimensional operation on a controller; a control step of controlling an object displayed on the display device based on a correspondence relationship between an operation coordinate system that determines an operation direction of the two-dimensional operation on the controller and a display coordinate system that determines a display position of the display device, and based on the operation information; An information processing method comprising:

13. an acquisition step of acquiring operation information of a two-dimensional operation on a controller; a control step of controlling an object displayed on the display device based on a difference between movement information of the controller from a reference state of the controller and movement information of the display device from a reference state of the display device, and the operation information; An information processing method comprising:

14. A program for causing a computer to function as each of the means of the information processing device according to claim 1 or 9.

Images