Information processing device, information processing system, information processing method, and program

Abstract

To set an operation coordinate system of a controller at a direction intended by a user while reducing burden on the user.SOLUTION: An information processing device includes: first acquisition means for acquiring first information related to a position or an attitude of a controller; second acquisition means for acquiring operation information about a two-dimensional operation from the controller; control means for executing processing based on the operation information about the two-dimensional operation; and changing means for changing a correspondence between the operation information about the two-dimensional operation and the processing executed by the control means on the basis of information related to a change in the first information in a predetermined period. The control means executes the processing based on the operation information about the two-dimensional operation on the basis of the correspondence changed by the changing means.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] In cross reality (XR) systems that allow users to experience virtual reality, hand controllers have been used to convert hand movements into actions in the virtual space when controlling the display of a head-mounted display (HMD). The HMD is a glasses-type device equipped with a small display that the user wears on his or her head.

[0003] Patent Document 1 proposes a method of adjusting the operation coordinate system of the controller in accordance with the orientation of the HMD and the orientation of the controller.

[0004] Moreover, Patent Document 2 proposes a method in which the user specifies the direction in which the controller is operated. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-129263 A [Patent Document 2] JP 2020-35293 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology disclosed in Patent Document 1, the coordinate axes of the controller are determined by the direction of the HMD and the direction of the controller, and there is a problem in that the determined coordinate system may differ from the user's intention.

[0007] Furthermore, the technique disclosed in Patent Document 2 requires the user to specify each of the operation directions of the controller, which imposes a heavy burden on the user.

[0008] Therefore, an object of the present invention is to provide a technique that enables the operation coordinate system of a controller to be set in a direction intended by the user while reducing the burden on the user. [Means for solving the problem]

[0009] One aspect of the present invention is an information processing device comprising: first acquisition means for acquiring first information regarding a position or posture of a controller; second acquisition means for acquiring operation information of a two-dimensional operation from the controller; control means for executing processing based on the operation information of the two-dimensional operation; and modification means for changing the correspondence between the operation information of the two-dimensional operation and the processing executed by the control means based on information regarding changes in the first information over a predetermined period of time, wherein the control means executes processing based on the operation information of the two-dimensional operation based on the correspondence changed by the modification means. Effect of the Invention

[0010] According to the present invention, it is possible to set the operation coordinate system of a controller in a direction intended by the user while reducing the burden on the user. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining the internal configuration of an HMD according to the first embodiment. [Diagram 3] 2 is a flowchart of a process according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 6] FIG. 3 is a diagram showing a coordinate system of a controller according to the first embodiment. [Figure 7] 4A to 4C are diagrams illustrating an example of movement of a controller according to the first embodiment. [Figure 8] 5A to 5C are diagrams illustrating an example of correction of a coordinate system of the controller according to the first embodiment. [Figure 9] 5A to 5C are diagrams illustrating an example of correction of a coordinate system of the controller according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 11] 13A to 13C are diagrams illustrating an example of movement of a controller according to the second embodiment. [Figure 12] 13A to 13C are diagrams illustrating an example of movement of a controller according to the second embodiment. [Figure 13] FIG. 11 is a diagram showing a coordinate system of a controller before correction according to the second embodiment. [Figure 14] FIG. 11 is a diagram showing a coordinate system of a controller after correction according to the second embodiment. [Figure 15] FIG. 11 is a diagram showing a coordinate system of a controller after correction according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] (Embodiment 1) An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, a PC (personal computer) 110, and a controller 120.

[0014] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user captured by the HMD 100 with content such as CG in a form that corresponds to the posture of the HMD 100.

[0015] The PC 110 controls the HMD 100. The PC 110 is connected to the HMD 100 by a wired connection such as a USB cable, or wirelessly by a connection such as Bluetooth (registered trademark) or Wi-Fi (Wireless Fidelity) (registered trademark). The PC 110 generates a composite image by combining a captured image with CG, and transmits the composite image to the HMD 100. Note that, although a PC will be described here as an example of an information processing device, the information processing device is not limited to this. For example, the information processing device may be a smartphone or a tablet terminal, and each component of the PC 110 may be included in the HMD 100.

[0016] The controller 120 performs various controls of the HMD 100. If the PC 110 is in a specific control mode, when a user operates the controller 120, the HMD 100 is controlled in response to the user's operation. As shown in FIG. 1, the controller 120 may be in the shape of a finger ring that can be worn and supported by the user, or in the shape of a handheld type that can be held in the hand. The controller 120 also has physical buttons for performing a decision operation or a selection operation on the display. The controller 120 performs wireless communication with the PC 110 via Bluetooth. The controller is not limited to a system that communicates with the PC 110, and may be a system that communicates with the HMD 100.

[0017] By moving the controller 120, the user can change the indicated position on the display according to the movement of the controller. The indicated position may be represented by a point, or may be represented by a virtual ray (ray) connecting the point of the indicated position and the controller with a straight line (line segment) or a dotted line. By pressing a physical button, a menu determination operation or a selection operation can be performed. Although the shape of the controller 120 is assumed to be a ring type or a hand-held type, it is not limited thereto as long as it can be supported by a finger, a hand, or an arm. Also, although the button is assumed to be a physical button, it may be operable like a track pad, a touch panel, a wheel, or a track ball, and in addition to pressing the button, a slide operation, a flick operation, or a touch operation may be used.

[0018] Note that the controller may be worn on at least one of a finger, a hand, or an arm.

[0019] Note that the controller may be attached to an object held by hand, and used to acquire position information and attitude information of the attached position from a sensor. Examples of such an object include an object imitating a tool.

[0020] <Internal Configuration of HMD> Referring to FIG. 2, the internal configuration of the HMD 100 will be described. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, an attitude sensor unit 204, a non-volatile memory 205, a working memory 206, and an imaging unit 207 for the line of sight.

[0021] The HMD control unit 201 is a CPU that controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by synthesizing an imaging image captured by the imaging unit 202 of the space in front of the user and a CG) from the PC 110, the HMD control unit 201 displays the composite image on the image display unit 203. Note that instead of the HMD control unit 201 controlling the entire device, a plurality of hardware may share the processing to control the entire device.

[0022] The imaging unit 202 includes two cameras (imaging devices). The two cameras are for capturing captured images used for compositing with an image of a virtual space and generating position and orientation information, and have an imaging unit for a left eye and an imaging unit for a right eye. The imaging unit for the left eye captures a moving image of the real space corresponding to the left eye of the person wearing the HMD 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the left eye. The imaging unit for the right eye captures a moving image of the real space corresponding to the right eye of the person wearing the HMD 101, and an image (captured image) of each frame in the moving image is output from the imaging unit for the right eye. In other words, the imaging unit 202 acquires captured images as stereo images having parallax that approximately matches the positions of the left eye and right eye of the person wearing the HMD 101. By measuring distances with the stereo cameras, information on the distances from the two cameras to the subject can be acquired as distance information. Note that in an HMD for an MR system, it is preferable to arrange the imaging unit so that the central optical axis of the imaging range of the imaging unit approximately matches the line of sight direction of the person wearing the HMD.

[0023] Each of the left eye imaging section and the right eye imaging section has an optical system and an imaging device. Light entering from the outside world enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the entering light as a captured image. Images of a subject (a range in front of the user) captured by the two cameras are output to the PC 110 and the control unit 201. Note that the imaging unit 202 may capture and output a video instead of a captured image.

[0024] The image display unit 203 displays a composite image. The image display unit 203 has a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, the image display unit 203 is disposed in front of each eye of the user. Note that a device using a semi-transmissive half mirror can also be used for the image display unit 203. In this case, for example, the image display unit 203 may display an image by using a technology generally called AR (Augmented Reality) so that the CG is directly superimposed on the real space seen through the half mirror. The image display unit 203 may also display an image of a complete virtual space without using a captured image by using a technology generally called VR (Virtual Reality).

[0025] The attitude sensor unit 204 acquires attitude (and position) information of the HMD 100. The attitude sensor unit 204 may acquire attitude information of a user (a user wearing the HMD 100) corresponding to the attitude (and position) of the HMD 100. The attitude sensor unit 204 has an inertial measurement unit (IMU) composed of an acceleration sensor, an angular acceleration sensor, and a geomagnetic sensor. The attitude sensor unit 204 is used when acquiring information on the user's attitude (attitude information), and the HMD control unit 201 outputs the information on the user's attitude (attitude information) to the PC 110. The attitude information may be acquired from one or more of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor.

[0026] The HMD control unit 201 estimates the position or posture of each joint point of the user's hand and fingers from the two camera images obtained by the imaging unit 202. The joint points include points that are characteristic of parts such as the joints and fingertips of the fingers, the back of the hand (palm), and the arms. Each joint point indicates a coordinate position, and the posture can be estimated from information on a plurality of joint points. A known object recognition or pose estimation method of machine learning using a convolutional neural network can be used as a method for estimating the position or posture of the hand and each joint point of the hand. In addition, the position information of each joint point of the hand in the depth direction can be obtained by calculating the distance from the imaging unit 202 to each joint point by triangulation using stereo matching using the two camera images obtained by the imaging unit 202. The estimated coordinate information of each joint point of the hand is output from the control unit 201 to the PC 110.

[0027] The non-volatile memory 205 is an electrically erasable and recordable non-volatile memory, and stores programs executed by the control unit 211, which will be described later.

[0028] The working memory 206 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202, an image display memory for the image display unit 203, a working area for the control unit 201, and the like.

[0029] The gaze imaging unit 207 is a camera that acquires an image for detecting the gaze of the user, and is attached inside the HMD to capture an image of the user's eyes when the user wears the HMD 100. The image of the subject (user's eyes) captured by the camera is output to the control unit 211 of the PC 110 via the HMD control unit 201. The control unit 211 detects the gaze of the user wearing the HMD 100 from the image captured by the gaze imaging unit 207, and identifies the location on the image display unit 203 that the user is gazing at.

[0030] <Internal configuration of the controller> The internal configuration of the controller 120 will be described with reference to Fig. 2. The controller 120 has a controller control unit 221, a communication unit 222, a button operation unit 223, a touch sensor 224, an inertia detection unit 225, and a vibration unit 226.

[0031] The controller control unit 221 controls each component of the controller 120 .

[0032] The communication unit 222 performs wireless communication with the PC 110 (the communication unit 214).

[0033] The button operation unit 223 is a button that can be pressed.

[0034] The touch sensor 224 is a sensor that can detect the presence or absence of a touch state and the amount of planar movement, and is configured as an OTP in this embodiment. The touch sensor 224 may be configured to be built into the button operation unit 223. Information (operation information) regarding pressing a button or sliding a finger over the OTP is transmitted to the PC 110 via the controller control unit 221 and the communication unit 222.

[0035] The user can move the pointer displayed on the HMD 100 to a predetermined position by sliding the finger on the OTP. The user can also instruct the HMD 100 or the PC 110 to perform a specific process by pressing a button. In this way, the user can control the HMD 100 by a combination of sliding the finger on the OTP and pressing a button. More specifically, the user can control the HMD 100 to display a menu by pressing and holding a button, for example, and select a desired location by pressing the button after positioning the pointer thereon. The touch sensor 224 may be any sensor capable of sensing a planar amount of movement, and may include, instead of the OTP, any of a touch panel, a joystick, a trackball, and a trackpad device, for example.

[0036] The inertia detection unit 225 has an IMU (inertial measurement unit; inertial sensor) that can detect the inertial information (spatial movement amount and angle) of the controller 120. Note that the inertia detection unit 225 may have any device as long as it does not hinder the miniaturization of the controller 120 and can detect inertial information (such as information on displacement, velocity, or acceleration).

[0037] The vibration unit 226 is a device that vibrates itself and can give various sensations to the user by transmitting the vibration to the housing of the controller 120. Also, the vibration unit 226 vibrates at the vibration frequency and vibration intensity instructed by the controller control unit 221.

[0038] <Internal Configuration of the PC> Referring to FIG. 2, the internal configuration of the PC 110 will be described. The PC 110 has a control unit 211, a non-volatile memory 212, a working memory 213, a communication unit 214, and a recording medium 215.

[0039] The control unit 211 is a CPU that controls each part of the PC 110 according to the input signals and the programs described later. Note that instead of the control unit 211 controlling the entire device, a plurality of hardware components may share the processing to control the entire device. The control unit 211 receives the image (captured image) acquired by the imaging unit 202 and the attitude information acquired by the attitude sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel the aberration in the optical systems of the imaging unit 202 and the image display unit 203. Then, the control unit 211 synthesizes the captured image and arbitrary CG to generate a synthesized image. The control unit 211 transmits the synthesized image to the HMD control unit 201 in the HMD 100.

[0040] Also, the control unit 211 calculates the number of controllers included in the captured image. Also, using the information obtained via the communication unit 214, it executes processing for recognizing the attached positions of the respective controllers. Then, according to the recognized results, it performs control to change the operation content for the input information of each controller for each controller.

[0041] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 is closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG object that is not placed in real space appears as if it were placed in real space.

[0042] In addition, the control unit 211 receives information estimated by the control unit 201 of the HMD 100. The received information is temporarily stored in the working memory 213.

[0043] Furthermore, in the control unit 211, the communication unit 214 receives information on a change in the position or attitude of the controller 120 from the communication unit 223 of the controller 120. The control unit 211 superimposes on the combined image an indication position according to the information on the change in the position or attitude of the controller 120. Note that the control unit 211 may also superimpose on the combined image an indication position according to the information on the change in the position and attitude of the controller 120.

[0044] The non-volatile memory 212 is an electrically erasable and recordable non-volatile memory, and stores information such as programs to be described later and CG executed by the control unit 211. The control unit 211 can switch the CG read from the non-volatile memory 212 (i.e., the CG used to generate a composite image).

[0045] The working memory 213 is used as a buffer memory for temporarily storing image data captured by the imaging unit 202 and time-series information on the estimated coordinate positions of each joint point of the hand, as an image display memory for the image display unit 203, a working area for the control unit 211, etc.

[0046] The wrist joints may be estimated in the PC 110. In this case, after the captured image is output from the imaging unit 202 to the PC 110, the control unit 211 of the PC 110 estimates the position or posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100. Note that the control unit 211 may estimate the position and posture of each joint point of the hand, processes the image using the information, and outputs the image to the HMD 100.

[0047] In addition to the above, the MR system also includes other components, such as a control unit of either the HMD 100 or the PC 110, which performs various image processing on the captured image acquired by the imaging unit 202 and the display image displayed on the image display unit 203. However, this is not the focus of the present invention, so a description thereof will be omitted.

[0048] The process according to this embodiment will be described below with reference to the flowchart in FIG.

[0049] When the operation starts, the control unit 211 causes the image display unit 203 to display a plurality of virtual objects 402, 403, and 404 in an image display area 401, as shown in Fig. 4. The virtual objects 402, 403, and 404 are generated based on information stored in the recording medium 215. When the user selects one of the displayed virtual objects, the virtual object is placed in a selected state, and a more detailed item within the virtual object can be selected. At this time, the image captured by the imaging unit 202 and the virtual object may be compositely displayed.

[0050] In step S1001, the control unit 211 displays a ray for operating a virtual object. Here, a ray is a display item of a light beam that indicates a position. An example of a ray display is shown in FIG. 5. A ray 420 is displayed from the controller 120 attached to the user's hand 405 toward the inside of the area of ​​the virtual object 404. The tip position of the ray and the light beam are determined based on the position information and the attitude information of the controller 120. A selection area 430 represented by a dashed line is displayed around the virtual object 404. By displaying the selection area 430, the user can recognize that he or she is selecting the virtual object 404 using the controller 120.

[0051] In step S1002, it is determined whether or not to start a correction process for the input coordinate system of the touch sensor 224 of the controller 120. Whether or not to start the correction process is determined, for example, based on whether the touch sensor 224 is in a touched state, and if it is in a touched state, it is determined that the correction process is to be started. Instead of determining based on the touched state of the touch sensor 224, it may also be determined based on the pressed state of the button operation unit 223.

[0052] A condition for starting the correction process may be added. For example, the correction process may be started only when the line of sight of the user detected by the line of sight image capturing unit 207 is on the currently selected virtual object 404. As another condition, for example, the correction process may be started only when it is determined that the orientation of the HMD 100 obtained by the orientation sensor unit 204 is in a substantially stationary state. By determining the user's line of sight information and the orientation information of the HMD 100 together, it is possible to prevent correction from being performed at a timing not intended by the user.

[0053] In step S1002, if it is determined that the correction process is to be started, the control unit 211 proceeds to step S1003, otherwise, the control unit 211 proceeds to step S1008.

[0054] Steps 1003 to 1007 are processes related to correction of the input coordinate system of touch sensor 224. In step S1003, control unit 211 performs display control to stop the ray operation by controller 120. While the ray operation is stopped, display control is performed so that the tip position of the ray does not change even if the position or attitude of controller 120 changes. Note that display control may be performed so that the ray is not displayed.

[0055] In step S1004, the control unit 211 acquires data from the inertia detection unit 225 of the controller 120 obtained via the communication unit 214, and stores the data in chronological order.

[0056] In step S1005, the control unit 211 determines whether or not to end the correction process of the input coordinate system of the touch sensor 224 of the controller 120. Whether or not to end the correction process is determined based on, for example, whether the touch state of the touch sensor 224 has been released, and if the touch state has been released, the control unit 211 determines to end the correction process. In step S1005, if it is determined to end the correction process, the control unit 211 proceeds to step S1006, and if not, returns to step S1004 and repeats obtaining data from the inertia detection unit 225 of the controller 120. Note that the correction process of the input coordinate system of the touch sensor 224 of the controller 120 may be determined based on whether a predetermined time has passed or whether a button operation has been performed.

[0057] In step S1006, the control unit 211 corrects the input coordinate system of the touch sensor 224 using the data of the inertia detection unit 225 of the controller 120 acquired in step S1004. That is, the control unit 211 corrects the input coordinate system of the touch sensor 224 using the data of the inertia detection unit 225 of the controller 120 acquired during the period from step S1002 to step S1005.

[0058] A method for correcting the input coordinate system of the touch sensor 224 will be described in detail below with reference to Fig. 6 to Fig. 9. Fig. 6 is a schematic diagram showing how the controller 120 looks when viewed from above. The touch sensor 224 is disposed on the top of the controller 120, and the amount of movement in a plane when the user operates the touch sensor 224 is expressed by an X coordinate indicating the amount of movement in the X-axis direction and a Y coordinate indicating the amount of movement in the Y-axis direction.

[0059] A method of correcting the input coordinate system of the touch sensor 224 when the user translates the controller 120 from position P in FIG. 7(a) to position P' in FIG. 7(b) during the process of step S1004 will be described. Here, P indicates the center position of the touch sensor 224. The control unit 211 analyzes the time series data of the inertia detection unit 225 acquired in step S1004, and detects that the controller 120 has translated and been displaced in the P-P' direction. Note that the translation of the controller 120 may be detected by analyzing an image captured by the imaging unit 202 of the HMD 100. In FIG. 7, FIG. 7(a) shows the time when it is determined that the correction process of the input coordinate system of the touch sensor 224 has started in step S1002, and FIG. 7(b) shows the time when it is determined that the correction process of the input coordinate system of the touch sensor 224 has ended in step S1005.

[0060] Fig. 8 is a top view showing the movement of the controller 120 shown in Fig. 7. The movement direction P-P' is rotated by θ with respect to the Y axis.

[0061] 9 is a top view of the coordinate system after the control unit 211 performs the correction in step S1006. The X'Y' coordinate system adjusted so that P-P' is in the positive direction of the Y axis is the coordinate system after the correction. In other words, the coordinate system after the correction is the coordinate system obtained by rotating the XY coordinate system by the angle θ of the movement direction as seen from the Y axis.

[0062] After the correction process of the input coordinate system of the touch sensor 224 is completed, an icon 440 indicating the correction status of the input coordinate system of the touch sensor 224 is displayed in the image display area 401 as shown in Fig. 10. The user can know whether the correction of the input coordinate system of the touch sensor 224 has been performed as intended by the user depending on the direction in which the icon 440 is displayed.

[0063] Furthermore, when the correction process of the input coordinate system of the touch sensor 224 is completed, the control unit 211 may notify the user that the correction has been completed. For example, the control unit 211 may notify the user by vibrating the vibration unit 226 of the controller 120 via the communication unit 214, or may notify the user by changing the display method of the icon 440 in the image display area 401.

[0064] In step S1007, the control unit 211 restarts the ray operation by the controller 120, and the process returns to step S1001.

[0065] Step S1008 is a process when it is determined in step S1002 that correction processing of the input coordinate system of the touch sensor 224 is not to be started. In step S1008, the control unit 211 determines whether or not to start an operation using input information of the touch sensor 224 of the controller 120. Whether or not to start an operation using input information of the touch sensor 224 is determined, for example, by the pressed state of the button operation unit 223, and if the button operation unit 223 is in the pressed state, it is determined that the operation is to be started. In step S1008, if it is determined that the operation is to be started, the control unit 211 proceeds to step S1009, and if not, returns to step S1001 and repeats the ray operation.

[0066] In step S1009, the control unit 211 performs an operation using the input information of the touch sensor 224. For example, the selection area 431 in the virtual object 404 shown in Fig. 5 is moved based on the input information of the touch sensor 224. At this time, the input coordinate system of the touch sensor 224 uses the coordinate system corrected in step S1006.

[0067] According to the first embodiment, the operation coordinate system of the controller can be set in a direction intended by the user.

[0068] (Embodiment 2) In the second embodiment, a method of correcting the input coordinate system of the touch sensor 224 when a rotational movement, not a parallel movement, is detected in step S1006 in Fig. 3 will be described. Fig. 11 is a diagram showing that the user has rotated the controller 120 from position (a) to position (b) in step S1004. Figs. 12, 13, and 14 are diagrams that typically show the rotational movement of the controller 120 shown in Fig. 11, with Fig. 12 being a view from the side, and Figs. 13 and 14 being views from above. The attitude of the controller 120 in Fig. 11(a) corresponds to Fig. 12(a), and the attitude of the controller 120 in Fig. 11(b) corresponds to Fig. 12(b).

[0069] When the user rotates the controller 120 from FIG. 12(a) to FIG. 12(b), the control unit 211 analyzes the data of the inertia detection unit 225 acquired in step S1004 in chronological order.

[0070] In the change from FIG. 12(a) to FIG. 12(b), it is detected that the controller 120 has rotated around the axis h-h' and that the point Q' has moved below the operation plane and the point Q has rotated above the operation plane. When the rotational movement has been performed in this way, in step S1006, the control unit 211 corrects the coordinate system so that the side of the point Q' that has moved below the operation plane becomes the positive direction of the Y axis. In addition, the control unit 211 translates the h-h' axis, which is the rotation axis, to the H-H' axis, which becomes the X axis after correction. In other words, the side that has moved below the operation plane as a result of the rotational movement becomes the positive direction of the Y axis after correction, and the control unit 211 rotates the Y axis before correction by θ so that it overlaps with the Y axis, and sets the X'Y' coordinate system as the coordinate system after correction. FIG. 13 shows the coordinate system before correction, and FIG. 14 shows the coordinate system after correction due to the change from FIG. 12(a) to FIG. 12(b). When the XY coordinate system is set as shown in Fig. 13, the change from Fig. 12(a) to Fig. 12(b) changes the coordinate system so that the XY coordinate system is rotated by θ to become the X'Y' coordinate system as shown in Fig. 14. Note that the side moved above the operation plane may be set as the positive direction of the Y' axis after correction, and the X'Y' coordinate system may be set by rotating the Y axis before correction by θ so that it overlaps with the Y axis, to become the coordinate system after correction.

[0071] Also, in the change from FIG. 12(a) to FIG. 12(b), the controller 120 rotates around the axis h-h' and the axis of rotation is near point Q, so the amount of movement of point Q' is larger than the amount of movement of point Q. When the controller 120 rotates in this way, in step S1006, the control unit 211 may correct the coordinate system so that the side of point Q' with the larger amount of movement is in the positive direction of the Y axis. In this case, the control unit 211 also corrects the H-H' axis obtained by translating the h-h' axis, which is the axis of rotation, to the X axis after correction. That is, as a result of the rotation, the point on the outer periphery of the operation plane where the amount of movement is the largest is set to the positive direction of the Y axis, and the control unit 211 rotates the Y axis before correction by θ so that it overlaps with the Y' axis to set the X'Y' coordinate system as the coordinate system after correction. In this case, too, as described above, the coordinate system is changed so that the XY coordinate system is rotated by θ to become the X'Y' coordinate system as shown in FIG. 14. Note that the side with the smaller amount of movement may be set as the positive direction of the Y' axis after correction, and the Y axis before correction may be rotated by θ so as to overlap with the Y' axis to set the X'Y' coordinate system as the coordinate system after correction. By making it possible to change the coordinate system in this way, the user can change to a coordinate system that is easy to operate by tilting the controller in the direction that he or she wants to set as the Y axis.

[0072] The control unit 211 may analyze the data of the inertia detection unit 225 acquired in step S1004 in a time series manner to detect a transition from the state shown in FIG. 13 to the state shown in FIG. 15. In FIG. 15, the data of the inertia detection unit 225 is analyzed in a time series manner to detect a rotational movement of θ about an axis perpendicular to the two-dimensional operation surface (X'Y' plane in FIG. 15) as a result of the state transition. Here, when the user rotates the controller, there are cases where the rotational movement is not performed about one axis, that is, the rotational movement is detected about each of the three rotational axes. In such a case, an axis of the normal to the two-dimensional operation surface is defined, and the amount of change θ in the rotational movement is detected from the component of the rotational movement about that axis.

[0073] In the second embodiment, the operation unit of the controller that changes the coordinate system is flat as shown in Fig. 12, but it may be a curved operation unit such as a trackball, or an operation unit that allows directional input with a stick or lever such as a joystick. Even if the operation unit is curved, stick, or lever, it is sufficient that the correspondence between the input direction and the process corresponding to that input direction can be changed.

[0074] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0075] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0076] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0077] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0078] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0079] [Configuration 1] a first acquisition means for acquiring first information relating to a position or an attitude of a controller; A second acquisition means for acquiring operation information of a two-dimensional operation from the controller; A control means for executing a process based on operation information of the two-dimensional operation; a change means for changing a correspondence relationship between the operation information of the two-dimensional operation and the process executed by the control means, based on information regarding a change in the first information for a predetermined period of time; the control means executes a process based on the operation information of the two-dimensional operation based on the correspondence relationship changed by the change means. 23. An information processing apparatus comprising:

[0080] [Configuration 2] The change means changes a coordinate system for the two-dimensional operation as the correspondence relationship; the control means executes a process based on the operation information of the two-dimensional operation based on the coordinate system changed by the change means. 2. The information processing device according to configuration 1.

[0081] [Configuration 3] the change means changes the coordinate system by rotating the coordinate system based on an amount of change in the information relating to the attitude of the controller acquired by the first acquisition means. 3. The information processing device according to configuration 2.

[0082] [Configuration 4] the change means changes the coordinate system based on a direction in which the position of the controller acquired by the first acquisition means has changed. 3. The information processing device according to configuration 2.

[0083] [Configuration 5] the change means changes the coordinate system by aligning an axis with a direction in which the position of the controller has changed, or by rotating the coordinate system at an angle based on a direction in which the position of the controller has changed. 5. The information processing device according to configuration 4.

[0084] [Configuration 6] the controller has an operation unit capable of two-dimensional operation, the change means changes the coordinate system by rotating the coordinate system about an axis that is a direction of a normal to an operation surface of the two-dimensional operation. 3. The information processing device according to configuration 2.

[0085] [Configuration 7] The operation information of the two-dimensional operation includes an operation direction of the two-dimensional operation, the change means changes a correspondence relationship between an operation direction of the two-dimensional operation and a process executed by the control means; the control means executes a process based on the operation information of the two-dimensional operation based on the correspondence relationship changed by the change means. 7. The information processing device according to any one of configurations 1 to 6.

[0086] [Configuration 8] the change means changes the correspondence relationship when specific operation information of the controller is received by the second acquisition means. 8. The information processing device according to any one of configurations 1 to 7.

[0087] [Configuration 9] the predetermined period is a period from when specific operation information of the controller is received until a predetermined time has elapsed since the specific operation information of the controller is received; 9. The information processing device according to configuration 8.

[0088] [Configuration 10] the predetermined period is a period from a time when a first operation of the controller is accepted to a time when a second operation of the controller is accepted; 10. The information processing device according to any one of configurations 1 to 9.

[0089] [Configuration 11] the time when the second operation of the controller is accepted is the time when the end of the first operation of the controller is accepted; 11. The information processing device according to configuration 10.

[0090] [Configuration 12] When the change means changes the correspondence relationship, the control means notifies a user that the correspondence relationship has been changed. 12. The information processing device according to any one of configurations 1 to 11.

[0091] [Configuration 13] the control means notifies a user that the correspondence relationship has been changed by vibrating the controller. 13. The information processing device according to configuration 12.

[0092] [Configuration 14] Further comprising a display control means for displaying an image on the display unit, The display control means controls to display information about the correspondence relationship changed by the change means on a display unit. 14. The information processing device according to any one of configurations 1 to 13.

[0093] [Configuration 15] A third acquisition means for acquiring information regarding a user's line of sight, The change means changes the correspondence relationship when it is determined that the user is gazing at a predetermined area based on the third acquisition means. 15. The information processing device according to any one of configurations 1 to 14.

[0094] [Configuration 16] A fourth acquisition means for acquiring a position or an attitude of the information processing device, The change means changes the correspondence relationship when it is determined that the information processing device is stationary based on the fourth acquisition means. 16. The information processing device according to any one of configurations 1 to 15.

[0095] [Configuration 17] The second acquisition means acquires operation information of the two-dimensional operation by any one of an OTP, a touch panel, and a track pad from a controller. 17. The information processing device according to any one of configurations 1 to 16.

[0096] [Control method] a first acquisition step of acquiring first information regarding a position or an attitude of a controller; a second acquisition step of acquiring operation information of a two-dimensional operation from the controller; a control step of executing a process based on operation information of the two-dimensional operation; a changing step of changing a correspondence between the operation information of the two-dimensional operation and the process executed by the control step, based on information regarding a change in the first information for a predetermined period of time; The control step executes a process based on the operation information of the two-dimensional operation based on the correspondence relationship changed by the change step. 23. A method for controlling an information processing apparatus comprising:

[0097] [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 17.

[0098] [system] A controller; A first acquisition device that acquires first information regarding a position or an attitude of the controller; A second acquisition device that acquires operation information of a two-dimensional operation from the controller; A control device that executes a process based on operation information of the two-dimensional operation; and a change device that changes a correspondence between the operation information of the two-dimensional operation and the process executed by the control device based on information regarding a change in the first information for a predetermined period. the control device executes a process based on the operation information of the two-dimensional operation based on the correspondence relationship changed by the change device. An information processing system comprising:

Claims

1. a first acquisition means for acquiring first information relating to a position or an attitude of a controller; A second acquisition means for acquiring operation information of a two-dimensional operation from the controller; A control means for executing a process based on operation information of the two-dimensional operation; a change means for changing a correspondence relationship between the operation information of the two-dimensional operation and the process executed by the control means, based on information regarding a change in the first information for a predetermined period of time; the control means executes a process based on the operation information of the two-dimensional operation based on the correspondence relationship changed by the change means.

23. An information processing apparatus comprising:

2. The change means changes a coordinate system for the two-dimensional operation as the correspondence relationship; the control means executes a process based on the operation information of the two-dimensional operation based on the coordinate system changed by the change means.

2. The information processing apparatus according to claim 1,

3. the change means changes the coordinate system by rotating the coordinate system based on an amount of change in the information relating to the attitude of the controller acquired by the first acquisition means.

3. The information processing apparatus according to claim 2.

4. the change means changes the coordinate system based on a direction in which the position of the controller acquired by the first acquisition means has changed.

3. The information processing apparatus according to claim 2.

5. the change means changes the coordinate system by aligning an axis with a direction in which the position of the controller has changed, or by rotating the coordinate system at an angle based on a direction in which the position of the controller has changed.

5. The information processing apparatus according to claim 4.

6. the controller has an operation unit capable of two-dimensional operation, the change means changes the coordinate system by rotating the coordinate system about an axis that is a direction of a normal to an operation surface of the two-dimensional operation.

3. The information processing apparatus according to claim 2.

7. The operation information of the two-dimensional operation includes an operation direction of the two-dimensional operation, the change means changes a correspondence relationship between an operation direction of the two-dimensional operation and a process executed by the control means; the control means executes a process based on the operation information of the two-dimensional operation based on the correspondence relationship changed by the change means.

2. The information processing apparatus according to claim 1,

8. the change means changes the correspondence relationship when specific operation information of the controller is received by the second acquisition means.

8. The information processing apparatus according to claim 1,

9. the predetermined period is a period from when specific operation information of the controller is received until a predetermined time has elapsed since the specific operation information of the controller is received; 9. The information processing apparatus according to claim 8,

10. the predetermined period is a period from a time when a first operation of the controller is accepted to a time when a second operation of the controller is accepted; 8. The information processing apparatus according to claim 1,

11. the time when the second operation of the controller is accepted is the time when the end of the first operation of the controller is accepted; 11. The information processing apparatus according to claim 10,

12. When the change means changes the correspondence relationship, the control means notifies a user that the correspondence relationship has been changed.

8. The information processing apparatus according to claim 1,

13. the control means notifies a user that the correspondence relationship has been changed by vibrating the controller.

13. The information processing apparatus according to claim 12.

14. Further comprising a display control means for displaying an image on the display unit, The display control means controls to display information about the correspondence relationship changed by the change means on a display unit.

2. The information processing apparatus according to claim 1,

15. A third acquisition means for acquiring information regarding a user's line of sight, The change means changes the correspondence relationship when it is determined that the user is gazing at a predetermined area based on the third acquisition means.

8. The information processing apparatus according to claim 1,

16. A fourth acquisition means for acquiring a position or an attitude of the information processing device, The change means changes the correspondence relationship when it is determined that the information processing device is stationary based on the fourth acquisition means.

8. The information processing apparatus according to claim 1,

17. The second acquisition means acquires operation information of the two-dimensional operation by any one of an OTP, a touch panel, and a track pad from a controller.

8. The information processing apparatus according to claim 1,

18. a first acquisition step of acquiring first information regarding a position or an attitude of a controller; a second acquisition step of acquiring operation information of a two-dimensional operation from the controller; a control step of executing a process based on operation information of the two-dimensional operation; a changing step of changing a correspondence between the operation information of the two-dimensional operation and the process executed by the control step, based on information regarding a change in the first information for a predetermined period of time; The control step executes a process based on the operation information of the two-dimensional operation based on the correspondence relationship changed by the change step.

23. A method for controlling an information processing apparatus comprising:

19. A program for causing a computer to function as each of the means of the information processing apparatus according to claim 1.

20. A controller; A first acquisition device that acquires first information regarding a position or an attitude of the controller; A second acquisition device that acquires operation information of a two-dimensional operation from the controller; A control device that executes a process based on operation information of the two-dimensional operation; a change device that changes a correspondence between the operation information of the two-dimensional operation and the process executed by the control device based on information regarding a change in the first information for a predetermined period of time; the control device executes a process based on the operation information of the two-dimensional operation based on the correspondence relationship changed by the change device. An information processing system comprising:

Images