Information processing device, information processing method, program, and storage medium

Abstract

To provide a user interface having high convenience as the user interface such as a VR system or an MR system.SOLUTION: An information processing apparatus according to the present invention comprises: acquisition means that acquires information indicating a position or an attitude of an object for operating a virtual object; and control means that controls a position or an attitude of a graphic corresponding to the object on the basis of the information acquired by the acquisition means. The control means controls the position or the attitude of the graphic in a first state, on the basis of first information being information indicating the position or the attitude of the graphic in the first state in which the information is acquired with a higher degree of accuracy than a predetermined degree of accuracy, and second information being information indicating the position or the attitude of the graphic in a second state in which the information is not acquired with a higher degree of accuracy than the predetermined degree of accuracy.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, a program, and a storage medium.

Background Art

[0002] In recent years, the development of virtual reality (VR) systems and mixed reality (MR) systems aimed at seamless integration between the real space and the virtual space has been actively carried out. A head-mounted display (HMD) is a glasses-type device equipped with a small display that a user wears on the head. When performing a user interface (UI) operation via an HMD, a hand controller is utilized to convert the movement of the hand into an operation within the virtual space. Based on the position and orientation (position and direction) of the hand controller, virtual light rays called rays or virtual pointers are displayed on the VR system or MR system, enabling the user to freely operate virtual objects.

[0003] In Patent Document 1, a hand controller has been proposed that can detect the position and orientation of the hand by emitting a plurality of infrared lights (IR lights) from the hand controller and having a camera mounted on the HMD receive the infrared lights. In Patent Document 2, an apparatus for estimating the position and orientation of a hand controller using an image captured by a camera mounted on the hand controller has been proposed.

Prior Art Documents

Patent Documents

[0004] [[ID=!26]]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technologies disclosed in Patent Document 1 or Patent Document 2 mentioned above, the accuracy of detecting the position and orientation of the hand controller may decrease. Furthermore, depending on the change in the accuracy of detecting the position and orientation of the hand controller, the position or orientation of graphics such as rays based on the hand controller's position and orientation may change significantly in a short period of time. For this reason, conventional user interfaces are less convenient.

[0006] Therefore, the present invention aims to provide a highly convenient user interface for use in VR systems or MR systems, etc. [Means for solving the problem]

[0007] A first aspect of the present invention is for manipulating virtual objects operation An acquisition means for acquiring information indicating the position or orientation of an object, and based on the information acquired by the acquisition means, operation It has control means for controlling the position or orientation of a graphic corresponding to an object, and the control means is When the operation object changes from a second state in which it is not visible in the captured image to a first state in which it is visible in the captured image, if the object indicated by the graphic in the second state is different from the object indicated by the graphic in the first state, before Record number In state 1, the above operation object The first piece of information is information indicating the position or orientation of the previous Record number Based on the second piece of information, which is information indicating the position or orientation of the graphic in state 2, the position or orientation of the graphic in state 1 is controlled. If the object pointed to by the graphic in the second state is the same as the object pointed to by the graphic in the first state, the position or orientation of the graphic in the first state is controlled based on the first information, not on the second information. This is an information processing device characterized by the following:

[0008] A second aspect of the present invention relates to a method for manipulating virtual objects. operation An acquisition step to acquire information indicating the position or orientation of an object, and based on the information acquired in the acquisition step, operation A control step that controls the position or orientation of a graphic corresponding to an object, and the control step in teeth, When the operation object changes from a second state in which it is not visible in the captured image to a first state in which it is visible in the captured image, if the object indicated by the graphic in the second state is different from the object indicated by the graphic in the first state, before Record number In state 1, the above operation object The first piece of information is information indicating the position or orientation of the previous Record number Based on the second piece of information, which is information indicating the position or orientation of the graphic in state 2, the position or orientation of the graphic in state 1 is controlled. If the object pointed to by the graphic in the second state is the same as the object pointed to by the graphic in the first state, the position or orientation of the graphic in the first state is controlled based on the first information, not on the second information. This is an information processing method characterized by the following:

[0009] A third aspect of the present invention is a program for causing a computer to function as one of the means of the information processing apparatus described above.

[0010] A fourth aspect of the present invention is a computer-readable storage medium that stores a program for causing a computer to function as one of the means of the information processing apparatus described above. [Effects of the Invention]

[0011] According to the present invention, a highly convenient user interface can be provided for use as a user interface for VR systems or MR systems, etc. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing an example of the system configuration according to the first embodiment. [Figure 2] This is a flowchart of the image generation process according to the first embodiment. [Figure 3] This figure shows an example of UI control according to the first embodiment. [Figure 4] This is a flowchart of the image generation process according to the second embodiment. [Figure 5] This figure shows an example of UI control according to the second embodiment. [Figure 6] This figure shows an example of UI control according to the third embodiment. [Figure 7] This is a block diagram showing a hardware configuration example according to the fourth embodiment. [Modes for carrying out the invention]

[0013] Hereinafter, with reference to the accompanying drawings, embodiments for implementing the present invention will be described in detail. The embodiments described below are examples for realizing the present invention, and should be appropriately modified or changed according to the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiments.

[0014] [First Embodiment] (Configuration) FIG. 1 is a block diagram showing a configuration example of a system according to the first embodiment. The system according to the first embodiment is an image processing system for presenting a composite reality space (MR space) that combines the real space and the virtual space to a system experiencer (user). In the first embodiment, a case of presenting the MR space to the user by displaying a composite image obtained by synthesizing an image of the virtual space drawn by computer graphics (CG) and an image of the real space will be described.

[0015] The system according to the first embodiment includes a display device 1000, an information processing device 1100, and an input device 1200.

[0016] The information processing device 1100 synthesizes an image of the real space captured from the display device 1000 with an image of the virtual space generated in the information processing device 1100 to generate a composite image. The information processing device 1100 outputs the composite image to the display device 1000 as a composite reality image (MR image). Note that the first embodiment relates to an information processing device that displays an image of the virtual space, and the system according to the first embodiment is not limited to an MR system that displays an MR image (an image obtained by synthesizing an image of the real space with an image of the virtual space). That is, the system according to the first embodiment may be a VR (virtual reality) system that presents only an image of the virtual space to the user, an AR (augmented reality) system that presents an image of the virtual space while transmitting the real space, or the like.

[0017] ​The display device 1000 has an imaging unit 1010 and a display unit 1020.

[0018] The imaging unit 1010 continuously captures images of the real space in a time series and outputs the captured images of the real space (captured images) to the information processing device 1100. The imaging unit 1010 may include a stereo camera consisting of two cameras fixed to each other so that it can capture images of the real space in the direction of the user's line of sight.

[0019] The display unit 1020 displays the MR image output from the information processing device 1100. The display unit 1020 may include two displays positioned to correspond to the user's left and right eyes, respectively. In this case, the display corresponding to the user's left eye displays the MR image for the left eye, and the display corresponding to the user's right eye displays the MR image for the right eye.

[0020] The display device 1000 is, for example, a head-mounted display device (HMD). However, the display device 1000 is not limited to an HMD; it may also be a handheld display (HHD). An HHD is a handheld display. In other words, the display device 1000 may be a display that the user holds in their hand and looks through like binoculars to observe images. Alternatively, the display device 1000 may be a display terminal such as a tablet or a smartphone.

[0021] The input device 1200 has an input unit 1210 and a position and orientation calculation unit 1220. The input device 1200 is a video game input device that can acquire values ​​such as a value corresponding to the direction of the stick's tilt or a value indicating the state of a button's press. However, the input device 1200 is not limited to an input device held in the user's hand, such as an input device for video games, but may also be an input device attached to the user's body or hand. The input device 1200 may be, for example, a ring-shaped device that can be attached to the user's finger.

[0022] The input unit 1210 outputs values ​​corresponding to the user's operation (such as values ​​corresponding to the direction of the stick's tilt or values ​​indicating the button's pressed state) as operation information to the input processing unit 1130.

[0023] The position and orientation calculation unit 1220 calculates (acquires) the position and orientation (position and attitude) of the input device 1200 in the world coordinate system. The position and orientation calculation unit 1220 then outputs the calculated position and orientation information to the input processing unit 1130. The position and orientation calculation unit 1220 has, for example, sensors (angular velocity sensor, acceleration sensor, or geomagnetic sensor, etc.) for calculating the position and orientation of the input device 1200. The position and orientation calculation unit 1220 may have multiple sensors. The position and orientation calculation unit 1220 also includes an imaging unit and an imaging processing unit, and calculates Simultaneous Localization and The position and orientation of the input device 1200 may be calculated by performing Mapping (SLAM) processing. Alternatively, the position and orientation calculation unit 1220 may calculate the position and orientation in conjunction with an optical sensor installed in real space. The position and orientation calculation unit 1220 may calculate either the position component or the orientation component, rather than both. Alternatively, the position and orientation calculation unit 1220 may output values ​​(values ​​output from the sensor) or images for calculating the position and orientation of the input device 1200 to the information processing device 1100, and the position and orientation calculation unit 1110 of the information processing device 1100 may calculate the position and orientation of the input device 1200.

[0024] Furthermore, the information processing device 1100 and the display device 1000 are configured to enable data communication between them. They are connected. The information processing device 1100 and the input device 1200 are connected to each other so that they can communicate data. The data communication may be wired or wireless. The display device 1000 may also have the information processing device 1100 inside.

[0025] The information processing device 1100 includes a position and orientation calculation unit 1110, a control unit 1120, an input processing unit 1130, an image generation unit 1140, an image synthesis unit 1150, and a data storage unit 1160.

[0026] The position and orientation calculation unit 1110 calculates the position and orientation of the imaging unit 1010 in the world coordinate system. Specifically, the position and orientation calculation unit 1110 extracts markers assigned to the world coordinate system from the image of real space captured by the imaging unit 1010. Then, the position and orientation calculation unit 1110 calculates the position and orientation of the imaging unit 1010 in the world coordinate system based on the position and orientation of the extracted markers. Finally, the position and orientation calculation unit 1110 stores the calculated information indicating the position and orientation of the imaging unit 1010 (position and orientation information) in the data storage unit 1160.

[0027] Furthermore, the position and orientation calculation unit 1110 calculates the position and orientation of the input device 1200 in the world coordinate system using position and orientation information of the input device 1200 obtained from the position and orientation calculation unit 1220 or images of real space captured by the imaging unit 1010. At this time, depending on the calculation method of the position and orientation calculation unit 1220, a difference (error) may occur between the calculated position and orientation of the input device 1200 and its actual position and orientation. For example, in a method that calculates the position and orientation using a combination of an angular velocity sensor, an acceleration sensor, and a geomagnetic sensor, the accumulated errors of each sensor may result in a calculated position and orientation that is an error relative to the actual position and orientation. Alternatively, this method may not be able to calculate the position and orientation of the input device 1200. Also, in a method that calculates the position and orientation using an optical sensor installed in real space, the optical sensor may be obscured by other real objects, resulting in a calculated position and orientation that is an error relative to the actual position and orientation. Alternatively, this method may also not be able to calculate the position and orientation. In such cases, the position and orientation calculation unit 1110 can accurately calculate the position and orientation of the input device 1200 based on the position and orientation of the markers by extracting the markers attached to the input device 1200 from the image of the real space captured by the imaging unit 1010. At this time, the position and orientation calculation unit 1110 may use all or part of the position and orientation calculation result obtained from the position and orientation calculation unit 1220 of the input device 1200. However, the calculation method of the position and orientation calculation unit 1110 is not limited to the method using markers, but may also be a method calculated by SLAM processing. The position and orientation calculation unit 1110 stores the calculated position and orientation information of the input device 1200 in the data storage unit 1160.

[0028] If the input device 1200 is not visible in the image of the real space captured by the imaging unit 1010, the position and orientation calculation unit 1110 calculates the position and orientation of the input device 1200 based on the value obtained from the position and orientation calculation unit 1220 of the input device 1200. At this time, the position and orientation calculation unit 1110 may calculate the position and orientation of the input device 1200 based on the value obtained from the position and orientation calculation unit 1220 and other information. For example, if the input device 1200 is not visible in the image of the real space, and the input device 1200 is equipped with an acceleration sensor or the like, the position and orientation calculation unit 1110 may calculate the position and orientation of the input device 1200 based on the detection result of the acceleration sensor or the like. For example, the position and orientation calculation unit 1110 adds the position of the input device 1200 at a past point in time when the input device 1200 was visible in the image of the real space, and the amount of movement of the input device 1200 from that point in time calculated from the acceleration. As a result, the position and orientation calculation unit 1110 may calculate the current position and orientation of the input device 1200. If the input device 1200 is not visible in the image of real space, the position and orientation calculation unit 1110 may only detect the orientation of the input device 1200, or it may not be able to calculate the position and orientation of the input device 1200.

[0029] The control unit 1120 controls the entire information processing device 1100. The control unit 1120 controls the position and orientation of the UI (graphics) displayed on the display unit 1020 based on, for example, the position and orientation information of the input device 1200 stored in the data storage unit 1160. The position of the UI is indicated by three-dimensional coordinate information according to a three-axis orthogonal coordinate system, for example, the X, Y, and Z axes. If the UI is a virtual ray emitted from the user's hand, the position of the UI is, for example, the start or end point of the ray. The orientation of the UI corresponds to the orientation of the UI in a three-dimensional virtual space. If the UI is a ray, the orientation of the UI corresponds to, for example, the direction in which the ray extends.

[0030] The input processing unit 1130 modifies the data related to each virtual object stored in the data storage unit 1160, as well as the data related to the light source illuminating the virtual space, based on the operation information obtained from the input unit 1210. The input processing unit 1130 then stores the modified data related to each virtual object and the modified data related to the light source illuminating the virtual space in the data storage unit 1160.

[0031] The image generation unit 1140 constructs a virtual space based on the virtual space data stored in the data storage unit 1160. The virtual space data includes data related to each virtual object that constitutes the virtual space, and data related to light sources that illuminate the virtual space. The image generation unit 1140 then obtains the position and orientation information of the imaging unit 1010 calculated by the position and orientation calculation unit 1110 from the data storage unit 1160. The image generation unit 1140 also obtains the position and orientation information of the UI controlled by the control unit 1120 from the data storage unit 1160. The image generation unit 1140 generates an image of the virtual space (virtual space image) corresponding to the position and orientation of the imaging unit 1010. The technology for generating an image of the virtual space corresponding to the position and orientation of the imaging unit 1010 is a well-known technology, so a detailed explanation is omitted.

[0032] The image synthesis unit 1150 generates an MR image by combining the image of the virtual space generated by the image generation unit 1140 with the image of the real space captured by the imaging unit 1010. At this time, the image of the virtual space generated by the image generation unit 1140 may be an image representing the entire virtual space or an image representing a part of the virtual space. The image synthesis unit 1150 may also generate the MR image by performing an affine transformation on the image, or by assigning it to a parametric surface. The image synthesis unit 1150 then outputs the generated MR image to the display unit 1020.

[0033] As described above, the data storage unit 1160 stores various types of information. The data storage unit 1160 includes RAM and a hard disk drive device. In addition to the information described above as information to be stored in the data storage unit 1160, the data storage unit 1160 also stores information that is described as known information in the first embodiment.

[0034] Furthermore, in the first practical application of the system, the user's hand may be used instead of the input device 1200. In this case, the input processing unit 1130 recognizes the user's hand movements (gestures) from the image of the real space captured by the imaging unit 1010. Based on the recognized gesture, the input processing unit 1130 modifies the data related to each virtual object stored in the data storage unit 1160, as well as the data related to the light source illuminating the virtual space. The input processing unit 1130 then stores the modified data related to each virtual object and the modified data related to the light source illuminating the virtual space in the data storage unit 1160.

[0035] (Image generation process) Figure 2 is a flowchart of the image generation process performed by the information processing device 1100 to generate an MR image and output it to the display device 1000.

[0036] In step S2010, the control unit 1120 sets the UI adjustment flag, which indicates whether or not to adjust the UI, to NO.

[0037] In step S2020, the control unit 1120 controls the position and orientation calculation unit 1110 to calculate the position and orientation of the display device 1000 (imaging unit 1010) (position and orientation of the virtual viewpoint) and the position and orientation of the input device 1200.

[0038] In step S2030, the control unit 1120 determines whether the position and orientation calculation accuracy state (hereinafter referred to as the "calculation accuracy state") of the input device 1200 is in a first state or a second state. The first state is a high-precision state in which the position and orientation are calculated with an accuracy higher than a predetermined accuracy. For example, if the input device 1200 is visible in the image of the real space captured by the imaging unit 1010, and the position and orientation of the input device 1200 is calculated using the image of the real space, the control unit 1120 determines that the calculation accuracy state is in the first state. For example, if the input device 1200 is not visible in the image of the real space captured by the imaging unit 1010, the control unit 1120 determines that the calculation accuracy state is in the second state (not high-precision state). The second state is a state in which the position and orientation are not calculated with an accuracy higher than a predetermined accuracy. For example, the second state may be a state in which the position and orientation are not calculated, or a state in which the position and orientation are calculated with an accuracy lower than a predetermined accuracy.

[0039] Furthermore, the control unit 1120 may determine the calculation accuracy state based on position and orientation information of the input device 1200 at multiple points in time. For example, the control unit 1120 may determine that the system is in the second state if the difference between the current position and orientation of the input device 1200 and the previous position and orientation is greater than or equal to a threshold (e.g., if there is a jump in the calculated position and orientation value, or if jitter is occurring). Also, if the information output from the input device 1200 includes information regarding the calculation accuracy state, the control unit 1120 may determine the calculation accuracy state based on this information. Alternatively, the control unit 1120 may determine the calculation accuracy state based on the aforementioned multiple pieces of information.

[0040] In step S2040, the control unit 1120 determines whether the calculation accuracy state is in the first state (high accuracy state). If the control unit 1120 determines that the calculation accuracy state is in the first state, it proceeds to step 2050. If it determines that the calculation accuracy state is not in the first state (is in the second state), it proceeds to step S2120.

[0041] In step S2050, the control unit 1120 determines whether the calculation control state has changed from the second state (low-precision state) to the first state (high-precision state) by the time of step S2040. For example, if the control unit 1120 determined in the previous step S2040 that the calculation control state was in the second state, it determines that the calculation control state has changed from the second state to the first state. If the control unit 1120 determines that the calculation control state has changed from the second state to the first state, it proceeds to step S2070; if it determines that the calculation control state has not changed (the first state is maintained), it proceeds to step S2060.

[0042] In step S2060, the control unit 1120 determines whether the UI adjustment flag is YES or NO. If the UI adjustment flag is YES, the control unit 1120 proceeds to step S2090; otherwise, it proceeds to step S2120.

[0043] In step S2070, the control unit 1120 sets the UI adjustment flag to YES.

[0044] In step S2080, the control unit 1120 sets the amount of change to the UI based on the position and orientation of the UI when the calculation accuracy state transitions from the second state (not high accuracy state) to the first state (high accuracy state). Here, the amount of change is the position and orientation of the UI when the calculation accuracy state transitions from the second state to the first state This parameter is used to prevent the UI's position and orientation from changing significantly when transitioning between states. The control unit 1120 sets the amount of change (the amount to change) to modify the UI's position and orientation according to the first information, based on the first information, which is the UI's position and orientation in the first state, and the second information, which is the UI's position and orientation in the second state. For example, in step S2080, the control unit 1120 sets the difference between the first information (for example, the UI's position and orientation immediately after transitioning to the first state) and the second information (for example, the UI's position and orientation immediately before transitioning to the first state) as the amount of change. The amount of change is calculated using a three-dimensional parameter. Note that if the calculation accuracy allows obtaining the difference in the UI's position and orientation when transitioning from the second state to the first state, the control unit 1120 may use a parameter other than the amount of change.

[0045] In step S2090, the control unit 1120 determines whether the difference in a specific two-dimensional plane coordinate system between the UI's position and orientation after considering (reflecting) the amount of change and the UI's position and orientation before considering the amount of change is less than or equal to a reference value ξ. The specific two-dimensional plane coordinate system is either a two-dimensional plane coordinate system perpendicular to the optical axis direction of the imaging unit 1010 (depth direction in the camera coordinate system) or a two-dimensional plane coordinate system perpendicular to the user's front direction. If the difference in position and orientation is less than or equal to the reference value ξ, the control unit 1120 proceeds to step S2100; otherwise, it proceeds to step S2110. The UI's position and orientation after considering the amount of change is the position and orientation of the UI according to the first information (information based on the calculation result of the position and orientation of the input device 1200) changed by the amount of change set in step S2080. The UI's position and orientation before considering the amount of change is the UI's position and orientation according to the first information, based on the calculation result of the position and orientation of the input device 1200.

[0046] In step S2100, the control unit 1120 changes the UI adjustment flag from YES to NO.

[0047] In step S2110, the control unit 1120 updates (determines) the UI's position and orientation based on the calculation result and change amount of the input device 1200. For example, the control unit 1120 determines the updated UI's position and orientation by changing the UI's position and orientation based on the calculation result of the input device 1200's position and orientation according to the change amount set in step S2080. The control unit 1120 then outputs the updated UI's position and orientation information to the data storage unit 1160.

[0048] In step S2120, the control unit 1120 updates the UI's position and orientation based on the calculation result of the input device 1200's position and orientation. The control unit 1120 then stores the updated UI's position and orientation information in the data storage unit 1160.

[0049] In step S2130, the control unit 1120 controls the image generation unit 1140 to generate an image of the virtual space based on the position and orientation of the virtual viewpoint calculated in step S2020 and the position and orientation of the UI set in step S2110 or step S2120.

[0050] In step S2140, the control unit 1120 controls the image synthesis unit 1150 to generate an MR image by combining the real image captured by the imaging unit 1010 in real space with the virtual space image generated in step S2130.

[0051] In step S2150, the control unit 1120 terminates the process if an instruction to terminate the process is given, or if the conditions for terminating the process are met. On the other hand, if no instruction to terminate the process has been given, and the conditions for terminating the process are not met, the control unit 1120 returns to step S2020 and repeats the process.

[0052] Figures 3(A) to 3(F) show the relationship between the image synthesis result generated by the image synthesis unit 1150 and the position and orientation of the input device 1200. Referring to Figures 3(A) to 3(F), an example of controlling the position and orientation of the UI linked to the input device 1200 will be explained.

[0053] Image 3010 is an image representing the real space, captured by the imaging unit 1010. The virtual object 3020 is a virtual object that is rendered as an image from the perspective of the display device 1000, representing the virtual space held by the data storage unit 1160. The hand 3030 is the hand of the user wearing the input device 1200. UIs 3050, 3070, 3090, 3110, 3140, and 3170 are UIs that move (are displayed) in conjunction with the movement of the input device 1200. UIs 3120 and 3150 are UIs that are not actually displayed. Although the shape of the UIs in Figures 3(A) to 3(F) is triangular, the shape of the UIs can be any shape other than a triangle, as long as the user can recognize the area that the UI is pointing to.

[0054] Figure 3(A) shows the case where the input device 1200 is within the field of view (angle of view; imaging range) of the imaging unit 1010. In a case like Figure 3(A), the position and orientation calculation unit 1110 can calculate (acquire) the position and orientation of the input device 1200 using the image captured by the imaging unit 1010 in real space. Therefore, the position and orientation of the input device 1200 is calculated with high accuracy. Based on the calculation result of the position and orientation of the input device 1200, the control unit 1120 can display the UI 3050 and point to the virtual object 3020.

[0055] Figure 3(B) shows the case where the input device 1200 is outside the field of view of the imaging unit 1010. For example, if the user continues to use the device for a shorter time than a predetermined time while the input device 1200 is outside the field of view of the imaging unit 1010, the calculation result of the position and orientation of the input device 1200 will not have a significant error compared to the actual position and orientation. Therefore, the relationship between the position and orientation of UI3070 and the input device 1200 in Figure 3(B) is the same as the relationship between the position and orientation of UI3050 and the input device 1200 in Figure 3(A).

[0056] Figure 3(C) shows the case when the input device 1200 moves outside the field of view of the imaging unit 1010. The calculated position and orientation of the input device 1200 has an error compared to its actual position and orientation. Such an error may occur, for example, if the user continues to use the device for longer than a predetermined time while the input device 1200 is outside the field of view of the imaging unit 1010. Although Figure 3(C) shows a state in which there is an error in the calculated position and orientation of the input device 1200, the user is still able to point to the virtual object 3020, as in the states shown in Figures 3(A) and 3(B). When the input device 1200 is outside the field of view of the imaging unit 1010, the control unit 1120 may update the position and orientation of the UI 3090 based on the position and orientation information of the input device 1200 obtained from the input device 1200, or it may update only the orientation. Alternatively, the control unit 1120 does not have to update the position and orientation of the UI 3090. For example, if the user's hand is used instead of the input device 1200, the control unit 1120 does not need to update the position and orientation of the UI 3090 when the input device 1200 is outside the field of view of the imaging unit 1010.

[0057] Figure 3(D) shows the input device 1200 entering the field of view of the imaging unit 1010 from outside its field of view. UI3110 is displayed at a position and orientation that has been changed from UI3120 based on the calculated position and orientation of the input device 1200, by the amount of error (difference) between the position and orientation of the input device 1200 immediately before reaching the state in Figure 3(D) and the position and orientation calculated in the state in Figure 3(D). In Figure 3(D), since the input device 1200 is within the field of view of the imaging unit 1010, the position and orientation calculation unit 1110 can calculate the position and orientation of the input device 1200 with high accuracy, just as in Figure 3(A). Therefore, in Figure 3(D), the control unit 1120 can eliminate the error present in the state in Figure 3(C) and display the UI at the position of UI3120. However, if the UI is displayed at the position of UI3120, the temporary UI that was previously indicated will be lost. In some cases, the UI may point to a location different from the intended object, causing discomfort to the user. Therefore, the control unit 1120 displays UI3110, which is a modified version of UI3120, instead of UI3120, which is based on the calculated position and orientation of the input device 1200. For example, UI3110 immediately after reaching the state shown in Figure 3(D) will be displayed as if the input device 1200 had been moved from its position and orientation just before reaching the state shown in Figure 3(D).

[0058] Figure 3(E) shows the input device 1200 moving within the field of view of the imaging unit 1010 while the UI points to the same position as in Figure 3(D). The control unit 1120, as in the case of Figure 3(D), takes into account the amount of change in the UI and displays UI3140 instead of UI3150. UI3140 is able to point to the same position in the virtual object 3020 as in Figure 3(D).

[0059] Figure 3(F) shows the state when the hand 3030, which is fitted with the input device 1200, rotates while the UI points to the same position as in Figure 3(E). In the state shown in Figure 3(F), the difference between the position of the UI displayed in the image synthesis result (UI after considering the amount of change) and the position of the UI after the amount of change has been eliminated (UI before considering the amount of change) is less than or equal to the reference value ξ. For example, in Figure 3(E), suppose there is a difference between UI3140 and UI3150 only in the horizontal direction. From this state, when the hand 3030 is moved to rotate to the state shown in Figure 3(F), there is a difference in the depth direction of the camera coordinate system between the UI after considering the amount of change and the UI before considering it, but the difference in the plane perpendicular to the depth direction is less than or equal to the reference value ξ. Thus, if the difference in the displayed (projected) UI (the difference in a specific two-dimensional plane coordinate system) is less than or equal to the reference value ξ, the control unit 1120 eliminates the amount of change in the UI and displays the UI 3170 based on the original position and orientation of the input device 1200, as shown in Figure 3(A).

[0060] In the first embodiment, the amount of change is determined in a coordinate system based on, for example, the user's hand or an object worn on that hand (input device 1200). Therefore, even if the position of the UI is changed to the left (to the left from the user's perspective) at a certain time, the UI does not necessarily always change to the left. If the amount of change is set to change the position of the UI in the direction from the index finger to the thumb, the position of the UI will always change in the direction from the index finger to the thumb, and therefore, depending on the posture of the user's hand, it may change in a direction other than to the left. For this reason, changes like those shown in Figures 3(D) to 3(F) are possible.

[0061] Although it was explained that UI3120 and 3150 are not actually displayed, the control unit 1120 may display UI3120 and 3150 in a different display state (display mode) than UI3050, 3070, 3090, 3110, 3140, and 3170 (the main UI). For example, the control unit 1120 may display UI3120 and 3150 with different colors, transparency, illumination state, blinking state, etc., than the main UI. The control unit 1120 may display UI3120 and 3150 in any display state as long as it is a different display state from the main UI. Furthermore, the control unit 1120 may control UI3110 and 3140, which are displayed considering the amount of change, to have a different display state than UI3050, 3070, 3090, and 3160, which are displayed without considering the amount of change (in a position and orientation according to the first information).

[0062] [Differentiation] In the first embodiment, the control unit 1120 determines whether to change the UI by the amount of change based on whether the difference in the position and orientation of the UI in a specific two-dimensional coordinate system is less than or equal to a reference value when the amount of change is considered and when it is not. However, the determination conditions are not limited to this. For example, the control unit 1120 may determine whether to change the UI by whether the difference in the position and orientation of the UI in a three-dimensional coordinate system is less than or equal to a reference value when the amount of change is considered and when it is not.

[0063] Furthermore, the control unit 1120 indicates the UI depending on whether the amount of change is considered or not. The UI's position and orientation may be controlled by determining whether the object being pointed to changes into another object. In this case, the control unit 1120 considers the amount of change and updates the UI with the changed position and orientation if the object pointed to by the UI is different in either of the two cases. If the object pointed to by the UI is the same in either of the two cases, the control unit 1120 updates the UI with the position and orientation without considering the amount of change. For example, when a user is pointing to a virtually displayed menu or a button within a menu, the control unit 1120 determines whether the object pointed to by the UI changes into another object, depending on whether the amount of change is considered or not.

[0064] In addition, the control unit 1120 may determine whether or not the user is using the UI and control the position and orientation of the UI accordingly. In this case, if the control unit 1120 determines that the user is using the UI, it will consider the amount of change and update the UI with the changed position and orientation. If the control unit 1120 determines that the user is not using the UI, it will update the UI with a position and orientation that does not consider the amount of change. Furthermore, the control unit 1120 may control the position and orientation of the UI by combining multiple conditions. The method for determining whether or not the user is using the UI may be whether or not the user is holding the input device 1200, or whether or not the amount of movement of the input device 1200 is greater than or equal to a reference value. For example, if the user is holding the input device 1200 or if the amount of movement of the input device 1200 is greater than or equal to a threshold, the control unit 1120 will determine that the user is using the UI. Note that whether or not the user is using the UI may also be determined by whether or not there is an object at the point the UI is pointing to, or by whether or not the user's voice is acquired. Whether or not the user is using the UI may be determined by any known technique.

[0065] Furthermore, even when control is in place to consider the amount of change, the user may forcibly control the system to not consider the amount of change (to eliminate the amount of change). Also, if the position indicated by the UI or the position of the UI is within the peripheral field of view of the display device 1000 (for example, the edge range of the imaging range of the imaging unit 1010), the system may forcibly control the system to not consider the amount of change. In addition, when considering the amount of change, the control unit 1120 may change the position and orientation of the UI, or change either the position or the orientation.

[0066] [Second Embodiment] In the first embodiment, the amount of change used to control the position and orientation of the UI was controlled as constant, but the position and orientation of the UI may be controlled by gradually decreasing the amount of change.

[0067] Figure 4 is a flowchart of the image generation process performed by the information processing device 1100 to generate an MR image and output it to the display device 1000. In Figure 4, steps that are the same as in Figure 2 are numbered the same way as in Figure 2, and their explanations are omitted.

[0068] In step S4010, the control unit 1120 adjusts the change amount so that the change amount set in step S2080 is smaller than the change amount set previously. The adjustment method may be, for example, by multiplying the change amount by a coefficient with a value of less than 1, or by multiplying each position and orientation component of the change amount by different coefficients with values ​​of 1 or less. Alternatively, the adjustment method may be a function that adjusts the change amount according to the change over time, or a function that uses the position and orientation component of the imaging unit 1010 or input device 1200 as a variable. Any other method for adjusting the change amount is acceptable as long as the change amount gradually decreases and the position and orientation of the displayed UI gradually approaches the position and orientation of the UI corresponding to the first information.

[0069] Figures 5(A) to 5(F) show the relationship between the image synthesis result generated by the image synthesis unit 1150 and the position and orientation of the input device 1200. Referring to Figures 5(A) to 5(F), an example of controlling the position and orientation of the UI linked to the input device 1200 while gradually adjusting the amount of change will be explained. In Figures 5(A) to 5(F), the same part as in Figures 3(A) to 3(F) is shown. The minutes are numbered the same as in Figures 3(A) to 3(F), so their explanation is omitted. Figures 5(A) to 5(D) and 5(F) show the same state as in Figures 3(A) to 3(D) and 3(F), so their explanation is omitted.

[0070] Figure 5(E) shows the input device 1200 moving within the field of view of the imaging unit 1010 while the UI points to the same position as in Figure 5(D). By gradually reducing the amount of change, the control unit 1120 can display the UI 5030 so that it points to the same position as in Figure 5(D) within the virtual object 3020, while approaching the UI 5020 as it would be without considering the amount of change.

[0071] [Third Embodiment] In the first and second embodiments, the amount of change in the UI is the change (difference) in the calculated position and orientation of the UI when the state of the calculation accuracy of the input device 1200 (calculation accuracy state) transitions from a second state (not high accuracy state) to a first state (high accuracy state). On the other hand, the control unit 1120 may set the amount of change based on the difference between the location that the UI pointed to in the second state and the location that the UI, based on the position and orientation of the input device 1200, points to in the first state. For example, the control unit 1120 sets the amount of change so that the position based on the position and orientation of the input device 1200 when the calculation accuracy state becomes the first state is the starting point of the UI, and the position that the UI pointed to before the calculation accuracy state became the first state is the ending point of the UI. In the third embodiment, the control unit 1120 further gradually decreases the amount of change.

[0072] Figures 6(A) to 6(F) show the relationship between the image synthesis result generated by the image synthesis unit 1150 and the position and orientation of the input device 1200. Referring to Figures 6(A) to 6(F), an example of controlling the position or orientation of the UI, which is linked to the input device 1200, by changing the orientation (orientation) of the UI and adjusting the amount of change to be small will be explained. In Figures 6(A) to 6(F), the same numbers as in Figures 3(A) to 3(F) are used, and their explanations are omitted. Figures 6(A) to 6(C) and 6(F) show the same state as in Figures 3(A) to 3(C) and 3(F), so their explanations are omitted.

[0073] Figure 6(D) shows the input device 1200 entering the field of view of the imaging unit 1010 from outside its field of view. The control unit 1120 sets the starting point of the UI to a position based on the position and orientation of the input device 1200 as it was in Figure 6(D), without considering the amount of change. The control unit 1120 sets the amount of change so that the position that the UI was pointing to immediately before Figure 6(D) is maintained, and changes the UI3120 to display the UI6010.

[0074] Figure 6(E) shows the input device 1200 moving within the field of view of the imaging unit 1010 while the UI points to the same position as in Figure 6(D). By gradually reducing the amount of change, the control unit 1120 can display UI6040 so that it points to the same position as in Figure 6(D) within the virtual object 3020, while approaching UI6030, which is the position of the UI when the amount of change is not considered.

[0075] [Fourth Embodiment] In the first to third embodiments, examples were described in which all parts constituting the information processing device 1100 shown in Figure 1 are made of hardware. However, some parts may be made of software. In that case, by having a computer that implements the remaining parts as hardware run this software, the computer will perform the operation of the information processing device 1100 described in the first to third embodiments.

[0076] Figure 7 is a block diagram showing an example of a computer hardware configuration applicable to the information processing device 1100.

[0077] The CPU 7001 controls the entire computer using programs and data stored in the RAM 7002 and ROM 7003, and also executes the above-mentioned processes described in the first to third embodiments as processes performed by the information processing device 1100. The CPU 7001 can operate as a position and orientation calculation unit 1110, a control unit 1120, an input processing unit 1130, an image generation unit 1140, and an image synthesis unit 1150.

[0078] RAM 7002 has an area for temporarily storing programs and data loaded from the external storage device 7007 or the storage medium drive 7008. Furthermore, RAM 7002 has an area for temporarily storing data received from external devices via the I / F (interface) 7009. External devices refer to the display device 1000 and the input device 1200. In the configuration shown in Figure 1, the data received from external devices refers to images in real space and input values ​​generated by the input device 1200. Furthermore, RAM 7002 also has a work area used by the CPU 7001 when executing each process. In other words, RAM 7002 can provide various areas as appropriate. For example, RAM 7002 also functions as the data storage unit 1160 shown in Figure 1.

[0079] ROM7003 stores computer configuration data, boot programs, and other similar information.

[0080] The keyboard 7004 and mouse 7005 are examples of input devices, and can be used by the computer user to output various instructions to the CPU 7001.

[0081] The display unit 7006 is composed of a CRT (Cathode Ray Tube) or an LCD panel, and can display the processing results from the CPU 7001 as images or text. For example, the display unit 7006 can display messages that should be shown in order to measure the position and orientation of the display device 1000.

[0082] The external storage device 7007 is a storage device capable of storing large amounts of information, such as a hard disk drive. The external storage device 7007 stores the OS (operating system) and programs and data that cause the CPU 7001 to execute the above-mentioned processes, which are performed by the information processing device 1100. The programs include those corresponding to the position and orientation calculation unit 1110, the control unit 1120, the input processing unit 1130, the image generation unit 1140, and the image synthesis unit 1150. The data includes data from the virtual space and the information described above as known information. The programs and data stored in the external storage device 7007 are loaded into the RAM 7002 as appropriate, according to the control of the CPU 7001. The CPU 7001 executes the above-mentioned processes as performed by the information processing device 1100 by executing processing using these loaded programs and data. The external storage device 7007 may also be used as the data storage unit 1160 shown in Figure 1.

[0083] The storage medium drive 7008 reads programs and data recorded on storage media such as CD-ROMs and DVD-ROMs, and writes programs and data to such storage media. Some or all of the programs and data described as being stored on the external storage device 7007 may also be recorded on this storage medium. The programs and data read by the storage medium drive 7008 from the storage media are output to the external storage device 7007 and RAM 7002.

[0084] I / F7009 is an analog video connector for connecting the imaging unit 1010 of the display device 1000. It has an optoport or digital input / output port such as IEEE1394. I / F7009 also has an Ethernet® port for outputting a composite image to the display unit 1020 of the display device 1000. Data received via I / F7009 is output to RAM7002 or external storage device 7007. When a sensor system is used to acquire position and orientation, the sensor system is connected to I / F7009. Bus 7010 is a bus that connects the above-mentioned parts.

[0085] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0086] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0087] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0088] The above-disclosed embodiments include the following configurations, methods, programs, and media. (Composition 1) An acquisition means for acquiring information indicating the position or orientation of an object for manipulating a virtual object, A control means for controlling the position or orientation of a graphic corresponding to the object based on the information acquired by the acquisition means, It has, The control means is The first information is information indicating the position or orientation of the graphic in a first state in which the aforementioned information is acquired with a precision higher than a predetermined precision, Based on the second information, which is information indicating the position or orientation of the graphic in a second state in which the aforementioned information has not been acquired with a precision higher than the predetermined precision, Controlling the position or orientation of the graphic in the first state. An information processing device characterized by the following: (Configuration 2) The second state described above is a state in which the information described above has not been acquired. The information processing device according to configuration 1, characterized by the above. (Composition 3) The second state is a state in which the information is acquired with a precision lower than the predetermined precision. The information processing device according to configuration 1, characterized by the above. (Composition 4) When the control means transitions from the second state to the first state, it determines the position of the graphic in the first state or based on the first information and the second information. Controlling posture An information processing device according to any one of configurations 1 to 3. (Composition 5) When transitioning from the second state to the first state, if the difference between the first information and the second information is greater than a threshold, the control means controls the position or orientation of the graphic in the first state based on the first information and the second information. An information processing device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) When transitioning from the second state to the first state, the control means controls the position or orientation of the graphic in the first state based on the first information if the difference in position or orientation in a specific two-dimensional plane coordinate system between the graphic with a position or orientation corresponding to the first information and the graphic with a position or orientation corresponding to the second information changed by an amount of change based on the first and second information is greater than a threshold. An information processing apparatus according to any one of configurations 1 to 5. (Composition 7) When transitioning from the second state to the first state, if the user is using the graphics, the control means controls the position or orientation of the graphics in the first state based on the first information and the second information. An information processing device according to any one of configurations 1 to 6. (Composition 8) The use of the graphics by the user is defined as the user holding the object or the amount of movement of the object being greater than or equal to a threshold. The information processing apparatus according to configuration 7, characterized by the features described herein. (Composition 9) When transitioning from the second state to the first state, the control means changes the position or orientation of the graphic according to the first information by an amount of change based on the first information and the second information, thereby determining the position or orientation of the graphic in the first state. An information processing device according to any one of configurations 1 to 8. (Composition 10) When transitioning from the second state to the first state, the control means changes the position or orientation of the graphic according to the first information by an amount of change based on the difference between the first information and the second information. The information processing apparatus according to configuration 9, characterized by the features described therein. (Composition 11) When transitioning from the second state to the first state, the control means changes the position or orientation of the graphic according to the first information by an amount of change based on the difference between the location that the graphic was pointing to in the second state and the location that the graphic would point to if the position or orientation of the graphic were set to the position or orientation according to the first information. The information processing apparatus according to configuration 9, characterized by the features described therein. (Composition 12) The control means gradually reduces the amount of change. An information processing apparatus according to any one of the configurations 9 to 11, characterized by the above. (Composition 13) The control means controls the position or orientation of the graphic in the first state so that the location indicated by the graphic in the second state is maintained even after the transition from the second state to the first state. An information processing device according to any one of configurations 1 to 12, characterized by the above. (Composition 14) The information processing apparatus according to any one of configurations 1 to 13, characterized in that the control means controls the position or orientation of the graphic in the first state so as to gradually approach the position or orientation of the graphic corresponding to the first information. (Composition 15) The control means causes the display mode of the graphic to differ depending on whether the graphic is displayed on the display means in the first state at a position or orientation based on the first information and the second information, or at a position or orientation corresponding to the first information. An information processing apparatus according to any one of configurations 1 to 14, characterized by the features described herein. (Composition 16) The control means, in the first state, displays on the display means a graphic of position or orientation based on the first information and the second information, and a graphic of position or orientation corresponding to the first information, in a different display mode from the said graphic. An information processing apparatus according to any one of configurations 1 to 15, characterized by the above. (Composition 17) The acquisition means is capable of acquiring the information using the captured image, The first state is the state in which the object is visible in the image. The second state is a state in which the object is not visible in the image. An information processing device according to any one of configurations 1 to 16, characterized by the above. (Composition 18) The aforementioned object is a ring-shaped controller that can be worn on the user's finger. An information processing device according to any one of configurations 1 to 17, characterized by the features described herein. (Composition 19) The information processing device is a head-mounted display. An information processing apparatus according to any one of configurations 1 to 18, characterized by the above. (method) An acquisition step to obtain information indicating the position or orientation of an object for manipulating a virtual object, A control step that controls the position or orientation of the graphic corresponding to the object based on the information acquired in the acquisition step, It has, The control step is, The first information is information indicating the position or orientation of the graphic in a first state in which the aforementioned information is acquired with a precision higher than a predetermined precision, Based on the second information, which is information indicating the position or orientation of the graphic in a second state in which the aforementioned information has not been acquired with a precision higher than the predetermined precision, Controlling the position or orientation of the graphic in the first state. An information processing method characterized by the following: (program) A program for causing a computer to function as one of the means of an information processing device described in any one of items 1 to 19 of the configuration. (medium) A computer-readable storage medium containing a program for causing the computer to function as one of the means of the information processing device described in any one of items 1 to 19 of the configuration. [Explanation of symbols]

[0089] 1100: Information processing device 1110: Position and orientation calculation unit 1120: Control unit

Claims

1. An acquisition means for acquiring information indicating the position or orientation of an object for manipulating a virtual object, A control means that controls the position or orientation of the graphic corresponding to the manipulated object based on the information acquired by the acquisition means, It has, When the control means changes from a second state in which the manipulated object is not visible in the captured image to a first state in which the manipulated object is visible in the captured image, If the object indicated by the graphic in the second state is different from the object indicated by the graphic in the first state, The first information is information indicating the position or orientation of the manipulated object in the first state, Based on the second information, which is information indicating the position or orientation of the graphic in the second state, Control the position or orientation of the graphic in the first state, If the object indicated by the graphic in the second state is the same as the object indicated by the graphic in the first state, The position or orientation of the graphic in the first state is controlled based on the first information, not on the second information. An information processing device characterized by the following:

2. The second state is a state in which the information has not been acquired based on the captured image. The information processing apparatus according to feature 1.

3. The second state is a state in which the information is acquired with a precision lower than a predetermined precision. The information processing apparatus according to feature 1.

4. When transitioning from the second state to the first state, if the difference between the first information and the second information is greater than a threshold, the control means controls the position or orientation of the graphic in the first state based on the first information and the second information. The information processing apparatus according to feature 1.

5. When transitioning from the second state to the first state, the control means controls the position or orientation of the graphic in the first state based on the first information if the difference in position or orientation in a specific two-dimensional plane coordinate system between the graphic with a position or orientation corresponding to the first information and the graphic with a position or orientation corresponding to the second information changed by an amount of change based on the first and second information is greater than a threshold. The information processing apparatus according to feature 1.

6. When the control means transitions from the second state to the first state, If the user is using the graphic, the position or orientation of the graphic in the first state is controlled depending on whether the object pointed to by the graphic in the second state is the same as the object pointed to by the graphic in the first state. If the user is not using the graphics, the position or orientation of the graphics in the first state is controlled based on the first information, without relying on the second information. The information processing apparatus according to feature 1.

7. The case in which the user is using the graphics is when the user is holding the manipulated object or when the amount of movement of the manipulated object is greater than or equal to a threshold. The information processing apparatus according to feature 6.

8. When transitioning from the second state to the first state, if the object pointed to by the graphic in the second state is different from the object pointed to by the graphic in the first state, the control means changes the position or orientation of the graphic according to the first information by an amount of change based on the first information and the second information, thereby determining the position or orientation of the graphic in the first state. The information processing apparatus according to feature 1.

9. When transitioning from the second state to the first state, if the object pointed to by the graphic in the second state is different from the object pointed to by the graphic in the first state, the control means changes the position or orientation of the graphic according to the first information by an amount of change based on the difference between the first information and the second information. The information processing apparatus according to feature 8.

10. When transitioning from the second state to the first state, if the object pointed to by the graphic in the second state is different from the object pointed to by the graphic in the first state, the control means changes the position or orientation of the graphic according to the first information by an amount of change based on the difference between the location pointed to by the graphic in the second state and the location pointed to by the graphic when the position or orientation of the graphic is set to the position or orientation according to the first information. The information processing apparatus according to feature 8.

11. The control means gradually reduces the amount of change. The information processing apparatus according to feature 8.

12. The control means controls the position or orientation of the graphic in the first state so as to gradually approach the position or orientation of the graphic corresponding to the first information. The information processing apparatus according to feature 1.

13. The control means causes the display mode of the graphic to differ depending on whether the graphic is displayed on the display means in the first state at a position or orientation based on the first information and the second information, or at a position or orientation corresponding to the first information. The information processing apparatus according to feature 1.

14. The control means, in the first state, displays on the display means a graphic of position or orientation based on the first information and the second information, and a graphic of position or orientation corresponding to the first information, in a different display mode from the said graphic. The information processing apparatus according to feature 1.

15. The aforementioned object is a ring-shaped controller that can be worn on the user's finger. The information processing apparatus according to feature 1.

16. The information processing device is a head-mounted display. The information processing apparatus according to feature 1.

17. An acquisition step to obtain information indicating the position or orientation of an object to manipulate a virtual object, A control step that controls the position or orientation of the graphic corresponding to the manipulated object based on the information acquired in the acquisition step, It has, In the control step described above, when the operation object changes from a second state in which it is not visible in the captured image to a first state in which it is visible in the captured image, If the object indicated by the graphic in the second state is different from the object indicated by the graphic in the first state, The first information is information indicating the position or orientation of the manipulated object in the first state, Based on the second information, which is information indicating the position or orientation of the graphic in the second state, Control the position or orientation of the graphic in the first state, If the object indicated by the graphic in the second state is the same as the object indicated by the graphic in the first state, The position or orientation of the graphic in the first state is controlled based on the first information, not on the second information. An information processing method characterized by the following:

18. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 16.

19. A computer-readable storage medium storing a program for causing a computer to function as one of the means of the information processing apparatus described in any one of claims 1 to 16.

Images