Information processing device, information processing method, and program

Abstract

To achieve control of the operation of virtual objects favorable for a user in moving a virtual viewpoint in a space including the virtual objects.SOLUTION: An information processing apparatus has: acquisition means that acquires the position and attitude of a virtual viewpoint and the positions and attitudes of one or more virtual objects; moving means that moves the virtual viewpoint; control means that controls whether to interlock each of the one or more virtual objects with the movement of the virtual viewpoint based on if the virtual object satisfies a predetermined condition; and creation means that creates a virtual image including the one or more controlled virtual objects at the moved viewpoint.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for presenting a virtual object. [Background technology]

[0002] In recent years, there has been active development of virtual reality (VR) and mixed reality (MR) systems, which aim to seamlessly combine real and virtual spaces. Furthermore, in VR and MR systems, when an observer actually moves to a different location within a virtual space, it is necessary to move each piece of equipment and secure a real space equivalent to the virtual space. Therefore, a position movement instruction means for separately specifying the position movement has conventionally been provided. For example, a direction is specified using a position and orientation sensor, and then a movement in that direction is instructed by pressing a separately provided button, or a position movement is instructed using a device such as a joystick. This type of observation, in which the observer freely moves to a different location within a virtual three-dimensional space, is called a walk-through. To solve the problem of it becoming difficult to know where one is when continuously moving using the walk-through function, Patent Document 1 discloses the provision of a full-size observation mode and a reduced-size observation mode.

[0003] On the other hand, in VR and MR systems, virtual objects are not only assigned to a world coordinate system and displayed, but also assigned to a real object whose position and orientation can be measured, for example. When a user operates a real object, the display of a virtual object assigned to that real object (hereinafter referred to as a virtual object on a real object) follows the user's operation, so that the VR and MR systems can provide the user with intuitive operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-79174 Summary of the Invention [Problem to be solved by the invention]

[0005] When using a walk-through function in a VR or MR system, if a virtual object is assigned to a real object whose position and orientation can be measured, various virtual object behaviors are possible that are suitable for the user. For example, if a user wants to check a virtual object in a world coordinate system and a virtual object on a real object from different viewpoints, it is desirable to maintain the relative position and orientation relationship between the virtual object on the world coordinate system and the virtual object on the real object when the virtual viewpoint is moved using the walk-through function. In other words, it is desirable for the virtual object to move in conjunction with the walk-through function. Also, for example, if a user holds and freely manipulates a real object, it is desirable for the virtual object on the real object to not be affected by the movement of the virtual viewpoint when the virtual viewpoint is moved using the walk-through function. In other words, it is desirable for the virtual object to move independently of the walk-through function. However, in the past, it was not possible to control the behavior of a virtual object in a way that was suitable for the user.

[0006] Therefore, an object of the present invention is to realize movement control of a virtual object that is suitable for the user when moving a virtual viewpoint in a space including the virtual object. [Means for solving the problem]

[0007] The information processing device of the present invention includes an acquisition means for acquiring the position and orientation of a virtual viewpoint and the positions and orientations of one or more virtual objects, a movement means for moving the virtual viewpoint using a walk-through function, a control means for controlling whether each of the one or more virtual objects is linked to the movement of the virtual viewpoint based on whether a predetermined condition is satisfied, and a generation means for generating a virtual image including the controlled one or more virtual objects at the moved viewpoint. At least one of the one or more virtual objects is associated with a real object. It is characterized by: [Effects of the Invention]

[0008] According to the present invention, when a virtual viewpoint is moved in a space including a virtual object, it is possible to realize movement control of the virtual object that is suitable for the user. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the configuration of a system including an information processing device according to a first embodiment. [Figure 2] 4 is a flowchart showing information processing according to the first embodiment. [Figure 3] FIG. 2 is a diagram for explaining information processing according to the first embodiment. [Figure 4] FIG. 10 is a diagram for explaining information processing according to the second embodiment. [Figure 5] FIG. 1 is a diagram illustrating an example of a hardware configuration in which an information processing device can be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [Embodiment 1] FIG. 1 shows an example of the configuration of a system including an information processing device according to this embodiment. 1 is a diagram showing an example of the configuration of a system according to this embodiment. The system according to this embodiment is an image processing system for presenting a mixed reality space (MR space) that combines a real space and a virtual space to a person experiencing the system (hereinafter referred to as a user). In this embodiment, a case will be described in which an MR space is presented to a user by displaying a composite image that combines an image of a real space with an image of a virtual space rendered by computer graphics (CG).

[0012] The system according to this embodiment includes a display device 100, an information processing device 110, and an input device 120. The information processing device 110 synthesizes an image of real space captured from the display device 100 with an image of virtual space generated within the information processing device 110, and outputs the synthesized image to the display device 100 as a mixed reality image (MR image). Note that the present invention relates to an information processing device that displays an image of virtual space, and is not limited to an MR system that displays an MR image that combines an image of real space and an image of virtual space. In other words, the present invention can also be applied to a VR (virtual reality) system that presents only an image of virtual space to a user, and an AR (augmented reality) system that presents an image of virtual space to a user by transmitting the image through real space.

[0013] The display device 100 includes an imaging unit 101. The imaging unit 101 captures images of real space continuously in time series and outputs the captured images of real space to the information processing device 110. The imaging unit 101 may be configured to include a stereo camera consisting of two cameras fixed to each other so as to be able to capture images of real space in the line of sight from the viewpoint position of the user. The display device 100 also includes a display unit 102. The display unit 102 displays the MR image output from the information processing device 110. The display unit 102 may be configured to include two displays arranged corresponding to the left and right eyes of the user. In this case, the display corresponding to the user's left eye displays the MR image for the left eye, and the display corresponding to the user's right eye displays the MR image for the right eye.

[0014] The display device 100 may be, for example, a head-mounted display device (HMD). However, the display device 100 is not limited to an HMD and may also be a handheld display (HHD). An HHD is a handheld display. In other words, it may be a display that a user holds in their hand and peers into like binoculars to observe images. Furthermore, the display device 100 may also be a display terminal such as a tablet or a smartphone.

[0015] The input device 120 includes an input unit 121. The input unit 121 is a controller that can input movement instructions for directions and positions from the user, and inputs an input value corresponding to the input value to the information processing device 110. The input device 120 is a joystick that is a control stick-like device that can obtain a value corresponding to the tilt direction of the stick. However, the input device 120 is not limited to a joystick, and can be replaced by a device such as a keyboard or an input device for video games.

[0016] The information processing device 110 and the display device 100, and the information processing device 110 and the input device 120 are connected to each other so that data communication is possible between them. The connection between the information processing device 110 and the display device 100, and the connection between the information processing device 110 and the input device 120 may be either wired or wireless. The information processing device 110 may be disposed inside the housing of the display device 100.

[0017] The information processing device 110 includes a position and orientation calculation unit 111, a viewpoint movement processing unit 112, a control unit 113, an image generation unit 114, an image synthesis unit 115, and a data storage unit .

[0018] The position and orientation calculation unit 111 calculates the position and orientation of the image capture unit 101 in the world coordinate system and the position and orientation of a physical object in which a coordinate system (individual coordinate system) separate from the world coordinate system is set. Specifically, the position and orientation calculation unit 111 extracts markers assigned to the world coordinate system and markers assigned to the individual coordinate system from an image of physical space captured by the image capture unit 101. The markers assigned to the individual coordinate system are attached to operable physical objects. Then, based on the position and orientation of the markers, the position and orientation of the image capture unit 101 in the world coordinate system and the position and orientation of the physical object in the individual coordinate system are calculated. Then, the calculated position and orientation information is stored in the data storage unit 116. In this embodiment, the position and orientation calculation unit 111 acquires the position and orientation of the virtual viewpoint immediately before movement of the virtual viewpoint occurs by calculating the position and orientation of the image capture unit 101. Note that instead of the position and orientation of the image capture unit 101, a position and orientation that serves as a reference on the world coordinate system may be acquired. In this embodiment, a virtual object is associated with a physical object for which an individual coordinate system is set. The position and orientation calculation unit 111 calculates the position and orientation of the physical object in the individual coordinate system, thereby obtaining the position and orientation of the virtual object immediately before the movement of the virtual viewpoint occurs.

[0019] Note that there are various methods for calculating the position and orientation of the image capture unit 101 and the position and orientation of a physical object in a specific coordinate system, and the method is not limited to the above. For example, the position and orientation of the image capture unit 101 and the position and orientation of a physical object in a specific coordinate system may be calculated by performing Simultaneous Localization and Mapping (SLAM) processing based on feature points captured in an image. Alternatively, the position and orientation of the image capture unit 101 may be calculated by attaching a sensor whose relative position and orientation with respect to the image capture unit 101 is known to the HMD serving as the display device 100 and converting the measurement value of the sensor based on the relative position and orientation. Similarly, the position and orientation of an operable physical object in a specific coordinate system may be calculated by attaching a sensor whose relative position and orientation with respect to the physical object is known to the physical object. A motion capture system may also be used.

[0020] The viewpoint movement processing unit 112 calculates the movement amount of the virtual viewpoint based on an input value from the input unit 121, and performs processing to move the position and orientation of the virtual viewpoint based on the calculated movement amount of the virtual viewpoint. The virtual viewpoint refers to the viewpoint of the image in virtual space generated by the image generation unit 114. In this embodiment, the position and orientation of the virtual viewpoint are calculated by adding the calculated movement amount of the virtual viewpoint to the position and orientation of the imaging unit 101 calculated by the position and orientation calculation unit 111. The viewpoint movement processing unit 112 uses the input value from the input unit 121 to calculate, as the movement amount of the virtual viewpoint, the movement amount obtained by moving the position of the virtual viewpoint in each direction of three mutually perpendicular axes, or the movement amount obtained by rotating the orientation of the virtual viewpoint in each direction of three mutually perpendicular axes. The calculation results of the position and orientation of the virtual viewpoint and the calculation results of the movement amount of the virtual viewpoint are then stored in the data storage unit 116.

[0021] The control unit 113 performs processing for determining whether or not to link each of the real objects for which an individual coordinate system is set with the movement of the virtual viewpoint by the viewpoint movement processing unit 112. If the control unit 113 determines to link the real objects, the control unit 113 updates the position and orientation of the real object calculated by the position and orientation calculation unit 111 based on the amount of movement of the virtual viewpoint, and stores the updated position and orientation as the position and orientation of the virtual object in the data storage unit 116. If the control unit 113 determines not to link the real objects, the control unit 113 stores the position and orientation of the real object calculated by the position and orientation calculation unit 111 as is in the data storage unit 116 as the position and orientation of the virtual object.

[0022] The image generation unit 114 constructs a virtual space based on the virtual space data stored in the data storage unit 116. The virtual space data includes data related to each virtual object that constitutes the virtual space, data related to a light source that illuminates the virtual space, and the like. The image generation unit 114 sets the position and orientation of the virtual viewpoint calculated by the viewpoint movement processing unit 112 and stored in the data storage unit 116. The image generation unit 114 also arranges virtual objects in the virtual space in correspondence with the positions and orientations of the virtual objects having individual coordinate systems that are stored in the data storage unit 116. The image generation unit 114 then generates an image (virtual image) of the virtual space that can be seen from the viewpoint with the set position and orientation. Note that the technology for generating a virtual image that can be seen from the viewpoint with a predetermined position and orientation is well known, and therefore a detailed description thereof will be omitted.

[0023] The image synthesis unit 115 generates an MR image by synthesizing the virtual image generated by the image generation unit 114 and the image in real space captured by the imaging unit 101. Then, the generated MR image is output to the display device 100.

[0024] As described above, data storage unit 116 is used to store various pieces of information, and is configured with RAM, a hard disk drive, etc. In addition to the information described above as being stored in data storage unit 116, data storage unit 116 also stores what will be described in this embodiment as known information.

[0025] 2 is a flowchart showing information processing by the information processing device 110 for generating an MR image and outputting it to the display device 100. It is assumed that there are multiple manipulable real objects in the real space, and individual coordinate systems are set for each of the multiple real objects. In addition, the position and orientation of a virtual object are associated with the position and orientation of the real object for which the individual coordinate system is set.

[0026] First, in step S201, the position and orientation calculation unit 111 calculates the position and orientation of the image capture unit 101 and the position and orientation of the physical object in the individual coordinate system. Next, in step S202, the viewpoint movement processing unit 112 calculates the movement amount of the virtual viewpoint based on the input value from the input unit 121. Next, in step S203, the viewpoint movement processing unit 112 calculates the position and orientation of the virtual viewpoint by adding the movement amount of the virtual viewpoint calculated in step S202 to the position and orientation of the image capturing unit 101 calculated in step S201.

[0027] Next, in step S204, the control unit 113 sets the value k specifying the individual coordinate system of the kth physical object to 1 as an initial setting. Then, in step S205, the control unit 113 compares the value k with the total number of set individual coordinate systems. If the value k is equal to or less than the total number of individual coordinate systems, the process proceeds to step S206, and if the value k is greater than the total number of individual coordinate systems, the process proceeds to step S210. Next, in step S206, the control unit 113 determines whether the physical object for which the kth individual coordinate system is set satisfies a predetermined condition. If the control unit 113 determines that the predetermined condition is satisfied, the process proceeds to step S207, and if the control unit 113 determines that the predetermined condition is not satisfied, the process proceeds to step S208.

[0028] In step S207, the control unit 113 controls the position and orientation of the virtual object associated with the physical object for which the kth individual coordinate system is set, calculated in step S201, to move in conjunction with the movement of the virtual viewpoint. Specifically, the position and orientation of the virtual object is acquired by updating the position and orientation of the physical object in the kth individual coordinate system using the movement amount of the virtual viewpoint calculated in S202. This maintains the relationship between the position and orientation of the virtual viewpoint and the position and orientation of the virtual object associated with the physical object. In step S208, the control unit 113 controls the position and orientation of the virtual object associated with the physical object for which the kth individual coordinate system is set, calculated in step S201, so that it is not linked to the movement of the virtual viewpoint. Specifically, the position and orientation of the physical object in the kth individual coordinate system is acquired as is as the position and orientation of the virtual object. This maintains the relationship between the position and orientation of the image capture unit 101 and the position and orientation of the virtual object associated with the physical object. After steps S207 and S208, the process proceeds to step S209. In step S209, the control unit 113 increments the value of k by 1. Thereafter, the process returns to step S205.

[0029] Next, in step S210, the image generation unit 114 generates a virtual image using the position and orientation of the virtual viewpoint calculated in step S203 and the positions and orientations of the virtual object calculated in steps S207 and S208. Next, in step S211, the image synthesis unit 115 synthesizes the image in real space captured by the imaging unit 101 with the virtual image generated in step S210 to generate an MR image. Then, the generated MR image is output to the display device 100. Next, in step S212, the information processing device 110 determines whether an instruction to end this flowchart has been input, and if it determines that an instruction to end this flowchart has been input, the information processing device 110 ends the processing of this flowchart. On the other hand, unless an instruction to end this flowchart has been input, the processing returns to S201. Note that in this step, it may also be determined whether a condition for ending this flowchart has been met.

[0030] There are various possible conditions for determining whether or not a virtual object having an individual coordinate system should be linked to the movement processing of the virtual viewpoint, and one example will be described below. 3 shows an MR image as a result of image synthesis generated by the image synthesis unit 115. Using FIG. 3, the control of whether to synchronize or not synchronize the movement processing of the virtual viewpoint for a virtual object having an individual coordinate system will be described.

[0031] Real objects 302, 303, 304, and 305 are objects that exist in real space and can be operated by the user. In the example of FIG. 3, real object 302 is held in user hand 301. Marker 306 is a marker assigned to the world coordinate system. Markers 307, 308, 309, and 310 are markers assigned to individual coordinate systems. Virtual object 311 is an object that exists in virtual space and is assigned to the world coordinate system. Virtual objects 312, 313, 314, and 315 are virtual objects associated with real objects 302, 303, 304, and 305 to which markers 307, 308, 309, and 310 are assigned. In other words, individual coordinate systems are set for each of virtual objects 312, 313, 314, and 315. Arrow 316 in FIG. 3(B) schematically indicates the amount of movement of the virtual viewpoint. 3(B) shows that the virtual viewpoint has moved in the direction of arrow 316, resulting in a movement of the display position of virtual object 311. Arrows 317, 318, and 319 show that the positions and orientations of virtual objects 312, 313, 314, and 315 have moved in conjunction with the amount of movement of the virtual viewpoint. Note that arrows 316, 317, 318, and 319 are additional notations for the purpose of explaining this embodiment and are not actually displayed on the virtual image. Furthermore, the number of real objects to which markers assigned to individual coordinate systems are attached is not particularly limited as long as it is one or more.

[0032] 3(A) shows a state where there is no input from the input unit 121 and no movement of the virtual viewpoint. When there is no movement of the virtual viewpoint, a virtual image is generated by setting the position and orientation of the image capturing unit 101 calculated by the position and orientation calculation unit 111 as the position and orientation of the virtual viewpoint and setting the positions and orientations of the physical objects 302 to 305 calculated by the position and orientation calculation unit 111 as the positions and orientations of the virtual objects 312 to 315. Then, an MR image is generated by combining the virtual image with the image of the real space of the image capturing unit 101.

[0033] 3B shows a state in which movement of the virtual viewpoint occurs due to an input from the input unit 121. When movement of the virtual viewpoint occurs, the virtual object 311 on the world coordinate system moves in the direction of the arrow 316. In this embodiment, the relative distance between the image capture unit 101 and each of the real objects 302 to 305 is used as a predetermined condition for determining whether to link the position and orientation of the virtual object having the individual coordinate system to the movement of the virtual viewpoint. If the relative distance is equal to or greater than a threshold, the control unit 113 links the position and orientation of the virtual object on the individual coordinate system to the amount of movement of the virtual viewpoint, and if the relative distance is less than the threshold, the control unit 113 does not link the position and orientation of the virtual object on the individual coordinate system to the amount of movement of the virtual viewpoint. In the above description, the relative distance between the image capturing unit 101 and each of the physical objects 302 to 305 is used as the predetermined condition, but the relative distance between the image capturing unit 101 and each of the virtual objects 312 to 315 immediately before the movement of the virtual viewpoint occurs may also be used. Furthermore, the predetermined condition is not limited to the relative distance, but a relative angle may also be used, or both the distance and the angle may also be used.

[0034] 3B, the positions and orientations of virtual objects 313-315 associated with physical objects 303-305 whose relative distances to the image capture unit 101 are determined to be equal to or greater than a threshold are linked to the amount of movement of the virtual viewpoint. When the information processing device 110 determines that the relative distances are equal to or greater than a threshold, it controls the display positions of virtual objects 313-315 associated with the physical objects 303-305 to move in the directions of arrows 317-319, respectively. 3B, the position and orientation of the virtual object 312 associated with the physical object 302 whose relative distance to the image capture unit 101 is determined to be less than the threshold value is not linked to the movement of the virtual viewpoint. When the information processing device 110 determines that the relative distance is less than the threshold value, it controls the display position of the virtual object 312 associated with the physical object 302 so that it is not linked to the movement of the virtual viewpoint. In other words, the virtual object 312 is displayed at the same position as when the virtual viewpoint is not moved.

[0035] As described above, the information processing device 110 controls whether to link or not link the movement processing of the virtual viewpoint for virtual objects having individual coordinate systems. As a result, when a movement of the virtual viewpoint occurs, virtual objects that are linked to the movement amount of the virtual viewpoint can be viewed from a different viewpoint after the movement amount of the virtual viewpoint is reflected, along with virtual objects in the world coordinate system. On the other hand, virtual objects that are not linked to the movement amount of the virtual viewpoint are displayed in the same location as when there is no movement amount of the virtual viewpoint. As a result, a user who holds and operates a real object linked to a virtual object can continue operating the object in the same way as before the movement of the virtual viewpoint, even from a different viewpoint after the movement amount of the virtual viewpoint is reflected. As described above, when a movement of the virtual viewpoint occurs, it is possible to present to the user a virtual image generated by appropriately combining whether or not to link the movement amount of the virtual viewpoint for each virtual object having an individual coordinate system.

[0036] According to the information processing device 110 according to this embodiment as described above, when the virtual viewpoint moves in a space including a virtual object, it is possible to realize movement control of the virtual object that is suitable for the user.

[0037] In the above embodiment, the relative distance between the image capture unit 101 and the real object associated with the virtual object is used as the predetermined condition used to determine whether or not to link the position and orientation of a virtual object having an individual coordinate system to the movement of the virtual viewpoint. However, the present invention is not limited to using relative distance. Information regarding the position and orientation of a virtual object having an individual coordinate system and information regarding its movement may also be used. Specifically, the information processing device 110 analyzes, in time series, the positions of markers assigned to the world coordinate system and markers assigned to the individual coordinate system in an image in real space to acquire the relative movement speed and relative movement trajectory between the virtual viewpoint and the virtual object. The acquired relative movement speed and relative movement trajectory may then be used as the predetermined condition. Note that the movement speed and movement trajectory of the virtual object may also be used as the predetermined condition.

[0038] Furthermore, the contact state, including whether the user is holding or wearing a real object, may be used as a predetermined condition. Specifically, the information processing device 110 detects whether the user is holding or wearing a real object, and if the real object is being held or worn, controls the virtual object associated with the held or worn real object so as not to be linked with the movement of the virtual viewpoint. On the other hand, if the real object is not being held or worn, controls the virtual object associated with the not-held or not-worn real object so as to be linked with the movement process. The detection of the user's holding or wearing may be performed using a pressure sensor or a contact sensor attached to the real object. Alternatively, a detection sensor may be provided to detect the position and orientation of the user's hand and the position and orientation of the real object, and the information processing device 110 may acquire the contact state of the user's hand with the real object based on the time-series detection results of the detection sensor. Alternatively, instead of using a detection sensor, the information processing device 110 may detect the contact state of the user's hand with the real object by analyzing images captured by the imaging unit 101. The part of the user that comes into contact with the real object is not limited to the hand, but may be any predetermined part of the user that can come into contact with the real object.

[0039] In the above embodiment, an individual coordinate system is set for each virtual object, but it is not necessary to set an individual coordinate system. In this case, the information processing device 110 may control whether or not to link the movement process of the virtual viewpoint with the position and orientation information of the virtual object on the world coordinate system.

[0040] Furthermore, although the above embodiment has been described with respect to a single user, the present invention may be implemented by multiple users. In this case, the positions and orientations of the virtual objects, the virtual viewpoint, and the real objects on the world coordinate system are shared among the multiple users, and when the virtual viewpoint is moved for each user, the linkage / non-linkage control is performed for each user. In addition to the positions and orientations of the virtual objects, the virtual viewpoint, and the real objects on the world coordinate system, the movement amount of the virtual viewpoint and the linkage / non-linkage control results of the virtual viewpoint movement process for virtual objects having individual coordinate systems may also be shared among the multiple users.

[0041] [Embodiment 2] In this embodiment, a method for changing the display mode of a virtual object before and after the movement of the virtual viewpoint when the virtual object is not linked to the movement process of the virtual viewpoint will be described. For example, when the position and orientation of a virtual object are not linked, the information processing device 110 controls the virtual object so that another virtual object is duplicated and displayed at the display position of the virtual object.

[0042] Fig. 4 shows an MR image as an image synthesis result generated by the image synthesis unit 115. The same parts as in Fig. 3 are assigned the same numbers, and their explanation will be omitted. FIG. 4(A) shows the same state as FIG. 3(A). FIG. 4B, like FIG. 3B, illustrates a state in which movement of the virtual viewpoint occurs due to an input value from the input unit 121. In this embodiment, the information processing device 110 duplicates and displays a preset virtual object or a virtual object designated by the user, corresponding to the position and orientation of the virtual object determined not to be linked to movement of the virtual viewpoint. In this embodiment, as in the first embodiment, the relative distance between the imaging unit 101 and the physical object associated with the virtual object is used as a predetermined condition for determining whether to link a virtual object in an individual coordinate system to movement of the virtual viewpoint. If the relative distance is equal to or greater than a threshold, the information processing device 110 links the position and orientation of the virtual object in the individual coordinate system to movement of the virtual viewpoint, and if the relative distance is less than the threshold, the information processing device 110 does not link the position and orientation of the virtual object in the individual coordinate system to movement of the virtual viewpoint.

[0043] In the example of FIG. 4B , similar to the case of FIG. 3B , a virtual object 312 associated with a physical object 302 whose relative distance from the image capture unit 101 is less than a threshold is determined to be unlinked to the movement of the virtual viewpoint. When the information processing device 110 determines that the relative distance is less than the threshold, it controls the position and orientation of the virtual object 312 on the physical object 302 so that it is not linked to the movement of the virtual viewpoint. In this embodiment, the information processing device 110 further duplicates a virtual object 315, which is one of the virtual objects in the individual coordinate system that is linked to the movement of the virtual viewpoint, and displays the duplicated virtual object 315 in place of the virtual object 312. That is, the virtual object 312 on the physical object 302 is changed to a virtual object 401 and displayed in the same position as when the virtual viewpoint is not moved. As a result, when the virtual viewpoint is moved and observation is performed from a different virtual viewpoint, another virtual object is copied and newly associated with the physical object held by the user. This allows the user to hold the same physical object and use it for a new purpose.

[0044] According to the information processing device 110 of this embodiment as described above, similar to the first embodiment, when moving the virtual viewpoint in a space including the virtual object, it is possible to realize movement control of the virtual object that is suitable for the user.

[0045] In the above embodiment, a virtual object on an individual coordinate system that is linked to the movement of the virtual viewpoint is replicated, but a virtual object on a world coordinate system may also be replicated, or a virtual object on an individual coordinate system that is not linked to the movement of the virtual viewpoint may also be replicated.

[0046] [Hardware configuration example] 5 is a diagram showing the hardware configuration of a computer capable of implementing the information processing device 110 in each of the above-described embodiments. Each unit constituting the information processing device 110 in each of the above-described embodiments may be configured as separate hardware, or may be realized by a program executed by a CPU 501. 5, the computer serving as the information processing device 110 includes a CPU 501, a RAM 502, a ROM 503, a mouse 504, a keyboard 505, a display 506, an external storage device 507, a storage medium drive 508, and an I / F 509. These devices are connected to each other via a bus 510. The CPU 501 controls the information processing device 110. The RAM 502 temporarily stores data loaded from the external storage device 507 or the storage medium drive 508, as well as data required for information processing. The RAM 502 also functions as the data storage unit 116 shown in FIG. 1 . The RAM 502 also temporarily stores data received from external devices, such as the display device 100 or the input device 120, via the I / F 509. The data received from external devices includes images of real space captured by the imaging unit 101 and input values ​​generated by the input device 120. The RAM 502 also functions as a work area for the CPU 501. The ROM 503 stores computer setting data, boot programs, and the like. The mouse 504 and keyboard 505 are examples of operation input devices, and are operated by a computer user to input various instructions to the CPU 501. The display 506 is composed of a CRT, LCD screen, or the like, and displays the results of processing by the CPU 501 as images, text, or the like. For example, the display 506 can display messages and the like that should be displayed for measuring the position and orientation of the display device 100 .

[0047] The external storage device 507 is a large-capacity information storage device, such as a hard disk drive. The external storage device 507 stores an operating system (OS), programs executable by the information processing device 110, and various data. The programs and various data stored in the external storage device 507 are loaded into the RAM 502 as needed under the control of the CPU 501. The CPU 501 loads and executes programs stored in the external storage device 507 into the RAM 502, causing the computer to function as the position and orientation calculation unit 111, the viewpoint movement processing unit 112, the control unit 113, the image generation unit 114, and the image synthesis unit 115. The CPU 501 loads and executes programs stored in the external storage device 507 into the RAM 502, thereby implementing the processing shown in the flowchart of FIG. 2. The various data stored in the external storage device 507 include data for constructing a virtual space and data described as known information in each embodiment. The external storage device 507 may function as the data storage unit 116 shown in FIG. 1.

[0048] The storage medium drive 508 loads programs and data recorded on a storage medium such as a CD-ROM or a DVD-ROM, and writes programs and data to the storage medium. Note that some or all of the programs and data described as being stored in the external storage device 507 may be recorded on a storage medium. The programs and data loaded from the storage medium by the storage medium drive 508 are output to the external storage device 507 or RAM 502. The I / F 509 includes an analog video port for connecting the imaging unit 101 of the display device 100, a digital input / output port such as IEEE 1394, an Ethernet (registered trademark) port for outputting a composite image to the display unit 102 of the display device 100, and the like. Data received via the I / F 509 is input to the RAM 502 or the external storage device 507. Furthermore, when a sensor system is used to acquire position and orientation information of a virtual viewpoint or a real object, the sensor system is connected to the I / F 509.

[0049] [Other embodiments] The present invention also encompasses cases in which a software program is supplied directly or remotely to a system or device, and the computer of the system or device reads and executes the supplied program code to achieve the functions of the aforementioned embodiments. In this case, the supplied program is a computer-readable program corresponding to the flowcharts illustrated in the embodiments. Furthermore, the functions of the aforementioned embodiments may be realized by the computer executing the read program. Furthermore, the functions of the aforementioned embodiments may also be realized in cooperation with an operating system (OS) running on the computer based on instructions from the program. In this case, the OS (OS) performs some or all of the actual processing, and the functions of the aforementioned embodiments are realized through that processing. [Explanation of symbols]

[0050] 100: Display device, 110: Information processing device, 120: Input device

Claims

1. an acquisition means for acquiring the position and orientation of a virtual viewpoint and the position and orientation of one or more virtual objects; a moving means for moving the virtual viewpoint by using a walk-through function; a control means for controlling whether each of the one or more virtual objects is linked to the movement of the virtual viewpoint based on whether a predetermined condition is satisfied; a generating means for generating a virtual image including the one or more controlled virtual objects at the moved viewpoint; and At least one of the one or more virtual objects is associated with a real object.

1. An information processing device comprising:

2. The control means controlling the virtual object that satisfies the predetermined condition so that it moves in conjunction with the movement of the virtual viewpoint; The virtual object that does not satisfy the predetermined condition is controlled so as not to move in conjunction with the movement of the virtual viewpoint.

2. The information processing apparatus according to claim 1, wherein:

3. An acquisition means for acquiring the position and orientation of a virtual viewpoint and the positions and orientations of one or more virtual objects; a moving means for moving the virtual viewpoint by using a walk-through function; a control means for controlling whether each of the one or more virtual objects is linked to the movement of the virtual viewpoint based on whether a predetermined condition is satisfied; a generating means for generating a virtual image including the one or more controlled virtual objects at the moved viewpoint; and The control means controls the virtual object that is not linked to the movement of the virtual viewpoint to be changed to another virtual object.

1. An information processing device comprising:

4. further comprising a duplication means for duplicating the virtual object, 4. The information processing apparatus according to claim 3, wherein the control means replaces the virtual object that is not linked to the movement of the virtual viewpoint with another duplicated virtual object.

5. An acquisition means for acquiring the position and orientation of a virtual viewpoint and the position and orientation of one or more virtual objects; a moving means for moving the virtual viewpoint by using a walk-through function; a control means for controlling whether each of the one or more virtual objects is linked to the movement of the virtual viewpoint based on whether a predetermined condition is satisfied; a generating means for generating a virtual image including the one or more controlled virtual objects at the moved viewpoint; and The predetermined conditions are: At least one of a relative distance and a relative angle between the virtual viewpoint and the virtual object; Information about the movement of the virtual object, or At least one of a relative movement speed and a movement trajectory between the virtual viewpoint and the virtual object An information processing device characterized by using the above.

6. 6. The information processing apparatus according to claim 1, wherein at least one of the position and the orientation of the virtual object is used as the predetermined condition.

7. 7. The information processing apparatus according to claim 1, wherein information relating to movement of the virtual object is used as the predetermined condition.

8. 8. The information processing apparatus according to claim 1, wherein at least one of a relative moving speed and a moving trajectory between the virtual viewpoint and the virtual object is used as the predetermined condition.

9. a relative distance between the virtual viewpoint and the virtual object is used as the predetermined condition; The control means a virtual object whose relative distance to the virtual viewpoint is less than a threshold is controlled not to move in conjunction with the movement of the virtual viewpoint; The virtual object whose relative distance to the virtual viewpoint is equal to or greater than a threshold is controlled to move in conjunction with the movement of the virtual viewpoint.

6. The information processing apparatus according to claim 5,

10. a detection unit for detecting a contact state of a predetermined part of a user observing the virtual image with the real object; 2. The information processing apparatus according to claim 1, wherein the predetermined condition is a contact state of the predetermined portion with the physical object.

11. The control means When the predetermined part of the user is in contact with the real object, the virtual object associated with the real object is controlled so as not to move in conjunction with the movement of the virtual viewpoint; When the predetermined part of the user is not in contact with the real object, the virtual object associated with the real object is controlled to move in conjunction with the movement of the virtual viewpoint.

11. The information processing apparatus according to claim 10,

12. 12. The information processing device according to claim 1, wherein the moving means calculates the amount of movement of the virtual viewpoint based on a value input from an input device capable of inputting direction and / or position instructions from a user observing the virtual image.

13. further comprising image acquisition means for acquiring an image of real space; The generating means generates a mixed reality image by combining the image of the one or more virtual objects with the image of the real space.

13. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

14. an acquisition step of acquiring a position and orientation of a virtual viewpoint and a position and orientation of one or more virtual objects; a moving step of moving the virtual viewpoint by using a walk-through function; a control step of controlling whether to link each of the one or more virtual objects with the movement of the virtual viewpoint based on whether a predetermined condition is satisfied; generating a virtual image including the controlled one or more virtual objects at the moved viewpoint; Including, At least one of the one or more virtual objects is associated with a real object. An information processing method comprising:

15. An acquisition step of acquiring a position and orientation of a virtual viewpoint and a position and orientation of one or more virtual objects; a moving step of moving the virtual viewpoint by using a walk-through function; a control step of controlling whether to link each of the one or more virtual objects with the movement of the virtual viewpoint based on whether a predetermined condition is satisfied; generating a virtual image including the controlled one or more virtual objects at the moved viewpoint; Including, The control step controls the virtual object that is not linked to the movement of the virtual viewpoint to be changed to another virtual object. An information processing method comprising:

16. An acquisition step of acquiring a position and orientation of a virtual viewpoint and a position and orientation of one or more virtual objects; a moving step of moving the virtual viewpoint by using a walk-through function; a control step of controlling whether to link each of the one or more virtual objects with the movement of the virtual viewpoint based on whether a predetermined condition is satisfied; generating a virtual image including the controlled one or more virtual objects at the moved viewpoint; Including, The predetermined conditions are: At least one of a relative distance and a relative angle between the virtual viewpoint and the virtual object; Information about the movement of the virtual object, or At least one of a relative movement speed and a movement trajectory between the virtual viewpoint and the virtual object An information processing method characterized by using the above.

17. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 13.

Images