Information processing apparatus, information processing system, control method, storage medium, and program product

Abstract

The invention provides an information processing apparatus, an information processing system, a control method, a storage medium, and a program product. Provided is an information processing device capable of easily switching and displaying a real image and a dummy image. The position obtaining unit obtains a current position of a user. The presentation unit presents a position including the current position using a display device. The specifying unit specifies a position from among the positions in accordance with an instruction from a user. In a case where the specified position is the current position, the first display control unit displays, on the display device, an image obtained by capturing the real space by the imaging device as a real image indicating a field of view of the user at the current position. In a case where the specified position is not the current position, the generation unit generates a dummy image similar to a field of view of the user at the specified position. The second display control unit displays the dummy image on the display device.

Description

Technical Field

[0001] This disclosure relates to information processing equipment, information processing systems, and control methods and programs for information processing equipment. Background Technology

[0002] In recent years, MR (Mixed Reality) technology has seen significant innovation. Furthermore, technological innovation has led to the widespread adoption of MR devices such as HMD (Head-Mounted Display). HMDs are used in various MR content, such as sharing completed images on construction sites and showcasing new products in showrooms using CG (computer graphics). In MR content that provides this mixed reality space, users primarily move by walking when they want to change the position of the virtual object they are viewing. However, from the user's perspective, having to walk every time they want to change the position of the virtual object is cumbersome. As a related technology, for example, Japanese Patent Application Laid-Open No. 2020-204973 discloses a technique that records panoramic images in association with the user's position and posture at the time of shooting, and generates and displays a panoramic image viewed from the user's virtual walking position based on the user's head posture.

[0003] However, the technology disclosed in the above announcement has the following problem: although it can display a panoramic image of the location where the user is virtually walking, it is not easy to immediately return to the display of a panoramic image of the previous location. Summary of the Invention

[0004] This disclosure provides a technique for easily switching and displaying a realistic image indicating a user's field of vision at a current location and a pseudo-image similar to the user's field of vision at a specified location.

[0005] Therefore, one aspect of the present invention provides an information processing apparatus comprising: a location acquisition unit configured to acquire a user's current location; a presentation unit configured to present a plurality of locations including the current location using a display device; a designation unit configured to designate a location from the plurality of locations according to an instruction from the user; a first display control unit configured to display, on the display device, an image of real space captured by a camera device as a real image indicating the user's field of view at the current location when the location designated by the designation unit is the current location; a generation unit configured to generate a pseudo image similar to the user's field of view at the location designated by the designation unit when the location designated by the designation unit is not the current location; and a second display control unit configured to display the pseudo image on the display device.

[0006] Other features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0007] Figure 1A This is an image diagram illustrating common features in embodiments of this disclosure.

[0008] Figure 1B This is another image illustrating common features in embodiments of this disclosure.

[0009] Figure 2 This is a block diagram illustrating an example of the common hardware configuration of the HMD in the embodiment.

[0010] Figure 3 This is a block diagram illustrating the functional configuration of the HMD in the first embodiment.

[0011] Figure 4 This is a diagram illustrating an example of data combinations in the position and pose information and image information of the HMD in the first embodiment, in which position and pose information and images in real space (hereinafter referred to as "real images") are correlated.

[0012] Figure 5A This is an example diagram illustrating a scenario where the displayed content is switched in the HMD of the first embodiment.

[0013] Figure 5B This is an example diagram showing a screen displaying a real-world image of the current location on the display device of the HMD in the first embodiment.

[0014] Figure 5C This is a diagram showing an example of a screen displaying a pseudo-image at position B on the display device of the HMD in the first embodiment.

[0015] Figure 6A This is a diagram illustrating an example of a method for specifying a location using a location specification module in the HMD of the first embodiment.

[0016] Figure 6B This is a diagram illustrating another example of a method for specifying a location using a location specification module in the HMD of the first embodiment.

[0017] Figure 7 This is a flowchart illustrating the recording of position and pose information and the processing of real-world images in the position, pose, and image information of the HMD in the first embodiment.

[0018] Figure 8 This is a flowchart illustrating the process of switching between displayed content and pseudo-images in the HMD of the first embodiment.

[0019] Figure 9This is a flowchart illustrating the process of switching between displayed content and pseudo-images in the HMD of the first embodiment, and specifically for the case where the position is specified by the user's gesture.

[0020] Figure 10 This is a block diagram illustrating the functional configuration of the HMD in the second embodiment.

[0021] Figure 11 This is a flowchart illustrating the process of switching between displayed content and mixed reality images and mixed pseudo-images in the HMD of the second embodiment.

[0022] Figure 12 This is a flowchart illustrating the process of displaying the mixed reality image in the second embodiment. Detailed Implementation

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the configurations described in the following embodiments are merely examples, and the scope of this disclosure is not limited to the configurations described in the embodiments. For example, each of the units constituting this disclosure can be replaced by any unit capable of performing the same function. In addition, arbitrary configurations can be added. Furthermore, two or more arbitrary configurations (features) of the embodiments can be combined. In addition, not all combinations of features described in the various embodiments are necessary for the solution of this disclosure. The configurations of the various embodiments can be appropriately modified or changed according to the specifications of the device to which this disclosure is applicable and various conditions (such as usage conditions and usage environment). In the following embodiments, the same reference numerals are given to the same components.

[0024] First, the common configuration of the embodiments will be described with reference to the accompanying drawings. Figure 1A and Figure 1B These are image diagrams illustrating common features of embodiments of the present disclosure. The video see-through HMD 101 is an information processing device that can be worn on the head of a user 102. In fact, a frame (hereinafter referred to as a "real object") 103 exists in front of the user 102. The HMD 101 in the embodiments includes a shooting and display system and a processing system. On the other hand, when the size and weight of the HMD 101 are reduced, the shooting and display system and the processing system of the HMD 101 can be separated. In this case, the shooting and display system can be arranged in the HMD 101, while the processing system can be arranged in a small external box computer as an external device. In this configuration, the small external box computer corresponds to the information processing device. The external device is not limited to a small external box computer, and can be, for example, a portable computer such as a laptop PC, tablet PC, or smartphone, or a fixed computer such as a desktop PC.

[0025] Figure 1AThis is an example image illustrating a scenario where user 102 uses a gyroscope sensor and a camera mounted on HMD 101 to obtain information related to user 102's position and posture (hereinafter also referred to as "position and posture information") and an image at that position and posture. In this case, with HMD 101 mounted on user 102's head, position and posture information related to user 102 can be obtained using the gyroscope sensor. Therefore, HMD 101 can use the gyroscope sensor and camera to obtain information showing user 102's position and posture, as well as an image captured at that position and posture. HMD 101 records the position and posture information obtained in this way and the image in association with each other in a storage device.

[0026] Image 104, captured by a camera mounted on HMD 101, is displayed on an electronic display mounted on HMD 101, showing the field of view of user 102 wearing HMD 101. Image 104 shows arrows 105 guiding user 102 to other positions using HMD 101. User 102 can record images of various positions at the current location in HMD 101 by moving their head along arrows 105.

[0027] Figure 1B This is an example image illustrating a scenario where user 102 switches an image displayed on an electronic display mounted in HMD 101 from a mixed reality image 106 at their current location to a mixed pseudo-image 107 at a designated location (recording location 109, described later). Mixed reality image 106 is an image in mixed reality space viewed by user 102 wearing HMD 101 at their current location, and includes a real object 103, a three-dimensional virtual object (hereinafter referred to as a "virtual object") 108, and a marker 110 indicating the location of user 102 (hereinafter referred to as the "recording location") 109, as indicated by the position and posture information recorded as associated with the image as described above. Mixed pseudo-image 107 is an image in pseudo-mixed reality space assuming user 102 wearing HMD 101 is viewing it from recording location 109, and includes the real object 103 and the virtual object 108. Note that because recording location 109 is not included in the mixed pseudo-image, mixed pseudo-image 107 does not include the marker 110 indicating recording location 109.

[0028] When user 102 designates marker 110 via the controller or hand tracking function of HMD 101, the mixed reality image 106 at the current location is switched to the mixed pseudo-image 107 at the recording location 109. Furthermore, when user 102 makes a gesture (such as waving their hand), the mixed pseudo-image 107 at the recording location 109 is switched to the mixed reality image 106 at the current location. Additionally, when user 102 speaks the name indicating the location 109, the mixed reality image 106 at the current location is switched to the mixed pseudo-image 107 at the recording location 109. Thus, HMD 101 can switch the displayed content between the mixed reality image 106 at the current location and the mixed pseudo-image 107 at the recording location 109. However, the first embodiment described later assumes that the virtual object 108 does not exist.

[0029] Figure 2 This is a block diagram illustrating an example of the common hardware configuration of the HMD in the embodiment. CPU 201 is the system controller and controls the entire HMD 101. Additionally, CPU 201 performs information processing according to the embodiment by executing an information processing program. ROM 202 is a read-only memory that stores programs and parameters that do not need to be changed, such as basic programs and initial data. RAM 203 is a memory that temporarily stores input information, information processing, and the calculation results of image processing. Sensor 204 is a sensing component that detects the position and posture of HMD 101 (i.e., the position and posture of user 102), and is, for example, a gyroscope sensor.

[0030] Communication I / F 205 is an interface capable of sending and receiving data to and from external devices (such as external cameras or clouds). Communication I / F 205 inputs and outputs data via local connection using USB (registered trademark) or Bluetooth (registered trademark) and Internet connection using Ethernet (registered trademark) or Wi-Fi (registered trademark). Storage device 206 is capable of reading and writing various types of information. Specifically, storage device 206 is a memory card, removable disk, or IC card removable from HMD 101, or a hard drive or memory card built into or externally attached to HMD 101.

[0031] According to the embodiment, the information processing program should be recorded in the storage device 206, and read from the storage device 206 by the CPU 201, expanded into the RAM 203, and executed. The information processing program can be stored in the ROM 202. The storage device 206 can also record the data required by the information processing program executed by the CPU 201. The camera device 207 is, for example, a camera mounted on the HMD 101. The display device 208 is, for example, an electronic display mounted on the HMD 101. The display device 208 has a stereoscopic display configuration corresponding to the eyes of the user 102. The operation device 209 controls the input to the HMD 101. The operation device 209 includes a power button, a menu button, a controller, etc. In the HMD 101, all hardware configuration components are connected to the bus 210 and can communicate with each other.

[0032] The camera system and processing system in HMD 101 can be separated from HMD 101 as external devices. In this case, the information processing system can consist of an external device and a display system (display device 208) in HMD 101. In such an information processing system, the camera system corresponding to the camera device 207, which is separate from HMD 101, can be a camera held by the user 102 or a 360-degree camera.

[0033] In the following text, except as mentioned above Figure 1A , 1B and Figure 2 In addition, we will also refer to Figures 3 to 9 The first embodiment will be described. Figure 3 This is a block diagram illustrating the functional configuration of the HMD 101 in the first embodiment. The real-world image acquisition module 301 acquires a real-world image using the camera device 207 in the HMD 101. The real-world image indicates the user 102's field of view at the current location. The user position and posture acquisition module 302 acquires position and posture information related to the user 102 using the sensor 204 in the HMD 101. The recording position guidance display module 303 applies image recognition technology to the real-world image acquired by the real-world image acquisition module 301, and displays guidance related to the camera position and camera range to the user 102 while the recording module 304 records the real-world image. Arrow 105 is also displayed by the recording position guidance display module 303.

[0034] The recording module 304 obtains an image at the location and posture indicated by the position and posture information obtained by the user position and posture acquisition module 302 from the real-world image obtained by the real-world image acquisition module 301, and records the image in the position, posture, and image information 305 in association with the relevant position and posture information. In this way, the recording module 304 constructs the position, posture, and image information 305 into a dataset by having multiple data combinations that associate the position and posture information with the real-world image. When the user 102 issues a recording start command or when the user 102 maintains the current position at the same location for a certain period of time, the recording module 304 records the data in the position, posture, and image information 305.

[0035] Figure 4 This is a diagram illustrating an example of data combinations in position, pose, and image information 305, in which position and pose information is correlated with a real-world image. In position, pose, and image information 305, multiple data combinations are recorded that correlate position and pose information related to user 102 with a real-world image. The real-world image data is compressed in JPEG format, and... Figure 4 Image information is represented by filenames. Position and posture information related to user 102, after being divided into position information and posture information, is recorded in position, posture, and image information 305. Position information is represented by three-dimensional coordinates (x, y, z), and posture information is represented by angles (yaw, pitch, roll). In various control systems of HMD 101, the position and posture indicated by the position and posture information are converted relative to the current position of user 102, using the position and posture at any point in real space as a reference, and then used. In the following description, position and posture information related to user 102 is referred to as "position information" when focusing on position information and as "posture information" when focusing on posture information.

[0036] Description Return to Figure 3 The location presentation module 306 presents the location indicated by the location gesture and image information recorded in the location information 305, along with the current location of the user 102, using markers or lists. If the locations overlap, the location presentation module 306 presents only one location. The location designation module 307 designates a location from the locations presented by the location presentation module 306 based on instructions from the user 102. The user 102 issues instructions using the operating device 209 (e.g., a controller) in the HMD 101, gestures, or voice.

[0037] The image generation module 308 generates a pseudo-image by simulating the field of view of the user 102 at the location specified by the position designation module 307. The pseudo-image is an image of a simulated real space assuming the user 102 wearing the HMD 101 is viewing real space from the location specified by the position designation module 307. The image generation module 308 generates a real image as a pseudo-image, associated with position information indicating the location specified by the position designation module 307, recorded in the position pose and image information 305. However, when different pose information is associated with the position information indicating the location specified by the position designation module 307, the image generation module 308 uses an image set by the user 102 or an image corresponding to the median of different pose information to generate a pseudo-image. Subsequently, on the display device 208 of the HMD 101 displaying the pseudo-image, an arrow is displayed to guide the user 102 to the pose indicated by the pose information associated with the real image used to generate the pseudo-image.

[0038] The first display module 309 displays the real-world image obtained by the real-world image acquisition module 301 (i.e., the real-world image at the current position and posture of user 102) on the display device 208 of HMD 101. The second display module 310 displays the pseudo-image generated by the image generation module 308 on the display device 208 of HMD 101. The switching module 311 switches the display content according to the position specified by the position specification module 307. When the position specified by the position specification module 307 is the current position of user 102, the switching module 311 switches the display content to the real-world image. On the other hand, when the position specified by the position specification module 307 is not the current position of user 102 (i.e., when the position is indicated by the position information recorded in the position, posture, and image information 305), the switching module 311 switches the display content to the pseudo-image.

[0039] With the pseudo-image displayed on the display device 208 of the HMD 101, it is difficult for the user 102 wearing the HMD 101 to walk. Therefore, when the user 102's current position changes, the CPU 201 (first changing unit) of the HMD 101 controls the switching module 311 to switch the display content from the pseudo-image to the real image. When the real image is displayed by the first display module 309 and the pseudo-image is displayed by the second display module 310, position presentation by the position presentation module 306 and position assignment by the position assignment module 307 are also performed.

[0040] Figure 5A , Figure 5B and Figure 5C This is a diagram showing an example of what happens when the displayed content is switched. Figure 5AThis diagram illustrates an example of a scenario where user 102 switches the display content on the display device 208 of HMD 101 worn on the head to a real image or a pseudo-image. Positions A, B, and C indicate the positions indicated by various position information recorded in position pose and image information 305. In the following description, positions A, B, and C are the names of the respective positions.

[0041] Figure 5B This is an example diagram showing a screen displaying a realistic image of user 102 at their current location on the display device 208 of HMD 101. The position presentation module 306 displays markers 501, 502, and 503 on the screen. Marker 501 indicates position A, marker 502 indicates position B, and marker 503 indicates position C. Window 504 shows the position of user 102 whose field of view is displayed on the display device 208 of HMD 101. Hereinafter, the position of user 102 whose field of view is displayed on the display device 208 of HMD 101 is referred to as the "display position." Figure 5B In window 504, the message "Display position: 0. Current position" is displayed. Window 504 is displayed by the position rendering module 306.

[0042] In the list 505 displayed by the location presentation module 306, numbers and location names are displayed in association. List 505 is an interactive 3D UI, but it can also be a 2D UI. This also applies to markers 501, 502, and 503. A hand icon 506 is displayed at the location of the user 102's hand detected by the hand tracking function of the HMD 101. When the user 102 uses a gesture to press, for example, "2. Location B" or marker 502 on list 505 using the hand icon 506, Figure 5B The scene in the middle changes to Figure 5C The scene in the video.

[0043] Figure 5C This is an example diagram showing a screen displaying a pseudo-image at position B on the display device 208 of the HMD 101. In this case, marker 503 is displayed. Figure 5C On the screen. In addition, “Show position: 2. Position B” is displayed in window 504. When user 102 presses “3. Position C” or mark 503 on list 505 by using hand icon 506, a pseudo-image at position C is displayed on display device 208 of HMD 101.

[0044] Figure 6A and Figure 6BThis diagram illustrates an example of a method for specifying a location using the location specifying module 307. Methods for specifying a location using the location specifying module 307 include specifying it via an operating device 209 (e.g., a controller) in the HMD 101, specifying it via gesture, specifying it via voice, etc. Figure 6A Instructions are given through gestures, and Figure 6B Instructions are given via voice. When gestures are used for designation, the location designation module 307 identifies a location from those presented by the location presentation module 306 by recognizing the user 102's gesture using the hand tracking function of the HMD 101. When the hand icon 506 is pressed by the user 102's hand, any number or location name on list 505, the location indicated by the pressed number or location name is designated. When the hand icon 506 is pressed by the user 102's hand, any one of markers 501, 502, and 503, the location indicated by the pressed marker is designated. When the user 102 waves their hand, their current location is designated. When the user 102 claps their hands (a predetermined gesture), a previously designated location is designated.

[0045] When specified by voice, the location designation module 307 recognizes the user 102's voice using the voice recognition function of the HMD 101 and designates a location from the locations presented by the location presentation module 306. When the user 102 speaks a number from list 505, the location indicated by that number is designated. When the user 102 speaks a location name from list 505, the location indicated by that location name is designated. When the user 102 says "back," the user 102's current location is designated. When the user 102 says "return to the previous display location" (a pre-defined voice command), the previously designated location is designated. When the designated location is the user 102's current location, a real image of the user 102's current location is displayed on the display device 208 of the HMD 101. Conversely, when the designated location is not the user 102's current location (i.e., when the designated location is indicated by the location information recorded in the position gesture and image information 305), a pseudo-image of the designated location is displayed on the display device 208 of the HMD 101.

[0046] Figure 7 This is a flowchart illustrating the processing of position and pose information and real-world images recorded in the position, pose, and image information 305 of the HMD 101. Figure 7 The processing in the flowchart is achieved by the CPU 201 reading the program recorded in the storage device 206, expanding it to the RAM 203, and executing the program. When starting... Figure 7During the processing in the flowchart, in step S701, CPU 201 begins to acquire a real-world image using the real-world image acquisition module 301. CPU 201 (the guidance display unit) uses the recorded position guidance display module 303 to display guidance on the camera position and camera range on the display device 208 of HMD 101. Furthermore, CPU 201 (the first guidance unit) uses the recorded position guidance display module 303 to display arrow 105 (first information) on the display device 208 of HMD 101. In step S702, CPU 201 (the information acquisition unit) uses the user position and posture acquisition module 302 to obtain current position and posture information related to user 102.

[0047] In step S703, CPU 201 obtains a real-world image at the position and pose indicated by the position and pose information obtained in step S702 from the real-world image obtained by the real-world image acquisition module 301. Furthermore, CPU 201 (associated recording unit) uses recording module 304 to record the real-world image obtained in this manner in relation to the position and pose information obtained in step S702 in position, pose, and image information 305. As described above, recording module 304 records data in step S703 in response to a recording start instruction from user 102 or when user 102's current position remains at the same position for a certain period of time. Thereafter, Figure 7 The processing in the flowchart has ended.

[0048] Figure 8 This is a flowchart illustrating the process of switching between real and pseudo images when content is displayed in HMD 101. Figure 8 The process shown in the flowchart (control method of the information processing device) is implemented by the CPU 201 (computer) expanding the program stored in the storage device 206 onto the RAM 203 and executing the information processing program. This is for the purposes of the description later. Figure 9 The flowcharts are the same. When starting Figure 8 During the processing in the flowchart, in step S801, CPU 201 begins to acquire a real-world image using the real-world image acquisition module 301. In step S802, CPU 201 uses the first display module 309 to display the real-world image acquired by the real-world image acquisition module 301 on the display device 208 of HMD 101. Therefore, the real-world image of the current position of user 102 is displayed on the display device 208 of HMD 101. In step S803, CPU 201 (position acquisition unit) obtains the current position of user 102 from the position and posture information obtained by the user position and posture acquisition module 302 (position acquisition step).

[0049] In step S804, the CPU 201 (presentation unit) uses the position presentation module 306 to display markers 501, 502, and 503, list 505, etc. (presentation step). Therefore, the position presentation module 306 presents the position indicated by the position information recorded in the position gesture and image information 305 and the current position of the user 102 as markers 501, 502, and 503 and list 505. When multiple position information indicating the same position are recorded in the position gesture and image information 305, the position presentation module 306 presents only one position. In step S805, the CPU 201 (assignment unit) uses the position assignment module 307 to assign a position based on instructions from the user 102 using the operating device 209 (e.g., controller), gesture, or voice of the HMD 101 (assignment step). In this way, the position assignment module 307 obtains the position assigned by the user 102 (hereinafter referred to as the "assigned position") from the position presented by the position presentation module 306.

[0050] In step S806, CPU 201 uses position assignment module 307 to determine whether the specified position is different from the display position. When CPU 201 determines that the specified position is different from the display position, the process proceeds to step S807. On the other hand, when CPU 201 determines that the specified position is not different from the display position, the process returns to step S803. Therefore, the determination process in step S806 is repeated until the specified position becomes different from the display position. In step S807, CPU 201 uses position assignment module 307 to determine whether the specified position is the current position of user 102. When CPU 201 determines that the specified position is the current position of user 102, the process proceeds to step S814. On the other hand, when CPU 201 determines that the specified position is not the current position of user 102, the process proceeds to step S808. In other words, when the location specified by user 102 is the location indicated by the location information recorded in the position pose and image information 305, the process proceeds to step S808.

[0051] In step S808, the CPU 201 (generation unit) generates a pseudo-image using the image generation module 308 (generation step). At this time, the image generation module 308 uses a real image recorded in the position and pose and image information 305, which is associated with the position information used to indicate the specified position obtained in step S805, to generate the pseudo-image. In step S809, the CPU 201 uses the switching module 311 to switch the displayed content from the real image at the user 102's current position to the pseudo-image generated in step S808. In step S810, the CPU 201 (second display control unit) uses the second display module 310 to display the pseudo-image switched in step S809 on the display device 208 of the HMD 101 (second display control step). Therefore, the pseudo-image at the specified position obtained in step S805 is displayed on the display device 208 of the HMD 101.

[0052] In step S811, CPU 201 uses user position and posture acquisition module 302 to obtain current posture information related to user 102. In step S812, CPU 201 determines whether the posture information obtained in step S811 is consistent with the posture information associated with the real image used when generating the pseudo image using image generation module 308 in step S808. When CPU 201 determines that the posture information obtained in step S811 is consistent with the posture information associated with the real image used when generating the pseudo image using image generation module 308 in step S808, the process returns to step S803.

[0053] On the other hand, when the CPU 201 determines that the posture information obtained in step S811 is inconsistent with the posture information associated with the real image used when generating the pseudo image using the image generation module 308 in step S808, the process proceeds to step S813. In step S813, the CPU 201 (second guidance unit) uses the image generation module 308 to display an arrow (second information) on the display device 208 of the HMD 101. Displaying the arrow in this case guides the user 102 to the predetermined posture indicated by the posture information associated with the real image used when generating the pseudo image using the image generation module 308. After this, the process returns to step S811. As described above, when the current position of the user 102 changes while the pseudo image is displayed on the display device 208 of the HMD 101, the CPU 201 uses the switching module 311 to switch the displayed content from the pseudo image to the real image.

[0054] In step S814, CPU 201 uses switching module 311 to switch the displayed content from the pseudo-image to the real image being acquired by real image acquisition module 301. In step S815, CPU 201 (first display control unit) uses first display module 309 to display the real image switched in step S814 (first display control step) on display device 208 of HMD 101. Therefore, the real image of user 102's current position is displayed on display device 208 of HMD 101. After this, the process returns to step S803.

[0055] Figure 9 This is a flowchart illustrating the process of switching between real and pseudo-images when content is displayed in HMD 101, and specifically a flowchart illustrating the process when the position is specified by a gesture from user 102. When starting... Figure 9 In the process described in the flowchart, in step S901, CPU 201 obtains the gesture of user 102 using the position designation module 307. In step S902, CPU 201 uses the position designation module 307 to determine whether the gesture of user 102 is a press operation. When CPU 201 determines using the position designation module 307 that the gesture of user 102 is a press operation, the process proceeds to step S903. On the other hand, when CPU 201 determines using the position designation module 307 that the gesture of user 102 is not a press operation, the process proceeds to step S909, which will be described later.

[0056] In step S903, the CPU 201 uses the position designation module 307 to determine whether the position specified by the user 102's gesture press operation (hereinafter referred to as the "designated position") is the user 102's current position. When the CPU 201 determines using the position designation module 307 that the designated position is the user 102's current position, the process proceeds to step S907. On the other hand, when the CPU 201 determines using the position designation module 307 that the designated position is not the user 102's current position, the process proceeds to step S904. That is, when the designated position is the position indicated by the position information recorded in the position gesture and image information 305, the process proceeds to step S904.

[0057] Because steps S904 to S906 and Figure 8 Steps S808 to S810 are the same, therefore a detailed description of them will be omitted. After this, the process returns to step S901. Furthermore, since steps S907 and S908 are the same as... Figure 8Steps S814 and S815 are the same, so their detailed descriptions will be omitted. Afterwards, the process returns to step S901. In step S909, CPU 201 uses the position designation module 307 to determine whether the user 102's gesture is a waving operation. When CPU 201 determines using the position designation module 307 that the user 102's gesture is a waving operation, the process proceeds to step S907. On the other hand, when CPU 201 determines using the position designation module 307 that the user 102's gesture is not a waving operation, the process proceeds to step S910.

[0058] In step S910, the CPU 201 uses the position designation module 307 to determine whether the user 102's gesture is a clapping operation. When the CPU 201 determines that the user 102's gesture is a clapping operation, the process proceeds to step S911. On the other hand, when the CPU 201 determines that the user 102's gesture is not a clapping operation, the process returns to step S901. In step S911, the CPU 201 uses the position designation module 307 to determine whether the previously designated position is the user 102's current position. When the CPU 201 determines that the previously designated position is the user 102's current position, the process proceeds to step S907. Therefore, a real-world image of the user 102's current position is displayed on the display device 208 of the HMD 101.

[0059] On the other hand, when the CPU 201 determines using the position designation module 307 that the previously designated position is not the current position of the user 102, the process proceeds to step S912. That is, when the previously designated position is the position indicated by the position information recorded in the position pose and image information 305, the process proceeds to step S912. In step S912, the CPU 201 uses the position designation module 307 to set the previously designated position as the designated position. Afterwards, the process proceeds to step S904. Therefore, a pseudo-image of the previously designated position is displayed on the display device 208 of the HMD 101. When the position is designated by the user 102's voice, it is consistent with... Figure 9 The flowchart above describes how the content is switched between real and pseudo images. This is a description of the first embodiment.

[0060] In the following text, reference will be made to Figure 10 and Figure 11A second embodiment will be described. In this second embodiment, the differences from the first embodiment will be described. In the HMD 101 of the first embodiment, a real image of the user 102's current location and a pseudo-image of the location specified by the user 102 are switched and displayed. However, when the user 102 wearing the HMD 101 is experiencing mixed reality, the user 102 sees virtual objects in the virtual space of the mixed reality. Therefore, an example of displaying virtual objects in either the real image of the user 102's current location or the pseudo-image of the location specified by the user 102 will be described in the second embodiment.

[0061] Figure 10 This is a block diagram illustrating the functional configuration of HMD 101 in the second embodiment. The virtual object position and pose acquisition module 1001 acquires position and pose information related to virtual objects in the virtual space of mixed reality in a three-dimensional manner. The virtual CG rendering module 1002 obtains a CG image by rendering the virtual CG using pose information related to the user 102 at the position specified by the position assignment module 307 and position and pose information related to the virtual objects.

[0062] The image compositing module 1003 composites the pseudo-image generated by the image generation module 308 and the CG image generated by the virtual CG rendering module 1002. Thus, the image compositing module 1003 obtains a composite image (hereinafter referred to as a "hybrid pseudo-image") by compositing the pseudo-image and the CG image at the location specified by the position specification module 307. The third display module 1004 uses the real-world image obtained by the real-world image acquisition module 301 and the CG image obtained by the virtual CG rendering module 1002 to display the hybrid real-world image at the current position and pose of the user 102 on the display device 208 of the HMD 101. The fourth display module 1005 displays the hybrid pseudo-image obtained by the image compositing module 1003 on the display device 208 of the HMD 101. That is, the fourth display module 1005 displays the hybrid pseudo-image that is assumed to be viewed at the location specified by the position specification module 307 on the display device 208 of the HMD 101.

[0063] The switching module 1006 switches the display content to the image at the location specified by the location specification module 307. When the location specified by the location specification module 307 is the current location of the user 102, the switching module 1006 switches the display content to a mixed reality image. When the location specified by the location specification module 307 is not the current location of the user 102 (i.e., when the location is indicated by the location information recorded in the position gesture and image information 305), the switching module 1006 switches the display content to a mixed pseudo-image. When the mixed pseudo-image is displayed on the display device 208 of the HMD 101, it is difficult for the user 102 wearing the HMD 101 to walk. Therefore, when the current location of the user 102 changes, the CPU 201 (second changing unit) of the HMD 101 uses the switching module 1006 to switch the display content of the mixed pseudo-image to a mixed reality image. In the mixed reality image displayed by the third display module 1004 and the mixed pseudo-image displayed by the fourth display module 1005, the location presentation module 306 and the location specification module 307 also perform position presentation.

[0064] Figure 11 This is a flowchart illustrating the process of switching between displayed content and mixed reality images and mixed pseudo-images in the HMD 101. Figure 11 The processing (control method of information processing device) in the flowchart is implemented by the CPU 201 (computer) reading the program recorded in the storage device 206, expanding the program to the RAM 203, and executing the program. When starting... Figure 11 During the processing in the flowchart, in step S1101, CPU 201 begins to acquire a real-world image using the real-world image acquisition module 301. In step S1102, CPU 201 performs mixed reality image display processing. Figure 12 This is a flowchart illustrating the mixed reality image display processing in step S1102. When starting... Figure 12 During the processing in the flowchart, in step S1201, CPU 201 uses user position and posture acquisition module 302 to obtain current position and posture information related to user 102.

[0065] In step S1202, CPU 201 uses the virtual object position and pose acquisition module 1001 to acquire position and pose information related to the virtual object in a three-dimensional manner. In step S1203, CPU 201 uses the virtual CG rendering module 1002 to acquire a CG image at the current position of user 102. In step S1204, CPU 201 uses the third display module 1004 to display a mixed reality image of the current position and pose of user 102 on the display device 208 of HMD 101. At this time, the third display module 1004 uses the real image acquired by the real image acquisition module 301 and the CG image acquired in step S1203. As a result, a mixed reality image of the current position of user 102 is displayed on the display device 208 of HMD 101. After this, the process returns to... Figure 11 The flowchart in the document is shown, and the process proceeds to step S1103.

[0066] Description Return to Figure 11 Because steps S1103 to S1105 are related to... Figure 8 Steps S803 to S805 are the same, so detailed descriptions of them will be omitted. In step S1106, CPU 201 uses position specification module 307 to determine whether the specified position is different from the display position. When CPU 201 uses position specification module 307 to determine that the specified position is different from the display position, the process proceeds to step S1107. On the other hand, when CPU 201 uses position specification module 307 to determine that the specified position is not different from the display position, the process proceeds to step S1119, which will be described later. In step S1107, CPU 201 uses virtual object position and pose acquisition module 1001 to obtain position and pose information related to the virtual object in a three-dimensional manner. In step S1108, CPU 201 (rendering unit) uses virtual CG rendering module 1002 to obtain the CG image at the specified position obtained in step S1105 (rendering step).

[0067] In step S1109, CPU 201 uses the position designation module 307 to determine whether the designated location is the current location of user 102. When CPU 201 determines that the designated location is the current location of user 102 using the position designation module 307, the process proceeds to step S1117. On the other hand, when CPU 201 determines that the designated location is not the current location of user 102 using the position designation module 307, the process proceeds to step S1110. That is, when the location designated by user 102 is the location indicated by the position information recorded in the position pose and image information 305, the process proceeds to step S1110.

[0068] In step S1110, the CPU 201 (generation unit) generates a pseudo-image using the image generation module 308 (generation step). At this time, the image generation module 308 generates the pseudo-image using a real image recorded in the position pose and image information 305, which is associated with the position information used to indicate the specified location obtained in step S1105. In step S1111, the CPU 201 (compositing unit) uses the image compositing module 1003 to composite the pseudo-image obtained in step S1110 and the CG image obtained in step S1108 (compositing step). Therefore, the image compositing module 1003 obtains a mixed pseudo-image at the specified location obtained in step S1105.

[0069] In step S1112, CPU 201 uses switching module 1006 to switch the displayed content from the mixed reality image at the current position of user 102 to the mixed pseudo-image obtained in step S1111. In step S1113, CPU 201 (fourth display control unit) uses fourth display module 1005 to display the mixed pseudo-image switched in step S1112 (fourth display control step) on display device 208 of HMD 101. Therefore, the mixed pseudo-image at the specified position obtained in step S1105 is displayed on display device 208 of HMD 101.

[0070] Because steps S1114 to S1116 and Figure 8 Steps S811 to S813 are the same, so their detailed description will be omitted. In step S1117, CPU 201 uses switching module 1006 to switch the displayed content from a mixed pseudo-image to a mixed reality image. In step S1118, CPU 201 (third display control unit) uses third display module 1004 to display the mixed reality image on display device 208 of HMD 101 (third display control step). At this time, third display module 1004 uses the real image obtained by real image acquisition module 301 and the CG image obtained in step S1108. As a result, a mixed reality image of the user 102's current position is displayed on display device 208 of HMD 101. After this, the process returns to step S1102.

[0071] In step S1119, CPU 201 uses the fourth display module 1005 to determine whether a mixed pseudo-image is being displayed. When CPU 201 determines, using the fourth display module 1005, that a mixed pseudo-image is being displayed, the process returns to step S1103. That is, when a mixed pseudo-image is being displayed on the display device 208 of HMD 101, the process returns to step S1103. On the other hand, when CPU 201 determines, using the fourth display module 1005, that a mixed pseudo-image is not being displayed, the process returns to S1102. That is, when a mixed reality image is being displayed on the display device 208 of HMD 101, the process returns to step S1102. The above is a description of the second embodiment.

[0072] As described above, the HMD 101 in the first embodiment allows the user 102 to switch and display a real image at the current location and a pseudo image at a specified location by specifying a location from the presented locations without moving. Similarly, the HMD 101 in the second embodiment allows the user 102 to switch and display a mixed real image at the current location and a mixed pseudo image at a specified location by specifying a location from the presented locations without moving. Therefore, the HMD 101 in both the first and second embodiments can easily switch and display the image viewed by the user 102 at the current location and the pseudo image that the user 102 would view assuming they are at a specified location.

[0073] Although preferred embodiments of the present disclosure have been described above, the present disclosure is not limited to the above embodiments, and various modifications and changes can be made within its scope. For example, the position designation module 307 can designate a position from the positions presented by the position presentation module 306 using light extending from the controller held by the user 102 or light extending from the user 102's hand. Furthermore, the position designation module 307 can designate a position from the positions presented by the position presentation module 306 by analyzing the user 102's gaze using the eye-tracking function of the HMD 101. Additionally, when the user 102 designates a previously designated position in response to clapping or saying "return to the last displayed position," the position designation module 307 can exclude the user's current position from the previously designated positions. Furthermore, in the position, pose, and image information 305, the real-world image associated with the position and pose information related to the user 102 is a still image compressed in JPEG format using data compression, but it can also be a moving image at the same angle.

[0074] According to the present invention, it is possible to easily switch and display the image that the user is viewing at the current location and the pseudo-image that the user is supposed to view at a specified location.

[0075] Other embodiments

[0076] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0077] The disclosure of the embodiments includes the following configurations, systems, methods, and program products.

[0078] (Configuration 1) An information processing device, comprising:

[0079] A location acquisition unit, configured to acquire the user's current location;

[0080] A presentation unit is configured to present multiple locations, including the current location, using a display device;

[0081] A designated unit, configured to specify a location from the plurality of locations according to instructions from a user;

[0082] A first display control unit is configured to display an image of the real space captured by a camera device on the display device as an indication of the user's field of view at the current position, when the position specified by the designated unit is the current position.

[0083] A generation unit, configured to generate a pseudo-image similar to the user's field of view at the location specified by the designated unit, when the location specified by the designated unit is not the current location; and

[0084] A second display control unit is configured to display the pseudo-image on the display device.

[0085] (Configuration 2) The information processing device according to Configuration 1 further includes a first changing unit, which is configured to change the display content on the display device from the pseudo image to the real image when the current position changes during the period when the pseudo image is displayed on the display device.

[0086] (Configuration 3) An information processing device, comprising:

[0087] A location acquisition unit, configured to acquire the user's current location;

[0088] A presentation unit is configured to present multiple locations, including the current location, using a display device;

[0089] A designated unit, configured to specify a location from the plurality of locations according to instructions from a user;

[0090] A rendering unit is configured to render computer graphics images in which virtual objects enter the user's field of view at a location specified by the designated unit.

[0091] The third display control unit is configured to, when the location specified by the designated unit is the current location, display a mixed reality image on the display device using an image of the real space captured by a camera device and the computer graphics image, as a real image indicating the user's field of vision at the current location;

[0092] A generation unit is configured to generate a pseudo-image similar to the user's field of view at the location specified by the designated unit when the location specified by the designated unit is not the current location.

[0093] A synthesis unit configured to synthesize the pseudo-image and the computer graphics image to obtain a hybrid pseudo-image; and

[0094] A fourth display control unit is configured to display the hybrid pseudo-image on the display device.

[0095] (Configuration 4) The information processing device according to Configuration 3 further includes a second changing unit, which is configured to change the display content on the display device from the mixed pseudo-image to the mixed reality image when the current position is changed during the display of the mixed pseudo-image.

[0096] (Configuration 5) An information processing device according to any one of Configurations 1 to 4, wherein the presentation unit presents the plurality of locations on the display device by displaying markers at locations in an image being displayed on the display device.

[0097] (Configuration 6) According to any one of Configurations 1 to 4, the presentation unit presents the plurality of locations by displaying a list of location names of the plurality of locations on the display device.

[0098] (Configuration 7) An information processing device according to any one of Configurations 1 to 4, wherein the designating unit designates a location from the plurality of locations according to an instruction issued by the user to the controller.

[0099] (Configuration 8) An information processing device according to any one of Configurations 1 to 4, wherein the designating unit designates a location from the plurality of locations according to an instruction issued by a user's gesture.

[0100] (Configuration 9) The information processing device according to Configuration 8, wherein, when the user's gesture is a predetermined gesture, the designating unit designates a previously designated position from the plurality of positions.

[0101] (Configuration 10) An information processing device according to any one of Configurations 1 to 4, wherein the designating unit designates a location from the plurality of locations according to an instruction issued by the user via voice.

[0102] (Configuration 11) The information processing device according to Configuration 10, wherein, when the user’s voice is a predetermined voice, the designating unit designates a previously designated location from the plurality of locations.

[0103] (Configuration 12) In any one of Configurations 1 to 4, the designating unit designates a position from the plurality of positions according to an instruction issued by a light ray extending from the user's hand or from a light ray extending from the controller.

[0104] (Configuration 13) An information processing device according to any one of Configurations 1 to 4, wherein the designating unit designates a location from the plurality of locations according to an instruction issued through the user's line of sight.

[0105] (Configuration 14) The information processing device according to any one of Configurations 1 to 4 further includes:

[0106] An information acquisition unit is configured to acquire location information and posture information, the location information and posture information indicating the user's position and posture; and

[0107] An association recording unit is configured to, upon acquiring the real-world image, record the position information and pose information obtained by the information acquisition unit in association with the real-world image.

[0108] The generating unit uses the real-world image recorded in association with the location information indicating the location specified by the designating unit to generate the pseudo-image.

[0109] (Configuration 15) According to the information processing device of Configuration 14, wherein when different posture information is associated with the position information indicating the position specified by the designated unit, the generating unit uses an image set by the user or an image corresponding to the median of the different posture information to generate the pseudo image.

[0110] (Configuration 16) According to the information processing device of Configuration 14, wherein, when the current position remains the same for a certain period of time, the association recording unit records the position information and posture information in association with the real image.

[0111] (Configuration 17) The information processing device according to Configuration 14, wherein there are multiple data combinations, in which the position information and posture information are associated with the real image.

[0112] (Configuration 18) The information processing device according to Configuration 14 further includes a guidance display unit configured to display guidance on the display device for indicating the camera range or camera position when the camera device acquires the real image recorded by the associated recording unit.

[0113] (Configuration 19) The information processing device according to Configuration 14 further includes a first guidance unit configured to display first information on the display device, the first information being used to guide the user to a posture when a real image recorded by the associated recording unit is obtained through the camera device.

[0114] (Configuration 20) The information processing device according to Configuration 14 further includes a second guidance unit configured to display second information on the display device, the second information being used to guide the user to a posture indicated by the posture information recorded in association with the real image used by the generation unit.

[0115] (Configuration 21) An information processing device according to any one of Configurations 1 to 4, 9, 11 and 15 to 20, wherein the information processing device is a head-mounted display including the camera device and the display device.

[0116] (System 1) An information processing system, comprising:

[0117] The information processing device according to any one of configurations 1 to 21;

[0118] The camera device; and

[0119] The display device.

[0120] (System 2) The information processing system according to System 1, wherein the camera device is a camera held by the user.

[0121] (System 3) The information processing system according to System 1, wherein the camera device is a 360-degree camera.

[0122] (Method 1) A control method for an information processing device, the control method comprising:

[0123] The location acquisition step is used to obtain the user's current location;

[0124] A presentation step is used to present multiple locations, including the current location, using a display device;

[0125] A specified step is used to specify a location from the plurality of locations according to instructions from the user;

[0126] The first display control step is used to display an image of the real space captured by a camera device on the display device as a real image indicating the user's field of view at the current position, when the position specified in the specified step is the current position.

[0127] A generation step, configured to generate a pseudo-image similar to the user's field of view at the location specified in the specified step, when the location specified in the specified step is not the current location; and

[0128] A second display control step is used to display the pseudo-image on the display device.

[0129] (Method 2) A control method for an information processing device, the control method comprising:

[0130] The location acquisition step is used to obtain the user's current location;

[0131] A presentation step is used to present multiple locations, including the current location, using a display device;

[0132] A specified step is used to specify a location from the plurality of locations according to instructions from the user;

[0133] A rendering step is used to render a computer graphics image in which a virtual object is represented that enters the user's field of view at a location specified in the specified step.

[0134] The third display control step is used to display a mixed reality image on the display device using an image of the real space captured by a camera device and the computer graphics image, when the location specified in the specified step is the current location, as a real image indicating the user's field of vision at the current location;

[0135] A generation step is used to generate a pseudo-image similar to the user's field of view at the location specified in the specified step, when the location specified in the specified step is not the current location.

[0136] A compositing step is used to synthesize the pseudo-image and the computer graphics image to obtain a hybrid pseudo-image; and

[0137] The fourth display control step is used to display the hybrid pseudo-image on the display device.

[0138] (Program 1) A computer program product comprising a program that causes a computer to perform the control method according to method 1 or 2.

[0139] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation so as to cover all such modifications and equivalent structures and functions.

[0140] This application claims the benefit of Japanese Patent Application 2024-078469, filed on May 14, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device, comprising: A location acquisition unit, configured to acquire the user's current location; A presentation unit is configured to present multiple locations, including the current location, using a display device; A designated unit, configured to specify a location from the plurality of locations according to instructions from a user; A first display control unit is configured to display an image of the real space captured by a camera device on the display device as an indication of the user's field of view at the current position, when the position specified by the designated unit is the current position. A generation unit is configured to generate a pseudo-image similar to the user's field of view at the location specified by the designated unit when the location specified by the designated unit is not the current location; as well as A second display control unit is configured to display the pseudo-image on the display device.

2. The information processing device according to claim 1, further comprising a first changing unit, the first changing unit being configured to change the displayed content on the display device from the pseudo image to the real image when the current position changes during the period when the pseudo image is displayed on the display device.

3. An information processing device, comprising: A location acquisition unit, configured to acquire the user's current location; A presentation unit is configured to present multiple locations, including the current location, using a display device; A designated unit, configured to specify a location from the plurality of locations according to instructions from a user; A rendering unit is configured to render computer graphics images in which virtual objects enter the user's field of view at a location specified by the designated unit. The third display control unit is configured to, when the location specified by the designated unit is the current location, display a mixed reality image on the display device using an image of the real space captured by a camera device and the computer graphics image, as a real image indicating the user's field of vision at the current location; A generation unit is configured to generate a pseudo-image similar to the user's field of view at the location specified by the designated unit when the location specified by the designated unit is not the current location; A synthesis unit is configured to synthesize the pseudo image and the computer graphics image to obtain a hybrid pseudo image; as well as A fourth display control unit is configured to display the hybrid pseudo-image on the display device.

4. The information processing device according to claim 3, further comprising a second changing unit, the second changing unit being configured to change the displayed content on the display device from the mixed pseudo-image to the mixed reality image when the current position is changed during the display of the mixed pseudo-image on the display device.

5. The information processing device according to any one of claims 1 to 4, wherein, The presentation unit presents the plurality of locations on the display device by displaying markers at the locations in the image being displayed on the display device, or by displaying a list of location names of the plurality of locations on the display device.

6. The information processing apparatus according to any one of claims 1 to 4, wherein, The designated unit designates a position from the plurality of positions based on instructions issued by the user to the controller, instructions issued by the user's gestures, instructions issued by the user's voice, instructions issued using light extending from the user's hand or from the controller, or instructions issued through the user's line of sight.

7. The information processing apparatus according to any one of claims 1 to 4, further comprising: An information acquisition unit is configured to acquire location information and posture information, the location information and posture information indicating the user's position and posture; as well as An association recording unit is configured to, upon acquiring the real-world image, record the position information and pose information obtained by the information acquisition unit in association with the real-world image. The generating unit uses the real-world image recorded in association with the location information indicating the location specified by the designating unit to generate the pseudo-image.

8. An information processing system, comprising: Information processing apparatus according to any one of claims 1 to 7; The camera device; as well as The display device.

9. A control method for an information processing device, the control method comprising: The location acquisition step is used to obtain the user's current location; A presentation step is used to present multiple locations, including the current location, using a display device; A specified step is used to specify a location from the plurality of locations according to instructions from the user; The first display control step is used to display an image of the real space captured by a camera device on the display device as a real image indicating the user's field of view at the current position, when the position specified in the specified step is the current position. A generation step is used to generate a pseudo-image similar to the user's field of view at the location specified in the specified step, when the location specified in the specified step is not the current location. as well as A second display control step is used to display the pseudo-image on the display device.

10. A control method for an information processing device, the control method comprising: The location acquisition step is used to obtain the user's current location; A presentation step is used to present multiple locations, including the current location, using a display device; A specified step is used to specify a location from the plurality of locations according to instructions from the user; A rendering step is used to render a computer graphics image in which a virtual object is represented that enters the user's field of view at a location specified in the specified step. The third display control step is used to display a mixed reality image on the display device using an image of the real space captured by a camera device and the computer graphics image, when the location specified in the specified step is the current location, as a real image indicating the user's field of vision at the current location; A generation step is used to generate a pseudo-image similar to the user's field of view at the location specified in the specified step, when the location specified in the specified step is not the current location. A compositing step is used to combine the pseudo image and the computer graphics image to obtain a hybrid pseudo image; as well as The fourth display control step is used to display the hybrid pseudo-image on the display device.

11. A computer program product comprising a program that causes a computer to perform the control method according to claim 9 or 10.

12. A computer-readable storage medium storing a program that causes a computer to perform the control method according to claim 9 or 10.

Images