Information processing apparatus, information processing system, control method thereof, and program

Abstract

To place annotations related to objects in a mixed reality space at suitable positions.SOLUTION: An information processing apparatus comprises: acquisition means for acquiring annotation information related to an object; height information acquiring means for acquiring height information of a display used by an operator; and display control means for controlling the display of the annotation information placed based on a position corresponding to the height acquired by the height information acquisition means and a position in a mixed reality space corresponding to the object.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, a control method thereof, and a program. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become widespread. Using this technology, users wearing a head-mounted display (HMD) can experience a mixed reality world that combines reality and virtuality by displaying a mixed reality image in which a CG model is placed on a real image.

[0003] In order to allow a user wearing an HMD to operate a virtual or real object, it is sometimes necessary to convey information such as how to operate these objects and the operation procedures to the user. Patent Document 1 describes a method for conveying information about objects to a user wearing an HMD.

[0004] Patent Document 1 has been proposed to solve the problem that when a large number of people participate in MR and review it, the number of annotations increases and multiple annotations obstruct the field of view of the HMD wearer. Patent Document 1 proposes to include an image acquisition means for acquiring an image including a virtual object, a storage means for storing, for each part on the virtual object, annotation information indicating the part and additional information indicating the state when the annotation information was input, in association with each other, a means for acquiring information corresponding to the additional information input by the user, a selection means for selecting annotation information to be displayed from the annotation information for each part on the virtual object stored in the storage means based on the additional information, a generation means for generating a composite image in which the selected annotation information and the image are combined, and an output means for outputting the composite image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-10228 Summary of the Invention [Problem to be solved by the invention]

[0006] Patent document 1 proposes selecting and displaying annotations from annotation information for each part of a virtual object based on additional information that indicates the state when the annotation information was input, but does not mention the height at which the annotation information is displayed.

[0007] For example, when recreating the inspection of a virtual or real object in a mixed reality space, if annotations are displayed at a specific location on the object, it may be difficult to see the annotations depending on the height and posture (standing, crouching, etc.) of the HMD wearer.

[0008] The present invention aims to place annotations relating to objects in a mixed reality space at suitable positions. [Means for solving the problem]

[0009] The present invention is an acquisition means for acquiring annotation information related to an object; height information acquisition means for acquiring height information of a display used by an operator; a display control means for controlling the display of the annotation information arranged on the display based on a position corresponding to the height acquired by the height information acquisition means and a position in the mixed reality space corresponding to the object; The present invention is characterized by having the following. [Effects of the Invention]

[0010] The present invention provides an advantage in that it is possible to place an annotation relating to an object in a mixed reality space at a suitable position. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of a configuration diagram of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of various devices according to an embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating an example of the functional configuration of various devices according to an embodiment of the present invention. [Figure 4] FIG. 2 is a processing diagram showing an example of a module configuration of various devices in an embodiment of the present invention. [Figure 5] 1 is a flowchart showing an example of a series of processing steps in an embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating an example of a process for generating and displaying a mixed reality image according to an embodiment of the present invention. [Figure 7] 1 is a configuration diagram showing an example of HMD information, model information, coordinate information and vector information of a real object, annotation information, and annotation placement information in a mixed reality space in an embodiment of the present invention. [Figure 8] 1 is a schematic diagram showing an example of arrangement of a virtual object and annotations related to the virtual object displayed on an HMD in an embodiment of the present invention. FIG. [Figure 9] 1 is a schematic diagram showing an example of arrangement of a virtual object and annotations related to the virtual object displayed on an HMD in an embodiment of the present invention. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] An example of the configuration of an information processing system according to an embodiment of the present invention will be described with reference to FIG.

[0013] 1, various devices in the information processing system of the present invention are communicably connected via a network 150. For example, an information processing device 100 is communicably connected to an HMD 101.

[0014] The information processing device 100 stores a three-dimensional model (CG model / virtual object) to be superimposed on a real image captured by the HMD 101.

[0015] Furthermore, the information processing device 100 acquires a real image from the HMD 101 that it manages and stores it in a storage unit. The information processing device 100 also identifies and stores the position and orientation of the HMD 101. The position and orientation of the HMD 101 can be identified by a sensor (optical sensor 104 in FIG. 1) detecting the position and orientation of an optical marker 103 installed on the HMD 101 as the position and orientation of the HMD 101, and the information processing device 100 acquiring this. Alternatively, a two-dimensional marker in a real image captured by the HMD 101 may be used.

[0016] The information processing device 100 generates a mixed reality image by superimposing the 3D model on a real image, using the position and orientation of the HMD 101 and the 3D model and information on the position and orientation of the 3D model stored in the storage unit. The mixed reality image is then transmitted to the HMD 101 to be displayed on the display 222 of the HMD 101. The HMD 101 displays the received mixed reality image on the display. Note that, although the HMD 101 will be described below as being worn on the head of a user (operator), it may also be a handheld display device (not worn on the head) that allows the display 222 to be held in the hand. This concludes the explanation of FIG. 1.

[0017] Next, an example of the hardware configuration of various devices in the embodiment of the present invention will be described with reference to FIG.

[0018] The CPU 201 comprehensively controls each device and controller connected to the system bus 204 .

[0019] The ROM 202 also stores a BIOS (Basic Input / Output System), which is a control program for the CPU 201, an operating system (OS), and various other programs required to realize the functions executed by various devices.

[0020] The RAM 203 functions as a main memory, a work area, etc. for the CPU 201. The CPU 201 loads programs and the like required for executing processing into the RAM 203 and executes the programs to realize various operations.

[0021] Various programs and the like used by the information processing device 100 of the present invention to execute various processes described below are recorded in the external memory 211, and are loaded into the RAM 203 as needed and executed by the CPU 201. Furthermore, definition files and various information tables used by the programs related to the present invention are stored in the external memory 211.

[0022] An input controller (input C) 205 controls input from a pointing device (input device 209) such as a keyboard or a mouse.

[0023] The video controller (VC) 206 controls display on a display device such as a right-eye / left-eye display 222 included in the HMD 101. For example, an external output terminal (e.g., a Digital Visual Interface) is used to output to the right-eye / left-eye display 222. The right-eye / left-eye display 222 is composed of a display for the right eye and a display for the left eye. The input controller (input C) 205 controls display on a display device such as a display 210 (such as a liquid crystal display) included in the information processing device 100. Note that in FIG. 2, the display device is not limited to a liquid crystal display, and may be another display device such as an organic EL display.

[0024] The memory controller (MC) 207 controls access to an external memory 211 such as a hard disk (HD) or flexible disk (FD) that stores boot programs, browser software, various applications, font data, user files, edited files, various data, etc., or a card-type memory connected to a PCMCIA card slot via an adapter.

[0025] The communication I / F controller (communication I / FC) 208 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, internet communication using TCP / IP is possible. The communication I / F controller 208 also controls communication with the optical sensor 104 via Gigabit Ethernet (registered trademark) or the like.

[0026] The general-purpose bus 212 is used to capture images from the right-eye / left-eye video camera 221 of the HMD 101. Images are input from the right-eye / left-eye video camera 221 using an external input terminal (for example, an IEEE1394 terminal). The right-eye / left-eye video camera 221 is composed of a video camera for the right eye and a video camera for the left eye.

[0027] The CPU 201 enables display on the display by, for example, executing a process of expanding (rasterizing) an outline font into a display information area in the RAM 203. The CPU 201 also enables user instructions using a mouse cursor (not shown) on the display.

[0028] Various programs and the like used by the information processing device 200 of the present invention to execute various processes described below are recorded in the external memory 211, and are loaded into the RAM 203 as needed and executed by the CPU 201. Furthermore, definition files and various information tables used by the programs related to the present invention are stored in the external memory 211. This concludes the explanation of FIG. 2.

[0029] Next, an example of the functional configuration of various devices in the embodiment of the present invention will be described with reference to FIG.

[0030] The imaging command transmission unit 311 is a transmission unit that transmits an imaging command for the information processing device 100 to acquire a real image using the right-eye / left-eye video camera 221 provided in the HMD 101 .

[0031] The image capturing unit 301 captures a real image using the right-eye / left-eye video cameras 221 provided in the HMD 101 .

[0032] The real image transmitting unit 302 is a transmitting unit that transmits the real image captured by the imaging unit 301 to the information processing device 100 .

[0033] The real image receiving and storing unit 312 is a storing unit that receives and stores a real image from the HMD 101.

[0034] The real object information acquisition and storage unit 313 performs processing to recognize real objects from the real images stored by the real image reception and storage unit 312. The information to be recognized includes information on whether the real object is a hand or finger, a tool, a part, or furniture, and the position and orientation of the real object are acquired and stored. Methods for recognizing real objects include, for example, in the case of hands and fingers, hand joint detection, hand tracking, gesture recognition, and skin color area detection, and the position and orientation of the hand or finger are acquired and stored. In addition, in the case of real objects such as furniture or parts, deep learning can be used, for example, to recognize the real object, and the position and orientation of the real object can be acquired and stored. The type, position, and orientation of the real object stored in the real object information acquisition and storage unit are referenced by the contact determination unit 319, the additional information determination unit 320, etc.

[0035] The HMD information acquisition and storage unit 314 stores the position and orientation of the HMD 101 by the optical sensor 104 specifying the position and orientation of the optical marker 103 installed on the HMD 101 .

[0036] The CG model storage unit 315 is a storage unit that stores 3D models, which are virtual objects, in association with the positions and orientations of the 3D models. An example of information about 3D models stored in the CG model storage unit 315 is shown in 801 in FIG. 8.

[0037] The CG model synthesis unit 316 is a synthesis unit that synthesizes a CG model by arranging the 3D model stored in the CG model storage unit 316 in a mixed reality space.

[0038] The MR image generation unit 317 is a functional unit that generates an image by superimposing an image of real space transmitted from the right-eye / left-eye video camera 221 of the HMD 101 and an image of a 3D model based on the position and orientation of the HMD 101 acquired from the HMD information acquisition unit 314. The MR image generation unit 307 controls the display of the right-eye / left-eye displays 222 of the HMD 101.

[0039] The MR image transmission unit 318 is a transmission unit that transmits images to be displayed by the information processing device 100 on the right-eye / left-eye displays 222 of the HMD 101 .

[0040] The MR image receiving unit 303 is a receiving unit that allows the HMD 101 to receive the images transmitted from the information acquisition device 100.

[0041] The MR image display unit 304 is a display unit that displays the images received by the MR image receiving unit 303 on the right-eye / left-eye displays 222 .

[0042] The contact determination unit 319 is a determination unit that determines whether a real object (for example, a hand) has come into contact with a virtual object or a real object (for example, a real object) in the information processing device 100. The contact determination determines that contact has occurred when, for example, the position of the real object stored in the real object information acquisition / storage unit 313 and the position of the virtual object arranged in the mixed reality space (information from the CG model synthesis unit 316) come within a predetermined range.

[0043] When a real object (for example, a hand) comes into contact with a virtual object or a real object, the additional information determination unit 320 determines whether additional information is added to the virtual object or real object (real object), that is, whether the virtual object or real object is an object for displaying an annotation. Whether additional information is added will be described later with reference to annotation information 740 in FIG. 7.

[0044] The HMD height information acquisition unit 321 acquires the height information of the HMD 101 from the HMD information acquisition and storage unit 314 .

[0045] The annotation assignment command unit 322 is a command unit that commands the assignment of additional information, operation methods, operation procedures, etc. that have been assigned to an object contacted by the contact determination unit 319 as annotations in mixed reality space. At this time, the height at which the annotation is assigned is set to the height information acquired from the HMD height information acquisition unit 321. By adjusting the height at which the annotation is displayed in this way, it is possible to achieve the effect of displaying the annotation at a position that is easy for the user to see even when the user approaches the object to touch it.

[0046] In this embodiment, the information processing device 100 is equipped with the processing units (functions thereof) 311 to 322, but for example, the HMD 101 itself may be configured to have these components independently, or the HMD 101 may have some of the processing units 311 to 322.

[0047] Next, with reference to FIG. 4, an example of the module configuration of various devices in the embodiment of the present invention will be described.

[0048] The information processing device 100 is composed of an operating system (OS) 401, a graphics engine 402, a mixed reality platform 403 (also called an MR platform), and a mixed reality application 404 (also called an MR application or viewer application), and is controlled by a CPU 201.

[0049] The operating system 401 controls the input and output of the HMD 101 and transfers the real image obtained from the camera 221 via the input interface to the mixed reality platform 403. It also outputs the mixed reality image drawn by the graphics engine 402 to the display 222 via the output interface.

[0050] The graphics engine 402 generates an image to be drawn from a 3D model stored in the external memory 211, and superimposes and synthesizes it on a real image. The engine used for drawing may be, for example, a widely used graphics engine such as OpenGL or DirectX, or a proprietary graphics engine. In this embodiment, OpenGL is used as the graphics library.

[0051] The mixed reality platform 403 determines the position and orientation of the HMD 101 by acquiring the positions of the optical markers 103 from the optical sensor 104, and aligns the real space with the virtual space. Note that the position and orientation and alignment techniques can be realized using known techniques disclosed in Japanese Patent Application Laid-Open Nos. 2002-32784, 2006-072903, 2007-166427, etc.

[0052] It is also possible to identify the position and orientation of the HMD 101 based on the position measured by triangulation from the positions of multiple two-dimensional markers using a camera provided in the HMD 101, without using a position sensor.

[0053] The mixed reality application 404 receives the position and orientation of the HMD 101, shape information of the 3D model, and position and orientation information from the mixed reality platform 403, and issues a drawing command for the 3D model to the graphics engine 402. At this time, the OpenGL API is used to issue a command that sets identification information and position and orientation information for the 3D model to be drawn. This concludes the explanation of Figure 4.

[0054] First Embodiment Next, a series of processing steps according to an embodiment of the present invention will be described with reference to Fig. 5. Note that each step in Fig. 5 is a processing step executed by the CPU 201 of each of the information processing device 100 and the HMD 101.

[0055] 5, the CPU 201 of the information processing device 100 receives an operation to start the mixed reality application 404. When the mixed reality application 404 is started, it is started and executed by the mixed reality application 404, the mixed reality platform 403, the graphic engine 402, and the OS 401. In addition, an operation to end the mixed reality application 404 is received, and the mixed reality application 404 is ended when it is to be ended.

[0056] In step S501, the CPU 201 of the information processing device 100 transmits a command to the HMD 101 to display a mixed reality space.

[0057] In step S511, the HMD 101 displays the mixed reality space on the right-eye / left-eye displays 222 of the HMD 101. The processing of steps S501 and S511 will be described with reference to FIG.

[0058] 6 is a flowchart showing an example of a process for generating and displaying a mixed reality image according to an embodiment of the present invention. Note that each step in FIG. 6 is a processing step executed by the CPU 201 of each of the information processing device 100 and the HMD 101.

[0059] 6, CPU 201 of information processing device 100 transmits an imaging command to imaging unit 221 of HMD 101 to instruct the imaging unit 221 to capture an image of real space. Note that although the HMD of the present invention is a video see-through type having a video see-through display, it may also be an optical see-through type HMD that displays an image of virtual reality space superimposed on an optical see-through type display. Note that even in the case of an optical see-through type HMD, the imaging process of steps S601 to S603 may be executed to detect a real object (e.g., a hand), or contact between a real object and a virtual object may be determined by other methods (e.g., input of a voice command).

[0060] In step S611, the right-eye / left-eye video cameras (imaging units) 221 of the HMD 101 capture images of the real space, and in step S612, transmit the captured images to the information processing device 100.

[0061] In step S602, the CPU 201 of the information processing device 100 receives the captured image transmitted in step S612, and stores it in the RAM 203 or the external memory 211 in step S603.

[0062] In step S604, the CPU 201 of the information processing device 100 detects the position and orientation of the HMD 101 from the position and orientation of the optical marker 103 installed on the HMD 101 via the optical sensor 104, and acquires it as the position and orientation of the HMD 101.

[0063] In step S605, the CPU 201 of the information processing device 100 stores the position and orientation of the HMD 101 in the RAM 203 or the external memory 211. The stored position and orientation of the HMD 101 is stored, for example, as data such as that in the HMD information 710 in FIG.

[0064] In step S606, the CPU 201 of the information processing device 100 acquires the positions of real objects present in the captured image from the captured image stored in step S603. When acquiring information on hands as real objects, for example, hands (hand images) are detected from the captured image, and information on the number of detected hands and the positions of the hands in space is acquired. The positions of the hands are determined, for example, by measuring the distance from the camera of the HMD 101 to the hands using triangulation, and the information processing device 100 identifies the positions (coordinates) of the hands in space from information on the distance from the HMD 101 to the hands and information on the position and orientation of the HMD 101. The acquired positions and orientations of the hands are stored, for example, as data such as those in the real object information 730 in FIG. 7.

[0065] In step S607, the CPU 201 of the information processing device 100 synthesizes information about CG models (3D models) arranged in the mixed reality space. An example of the model data to be synthesized is registered as model information 720 in Fig. 7. For example, each model in the model information 720 in Fig. 7 is arranged in the mixed reality space.

[0066] In step S608, the CPU 201 of the information processing device 100 generates an image that can be viewed from the HMD 101 in the mixed reality space into which the CG model has been synthesized in step S607. Specifically, based on images captured by the right-eye and left-eye video cameras 221, respectively, an image into which the CG model has been synthesized is generated to be displayed on the right-eye and left-eye displays, respectively.

[0067] In step S609, the CPU 201 of the information processing device 100 transmits to the HMD 101 the image (MR image) in the mixed reality space generated in step S608.

[0068] In step S613, the right-eye / left-eye display (display unit) 222 of the HMD 101 receives the MR image transmitted from the information processing device 100, and in step S614, the MR image is displayed on the right-eye / left-eye display (display unit) 222 of the HMD 101.

[0069] The above processing is repeated as the processing of steps S501 and S511 until an operation to end the mixed reality application 404 is received. Returning to the description of the flowchart in FIG.

[0070] Once the MR image is displayed through the processes of steps S501 and S511 in FIG. 5, the process proceeds to step S502.

[0071] In step S502, the CPU 201 of the information processing device 100 determines whether a real object (for example, a hand) has come into contact with a virtual object or a real object (whether a predetermined operation has been received). The contact determination is made when, for example, 730 in Fig. 7, which is the position of the real object (for example, a hand) stored in the real object information acquisition and storage unit 313, and 720 in Fig. 7, which is the position of the virtual object arranged in the mixed reality space, come within a predetermined range.

[0072] In step S503, the CPU 201 of the information processing device 100 branches the process depending on whether or not contact was determined in step S502. If it is determined in step S502 that contact with a real object (for example, a hand) has occurred, the process proceeds to step S504, and if it is determined that no contact has occurred, the process proceeds to step S509, and if the mixed reality space display has not ended, the process returns to step S501.

[0073] In step S504, CPU 201 of information processing device 100 determines whether additional information has been added to the virtual object or real object contacted in step S502. If additional information has been added, the process proceeds to step S505. If additional information has not been added, the process proceeds to step S509. If the mixed reality space display has not ended, the process returns to step S501. The determination of whether additional information has been added to the virtual object or real object is made based on whether annotation information 740 corresponding to the contacted model information 720 in FIG. 7 has been stored. If annotation information 740 corresponding to the contacted model information 720 has been stored, annotation information (additional information) of the contacted model (object) is obtained, and the process proceeds to step S505.

[0074] In step S505, the CPU 201 of the information processing device 100 acquires the height information from the position information of the HMD 101 stored in step S605.

[0075] In step S506, the CPU 201 of the information processing device 100 transmits to the HMD 101 a command to assign the annotation information acquired in step S504 to the position of the height information of the HMD 101 acquired in step S505. In detail, the CPU 201 transmits a command to assign the annotation information acquired in step S504 to a position based on the height information of the HMD 101 and position information of the contacted model (object).

[0076] In step S512, the right-eye / left-eye display (display unit) 222 of the HMD 101 displays the annotation information acquired in step S504 on the height information transmitted in step S506.

[0077] In step S507, the CPU 201 of the information processing device 100 determines whether a virtual object or real object has left the field of view of the HMD (the imaging range of the video camera 221 of the HMD 101). If the object has left the field of view of the HMD, the process proceeds to step S508. If the object has not left the field of view of the HMD, the process proceeds to step S509. If the mixed reality space display has not ended, the process returns to step S501. This conditional branch is a process for temporarily concealing annotation information related to the object when the object has left the field of view of the HMD, and resetting the height at which the annotation information is arranged when the object re-enters the field of view of the HMD (the imaging range of the video camera 221 of the HMD 101).

[0078] In step S508, the CPU 201 of the information processing device 100 deletes (hides) the annotation information related to the object from the mixed reality space.

[0079] In step S509, the CPU 201 of the information processing device 100 determines whether to end the display of the mixed reality space. If an instruction to end the display of the mixed reality space is received from the user, the processing of this flowchart ends. If the display of the mixed reality space is not to be ended, the processing returns to step S501 and the display of the mixed reality space continues.

[0080] The above processing has the effect of enabling the HMD wearer to display annotations for objects at the right height at the right timing.

[0081] An example of a display that can be seen by the HMD wearer will be described with reference to FIGS.

[0082] 8 is a schematic diagram showing an example of annotation placement when the present invention is not implemented, illustrating a display screen displayed on the display 222 when inspecting a virtual object (rack) 801. For example, consider a case where the HMD-wearing user is away from the virtual object (rack) 801 as shown in FIG. 8(a), approaches the virtual object until it comes into contact with the virtual object, and the annotation is displayed for the first time upon contact (812 in FIG. 8(b)). FIG. 8(b) shows the display screen (field of view 814) of the display 222 when the HMD-wearing user comes into contact with the virtual object (rack) 801. Meanwhile, it is assumed that the annotation for the virtual object (rack) 801 is set to be placed at position 823 in FIG. 8(c), which corresponds to the height of the center of the virtual object. The field of view of an HMD wearer approaching virtual object (rack) 801 is 814. Therefore, if the HMD wearer inspects the virtual object (rack) while standing, for example, the annotation displayed in 823 is not included in the field of view 814 (814 in Figure 8(c)), and the HMD wearer may miss it.

[0083] FIG. 9 shows an example of annotation placement when the present invention is implemented. When the HMD wearer approaches a virtual object, a rack (901 in FIG. 9(a)), and touches the rack (901 in FIG. 9(b)) (for example, by extending a real hand 912), an annotation associated with the rack is placed (913) at the height of the HMD 101 in the mixed reality space (925 in FIG. 9(c)). The displayed annotation 913 is within the field of view (914) of the HMD wearer when inspecting from a standing height, for example, reducing the risk of overlooking the annotation 913. Furthermore, when the HMD wearer crouches down to inspect the virtual object, the placement position of the annotation can also be aligned with the height of the HMD 101, for example, at position 923 in FIG. 9(c). As described above, whether the inspection is performed while standing or while crouching, the annotations can be displayed at the positions of annotations 913 and 923, respectively, thereby reducing the risk of missing the annotations.

[0084] It is also possible to display annotations in a predetermined range on the display screen of the display 222 of the HMD 101 instead of at a position linked to the object, but in that case, the annotations would move along with the movement of the HMD 101 up, down, left, and right, blocking the view of the predetermined range on the display 222 and potentially interfering with inspection work, etc. In the present invention, annotations linked to objects are positioned based on the height position of the object and the HMD in mixed reality space, which has the effect of allowing them to be displayed without monopolizing a predetermined area on the display screen of the HMD.

[0085] <Second embodiment> In the first embodiment, the height at which the annotation is positioned in the mixed reality space is the height of the HMD 101 when the HMD wearer makes contact with a virtual or real object. However, the height at which the annotation is positioned in the mixed reality space displayed on the display 222 of the HMD 101 may be acquired from height information of the HMD 101 at that time (i.e., the annotation follows the height of the HMD 101 and is positioned at the same height). In this case, once contact with a real object (e.g., a hand) is confirmed, the process of S503 in FIG. 5 determines Yes thereafter, and height information of the HMD 101 is acquired and the annotation can be positioned at the acquired height until the object leaves the imaging range of the video camera 221 of the HMD 101 (step S507). This second embodiment has the effect that the HMD wearer can always check the annotation by checking the horizontal direction, which is the orientation indicating the height of the HMD 101.

[0086] In the second embodiment, an upper limit (for example, 200 cm) and a lower limit (for example, 30 cm) may be set for the height at which annotations are placed.

[0087] <Third embodiment> In the first embodiment, once a virtual object or real object moves out of the viewfinder of the video camera 221 of the HMD 101, an annotation associated with the object is not placed unless the object comes into contact with a real object (for example, a hand) again, even when the object is next captured by the video camera 221. However, in the third embodiment, once an object moves out of the viewfinder of the video camera 221 of the HMD 101 and then re-enters the capture range of the video camera 221, the annotation that was placed once may be placed again unconditionally. In this case, in the flowchart of Fig. 5, a flag (not shown) is set for the annotation to be deleted in step S508, and instead of step S502, it is determined whether "the object for which the flagged annotation was set in S508 is present within the field of view of the HMD." If an object for which the flagged annotation is present is present within the capture range of the video camera 221, steps S503 and S504 are skipped, and the annotation is placed at the height of the HMD. This third embodiment has the advantage that if an annotation that you have once checked disappears from view and you want to review it again, you can check the annotation information by looking at the position where the original annotation was located without having to move close to the object and touch it in order to place the annotation again.

[0088] <Fourth embodiment> In the first embodiment, an annotation placed in a mixed reality space was deleted by removing a virtual object or a real object from the field of view of the HMD (the imaging range of the video camera 221 of the HMD 101). However, in the fourth embodiment, a placed annotation can be deleted by an operation of the HMD wearer. In the fourth embodiment, a virtual panel displaying annotation information is placed as a virtual object that comes into contact with a real object (e.g., a hand) determined in the processing of step S502. When it is determined that the placed virtual panel has come into contact with a real object (e.g., a hand), the virtual panel displaying the placed annotation information can be deleted. The ability to delete an annotation by contact with a real object (e.g., a hand) has the effect of erasing annotations that no longer need to be displayed, thereby reducing annotations that obstruct the field of view, for example, during inspection.

[0089] As described above, the present invention provides the effect of making it possible to place annotations relating to objects in mixed reality space at suitable positions.

[0090] The present invention can also be embodied as, for example, a system, an apparatus, a method, a program, or a storage medium, and specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0091] The present invention also includes a case where a software program for realizing the functions of the above-described embodiments is supplied to a system or device directly or remotely, and a computer in the system or device reads and executes the supplied program code to achieve the functions.

[0092] Therefore, the program code installed on a computer to realize the functional processing of the present invention also realizes the present invention, and in other words, the present invention also includes the computer program itself for realizing the functional processing of the present invention.

[0093] In this case, as long as it has the functionality of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, or the like.

[0094] Recording media for supplying the program include, for example, flexible disks, hard disks, optical disks, magneto-optical disks, MOs, CD-ROMs, CD-Rs, CD-RWs, etc. Also available are magnetic tapes, non-volatile memory cards, ROMs, DVDs (DVD-ROMs, DVD-Rs), etc.

[0095] Another method of providing the program is to connect to an Internet homepage using the browser of the client computer, and then download the computer program of the present invention itself or a compressed file with an automatic installation function from the homepage to a recording medium such as a hard disk.

[0096] The program code constituting the program of the present invention can also be divided into multiple files and each file can be downloaded from a different website. In other words, the present invention also includes a WWW server that allows multiple users to download program files for implementing the functional processing of the present invention on a computer.

[0097] Alternatively, the program of the present invention may be encrypted, stored on a storage medium such as a CD-ROM, and distributed to users, and users who meet certain conditions may download key information for decrypting the program from a website via the Internet. The downloaded key information may then be used to execute the encrypted program and install it on a computer.

[0098] The functions of the above-described embodiments are realized by the computer executing the read program. Alternatively, the functions of the above-described embodiments can also be realized by an operating system or the like running on the computer performing some or all of the actual processing based on the instructions of the program.

[0099] Furthermore, the program read from the recording medium is written to a memory provided in a function expansion board inserted into a computer or a function expansion unit connected to the computer. Thereafter, based on the instructions of the program, a CPU or the like provided in the function expansion board or the function expansion unit performs part or all of the actual processing, and the functions of the above-mentioned embodiment are also realized by this processing.

[0100] It should be noted that the above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0101] 100 Information processing device (PC) 101 HMD 103 Optical Marker 104 Optical Sensor 150 Network

Claims

1. an acquisition means for acquiring annotation information related to an object; height information acquiring means for acquiring height information of a display used by an operator; a display control means for controlling the display of the annotation information arranged on the display based on a position corresponding to the height acquired by the height information acquisition means and a position in the mixed reality space corresponding to the object; An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein the display control means displays the annotation information on the display when a predetermined operation is received from an operator holding the display.

3. 3. The information processing system according to claim 2, wherein the predetermined operation is detecting that the hand of an operator holding the display has come into contact with the object.

4. The information processing system according to claim 1 , wherein the object includes a virtual object arranged in a mixed reality space.

5. 2. The information processing system according to claim 1, wherein the annotation information relating to the object is set for each of the objects.

6. The information processing system according to claim 2, characterized in that the height at which the annotation information is arranged and displayed on the display by the display control means is the height acquired by the height acquisition means when the specified operation is accepted.

7. 7. The information processing system according to claim 6, wherein when the object is moved out of the display range of the display, the height at which the annotation information is arranged is initialized.

8. The information processing system according to claim 7, characterized in that when the object leaves the display range of the display and then re-enters the display range of the display, the display of annotation information relating to the object is controlled without accepting the specified operation.

9. 2. The information processing system according to claim 1, wherein the height at which the annotation information is arranged on the display, the display being controlled by the display control means, follows the height of the display.

10. 10. The information processing system according to claim 9, wherein the height at which the annotation information is arranged and displayed on the display by the display control means is within a predetermined limit value.

11. 4. The information processing system according to claim 3, wherein the display control means, when receiving the predetermined operation on the object on which the annotation information is displayed, deletes the display of the annotation information.

12. 12. The information processing system according to claim 1, wherein the display is a head-mounted display.

13. an acquisition step of acquiring annotation information related to an object; a height information acquisition step of acquiring height information of a display used by an operator; a display control step of controlling the display of the annotation information arranged based on a position corresponding to the height acquired in the height information acquisition step and a position in the mixed reality space corresponding to the object; An information processing method comprising:

14. A program for causing at least one computer to function as each of the means of the information processing system according to any one of claims 1 to 12.

Images