head-mounted display

Abstract

To improve intuitive operability of a head-mounted display used for an MR system that superimposes a virtual object on a real space.SOLUTION: A head-mounted display which displays an AR object in a real space to form an MR space includes: a camera which captures the real space to obtain a captured image; a ranging camera which measures a distance of a real object in the real space; and a control apparatus. The control apparatus includes: imaging object processing to recognize the real object from the captured image; AR object processing which obtains the AR object and gives a position including a distance in the real space to the AR object; and display video generation processing which reflects perspective of the real object and the AR object to generate a display video in the MR space. The control apparatus further includes: processing to detect an operation screen display object from the captured image; and processing to display an operation screen of the MR space on the front of the operation screen display object. A video of the operation screen includes the AR object in the MR space.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a head mounted display (HMD) used in a mixed reality (MR) system that displays a real space and a virtual space (also referred to as a virtual object) in a superimposed manner. [Background technology]

[0002] AR objects (such as images and text) are placed in real space. Augmented The superimposition of images of AR objects (AR objects) on the screen and allowing users to view them is used in content such as games and maintenance work. One example of a game is a game in which players compete to catch characters (AR objects) placed in public places such as parks and train stations, and the type of character caught and the number of points scored. One example of maintenance work is when users perform work in a narrow elevator pit by following work instruction images (AR objects).

[0003] To display an AR object, an image called an AR trigger or mark is captured simultaneously with the background by a camera, and the AR object linked to the AR trigger is placed in real space. Alternatively, the real space where the user is located can be associated with a spatial coordinate system, and the AR object can be placed at any spatial coordinate position to be superimposed.

[0004] In an MR system, a user wears an HMD that integrates a camera, display optical system, and sensor. The camera captures the real space, and the sensor uses the real space to represent it in a spatial coordinate system. The display optical system places an AR object at any position in the spatial coordinate system, and the image of the AR object is superimposed on the real space. Furthermore, in an MR system, the camera captures the user's hand or other object placed in the real space, and the user can operate the AR object with hand movements, i.e., gestures. However, operating the AR object with gestures can be a nuisance to others in public places, and large movements like gestures are difficult to make in small spaces.

[0005] Prior art in this technical field is disclosed in Patent Document 1. Patent Document 1 discloses an information input device that projects an image for operation input onto the palm of a hand or the vicinity thereof, and identifies the operation input based on the movement of the palm. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2018-73170 Summary of the Invention [Problem to be solved by the invention]

[0007] In an MR system, an intuitive operation is required, in which an arbitrary AR object is selected in an MR space (a space in which an AR object is superimposed on a real space) and the AR object is operated in response to changes in the MR space. However, with the method disclosed in Patent Document 1, in which an input button for operation is selected and projected onto the palm, it is difficult to realize an intuitive operation in the MR space.

[0008] The present invention has been made in consideration of the above points, and its purpose is to provide an HMD for an MR system that can be used in public places, small spaces, etc., and that allows users to select any AR object in the MR space and intuitively operate the AR object in response to changes in the MR space. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the present invention provides, as an example, a head-mounted display that displays AR objects in real space to form an MR space, and includes a camera that photographs the real space and obtains the photographed image, a ranging camera that measures the distance of real objects in the real space, and a control device, wherein the control device includes a photographed object process that recognizes real objects from the photographed image, an AR object process that obtains an AR object and assigns a position including the distance in the real space to the AR object, and a display image generation process that reflects the perspective between the real object and the AR object and generates a display image in the MR space, and further includes a process that detects an operation screen display object from the photographed image and a process that displays an operation screen in the MR space in front of the operation screen display object, and the image of the operation screen includes the AR object in the MR space. [Effects of the Invention]

[0010] According to the present invention, a user can directly operate an AR object displayed on an operation screen, enabling intuitive operation in an MR space. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic external configuration diagram of an HMD according to a first embodiment. [Figure 2] FIG. 2 is a configuration block diagram of an HMD according to the first embodiment. [Figure 3] 10 is an example of a display image in the MR space in the first embodiment. [Figure 4] 1A to 1C are diagrams illustrating an operation method for an MR space in the first embodiment. [Figure 5]FIG. 2 is a flow diagram of the overall control process of the MR processing in the first embodiment. [Figure 6] FIG. 10 is a flow diagram of a photographing object process in the first embodiment. [Figure 7] FIG. 10 is a flow diagram of an AR object process according to the first embodiment. [Figure 8] FIG. 4 is a flowchart of a display image generation process in the first embodiment. [Figure 9] FIG. 4 is a flowchart of an operation recognition process according to the first embodiment. [Figure 10] FIG. 11 is an explanatory diagram illustrating selection of an AR object on the operation screen by one hand in the second embodiment. [Figure 11] FIG. 10 is an explanatory diagram showing how to enlarge or reduce the operation screen with one hand in the second embodiment. [Figure 12] FIG. 11 is an explanatory diagram for simplifying the image of the operation screen in the third embodiment to make it easier to see. [Figure 13] FIG. 11 is a flow diagram of generating and displaying an operation screen in the third embodiment. [Figure 14] FIG. 13 is an explanatory diagram showing only an AR object in an image on an operation screen in a fourth embodiment. [Figure 15] FIG. 13 is an explanatory diagram of an operation method in a comfortable position on the operation screen in the fifth embodiment. [Figure 16] 13 is an explanatory diagram of an operation method in which the user does not need to lift his / her hand and hold his / her palm out in front of him / her in the sixth embodiment. FIG. [Figure 17] FIG. 20 is an explanatory diagram showing the relationship between the imaging range of a wide-angle camera, the displayed image in the MR space, and the operation screen in the seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Example]

[0013] Fig. 1 is a schematic external configuration diagram of the HMD in this embodiment. In Fig. 1, 1 denotes the HMD, 10 denotes a camera, 11 denotes a distance measuring camera, 12a and 12b denote display optical systems (image projection units), 13 denotes a transmission optical system such as a lens and a screen, 14 denotes a nose pad, 15 denotes a controller (control device), 16 denotes a speaker, 17 denotes a microphone, and 18a, 18b, and 18c denote frame housings.

[0014] The user wears the HMD 1 on his or her face using the frame housings 18a, 18b and the nose pad 14.

[0015] The camera 10 is attached so as to capture an image in front of the user's line of sight, and the distance measuring camera 11 measures the distance between the image captured by the camera 10 and real objects in real space (including the background such as walls).

[0016] The distance measuring camera 11 may be a camera that calculates distance to feature points such as the contours of a real object using a method such as a stereo camera, or a camera that measures distance by projecting light rays two-dimensionally using a TOF (Time of Flight) method, as long as it can measure the distance to a real object in accordance with the image captured by the camera.

[0017] The display optical systems 12a and 12b display virtual objects (AR objects) by projecting an image to be viewed by the left eye and an image to be viewed by the right eye onto the transmission optical system 13. The user can see the scenery and real objects in front of them through the transmission optical system 13, and the virtual objects projected from the display optical systems 12a and 12b are visually recognized by the transmission optical system 13 as if they were located at predetermined positions in real space.

[0018] The controller 15 takes in real-space images captured by the camera 10 and real-space position data of real objects acquired by the ranging camera 11, and supplies this data to an internal memory and CPU. It also has built-in sensors such as a gyroscope, orientation sensor, position sensor, and contact sensor. It also generates images to be projected by the display optical systems 12a and 12b and sounds to be output to the speaker 16. The controller 15, camera 10, ranging camera 11, speaker 16, and microphone 17 are arranged in frame housings 18a, 18b, and 18c. The arrangement locations do not have to be as shown in FIG. 1.

[0019] Furthermore, the controller 15 includes a user interface (UI) with the user, which is mainly processed by the CPU. The user interface includes operation input processing, which will be described later.

[0020] Figure 2 is a configuration block diagram of the HMD 1 in this embodiment. In Figure 2, the same components as in Figure 1 are assigned the same numbers. Reference numeral 51 denotes a feature extraction processing unit, 52 denotes a distance calculation processing unit, 53 denotes a sensor group, 54 denotes a communication unit, 55 denotes a CPU, 56 denotes a RAM, 57 denotes an image RAM, 58 denotes a program Flash ROM (FROM), and 59 denotes a data FROM.

[0021] The display optical system 12 corresponds to the display optical systems 12a and 12b in Figure 1. The display optical system 12 projects the left-eye image and the right-eye image independently onto a transmission optical system 13, as shown by 12a and 12b. Alternatively, a single projector may project the interleaved left-eye image and right-eye image, and a shutter optical system may transmit the left-eye image and right-eye image to each eye. Furthermore, an optical system using a holographic lens may also be used.

[0022] The communication unit 54 can connect the HMD 1 to a network 2. An external server (not shown) on the network 2 may execute part of the processing of the HMD 1.

[0023] The program FROM 58 includes an overall control process 81, a photographing object process 82, an AR object process 83, a display image generation process 84, an operation recognition process 85, and the like, which constitute the processing program. These processing programs are loaded into the RAM 56 and executed by the CPU 55. Furthermore, the data FROM 59 can store data generated in the process and as a result of executing these processing programs.

[0024] The program FROM 58 and the data FROM 59 may be configured as separate memory media as shown in the figure, or may be configured as a single memory medium. They may also be two or more memory media, or may be non-volatile memory media other than FROM. Part of the data FROM 59 may be stored in an external server on the network 2. The image data generated by the display image generation process 84 is stored in the image RAM 57, read from the image RAM 57, and projected by the display optical system 12.

[0025] Fig. 3 shows an example of a display image of the MR space in this embodiment. In Fig. 3, 100 denotes the MR space and its image, 101 to 106 denote AR objects, and the image other than the AR objects is the background that the user views as real space through the transmission optical system 13.

[0026] In Figure 3, the background that the user is viewing is a street corner, and the user is using, for example, a route guidance app to head to XX Station. AR object 103 guides the user to turn right 30 meters ahead, and AR object 104 guides the user that XX Station is in the direction of the right turn. The AR objects also provide an explanation of the background. AR object 101 recognizes a person in the foreground and slightly to the left of the background, compares this with the user's social relationships, and identifies the person as Friend A. AR object 101 also recognizes the sign for a shop on the foreground right, "Tournessol," and displays an example of the shop's typical menu item using AR object 102.

[0027] Users also enjoy games in which they capture small animals (characters). AR objects 105 and 106 are game characters. The characters may move, and capturing a character requires an operation that matches the display position of the character.

[0028] 4A and 4B are diagrams illustrating a method for operating the MR space in this embodiment. In Fig. 4A, 1 denotes an HMD, 3 denotes a user, and 3a denotes the user's palm. When starting an operation in the MR space, the user 3 holds the user's palm 3a within the camera shooting range of the HMD 1.

[0029] FIG. 4(b) shows a display image of the MR space in which the user's palm 3a is detected and an operation screen 107 is displayed on or in front of the user's palm 3a. The operation screen 107 is an image in which the user's palm 3a is removed from the display image of the MR space of the HMD 1. The user's palm 3a can be removed, for example, by storing an image before the user holds out the palm 3a and replacing the area of ​​the user's palm 3a with the stored image. The user 3 also adjusts the position of the user's palm 3a, for example, by moving it left or right, so that the AR object that the user is trying to operate with the palm 3a is not hidden.

[0030] Reference numeral 3b denotes a finger of a hand different from the palm 3a held up by the user, and is a pointing object for pointing at an AR object in the operation screen 107 and operating the AR object. The content of the operation varies depending on the AR object; for example, in FIG. 4(b), selecting the AR object 106 allows the character of the AR object 106 to be captured. Alternatively, selecting an AR object may display a menu display that can be operated, and further sliding the finger may allow the user to select an operation from the menu.

[0031] In this way, the user can display the operation screen, which is a sub-screen for operation, on the palm of their hand or in front of them, and directly operate the AR object displayed on the operation screen. Therefore, operation can be performed on the palm of their hand, and there is no need to make large movements in the space, allowing intuitive operation in the MR space.

[0032] FIG. 5 is a flow chart of the overall control process 81 of the MR processing in this embodiment. In FIG. 5, the process starts in S100. In S101, a user logs in to use the HMD 1 as needed. The login server may be a personal computer on an intranetwork, or a server connected via an external network 2. Alternatively, the processing may be performed within the HMD 1. By logging in, pre-registered user-specific settings (user information) are called up from the data FROM 59. Examples of user-specific settings include display-related settings such as brightness, contrast, color scheme, and menu display position that are easy for the user to see. Other examples include the user's name or an icon such as an avatar.

[0033] In S102, the real space is photographed using the camera 10 and the ranging camera 11. The camera photographing may be performed at the timing when the entire MR processing is executed, or, for example, video shooting at 30 fps (frames per second) may be continued and the photographed video may be captured at the timing when the entire MR processing is executed.

[0034] In S103, the photographed object process 82 is executed to extract features from the captured camera image, select feature points, identify the shape of a real object for the set of feature points, and register it as a photographed object. For example, a person, a store sign, or something that characterizes the real space can be cited as a photographed object. The real object is treated as photographed object data in the HMD 1. In the real space, the walls of a room, distant scenery, etc. are treated as photographed objects that provide the background.

[0035] In S104, the AR object process 83 is executed to obtain data of the AR object to be placed in the real space from a memory such as the data FROM 59 in the HMD 1, or by downloading from a server connected via the external network 2. Alternatively, an AR object generated mainly by the CPU 55 of the controller 15, or generated by another application, may be imported.

[0036] S105 is a display image generation process 84, which generates a display image of the AR object and also generates an image of the operation screen.

[0037] S106 is an operation recognition process 85, which acquires operation information by tracing the movement of a pointing object or the like on the operation screen from the camera image, and determines which AR object to select and what changes to make to the selected AR object.

[0038] In the loop indicated by the dashed line from S104 to S106, the operation recognition process 85 is executed, and when the parameters of the AR object are changed or the user's palm is detected, the operation information of those parameters and status is provided to the AR object process 83 and the display image generation process 84, which then reflects the information in the display image.

[0039] Figure 6 shows the flow of the photographed object process 82 in MR processing. In Figure 6, the process starts at S150. At S151, the video captured by the camera is read. At S152, the features of the video are analyzed, for example, edges are extracted, and the vertices and inflection points of the edges are extracted as feature points. At S153, position data such as distance obtained by the ranging camera 11 or the sensor group 53 is assigned to the feature points.

[0040] In S154, the difference between the feature points and the previous one is evaluated, and in S155, the type of object is searched and identified from the set of feature points where the difference is significant based on the evaluation result. In S156, this result is registered as the photographed object. The flow ends in S157.

[0041] FIG. 7 shows the flow of the AR object process 83 in the MR processing. In FIG. 7, the process starts at S180. One photographic object is selected at S181, and an AR object is selected at S182. The AR object selection candidates may refer to data stored on an external server via the CPU 55 and the communication unit 54, for example. At S183, in order to associate the AR object with the photographic object, dependent parameters such as the relative position with respect to the photographic object are selected, and the position, size, and direction of the AR object on the displayed image are given. For example, positioning can be achieved by providing an offset relative to the position of a feature point of the selected photographic object.

[0042] In step S183, operation information is provided from the operation recognition process 85, and an instruction to change parameters, etc. is issued.

[0043] In S184, it is determined whether or not there are any photographic objects remaining to be linked to the AR object. If there are any (Yes), the process returns to S181; if No, the process ends in S185.

[0044] In addition, if an AR object unrelated to the photographed object, such as an AR object of a clock, is placed on the screen, it is not necessary to select the photographed object in S181, and it is also not necessary to link the AR object to the photographed object in S183.

[0045] The AR object process 83 is not limited to the above-described flow. For example, the AR object may be generated by a process such as drawing mainly performed by the CPU 55 in the HMD 1, or an AR object generated by another application may be imported by executing another application.

[0046] Figure 8 shows the flow of the display image generation process 84 in MR processing. In Figure 8, the process starts at S200. In S201, an AR object to be displayed is selected. The AR objects to be displayed are all AR objects linked to real objects (treated as photographed objects within HMD1) within the display range of HMD1, and processing is performed for each one. If the AR object is not set to be displayed in S202 (No), steps S203 to S205 are skipped.

[0047] If the display setting is Yes, in S203, rotation processing is performed taking into account the direction of the HMD1, and scaling processing is performed taking into account the distance of the AR object. In S204, the distance relationship between the AR object and the real object that overlaps it on the display is evaluated, and in S205, the AR object is displayed. However, if the real object is in front of the AR object and there is a hidden part, processing is performed so that that part of the AR object is not displayed. This allows for a three-dimensional display that takes into account the depth relationship between the real object and the AR object.

[0048] If there are any unprocessed AR objects (Yes) in S206, the process returns to S201. When all AR objects have been processed, the display image is complete. However, if palm detection information is included in the operation information from the operation recognition process 85, it is detected in S207, and an operation screen is generated in S208 and added to the display image. The flow ends in S209.

[0049] FIG. 9 is a flow chart of the operation recognition process 85 in MR processing. In FIG. 9, the process starts at S220. At S221, it is recognized whether a palm is present in the area close to the HMD 1 in the image captured by the camera, and if so, this is output as operation information. At S222, it is detected whether a pointing object such as a finger is present on the operation screen in the image captured by the camera, and if so, the position and movement of the pointing object are detected. The result is judged at S223, and it is determined which AR object is being instructed to perform what operation, and this is output as operation information. The flow ends at S224.

[0050] As described above, according to this embodiment, the user can display an operation screen on the palm of their hand or in front of them and directly operate the AR object displayed on the operation screen, thereby enabling intuitive operation in the MR space without making large movements.

[0051] In this embodiment, the operation screen is described as being displayed on the palm of the user's hand, but this is not limited to this. For example, any object that can be physically contacted by a pointing object (such as a finger) that points to the operation screen, including a part of the body such as the back of the hand or arm, or a predetermined object that can be held in the hand, such as a book, can be used as the operation screen display object. [Example]

[0052] In this embodiment, an example will be described in which an AR object can be operated with one hand on an operation screen.

[0053] Fig. 10 is an explanatory diagram of selecting an AR object on the operation screen with one hand in this embodiment. In Fig. 10, the same components as those in Fig. 4(b) are assigned the same reference numerals, and their description will be omitted. As shown in Fig. 10(a), with the hand open, one of the fingers is bent, and the AR object closest to the bent fingertip is selected.

[0054] FIG. 10(b) shows a case where an AR object at the bottom of the operation screen cannot be reached by simply bending the fingers. As shown in the figure, tilting the palm compresses the operation screen vertically, allowing the fingertips to approach the AR object at the bottom of the operation screen and select the AR object. The tilt of the palm is detected, for example, by a distance sensor, or by detecting a change in the aspect ratio from an image of the palm, thereby compressing the display of the operation screen vertically. As with the vertical direction, tilting the palm left and right compresses the horizontal direction, allowing the AR object to be selected with fingers that are easy to bend.

[0055] FIG. 11 is an explanatory diagram showing how the operation screen in this embodiment is enlarged or reduced by one-handed operation. In FIG. 11, the same components as those in FIG. 10 are assigned the same reference numerals, and their description will be omitted. In FIG. 11(a), the operation screen is enlarged by spreading the fingers apart in the direction of the arrow shown. The operation screen is not enlarged or reduced when the fingers are returned to their original position from the spread state, and is further enlarged when the fingers are spread apart again. This allows the operation screen to be enlarged to a size with almost no restrictions. The movement between the fingers can be detected, for example, by detecting the tips of the fingers from an image of the palm and detecting changes in the distance between the detected fingers.

[0056] Figure 11(b) shows an operation to reduce the operation screen. The operation screen is reduced by narrowing the gap between the fingers in the direction of the arrow shown in the figure. As with the case of enlarging, a reduced operation screen with almost no restrictions can be obtained by performing successive operations.

[0057] As described above, according to this embodiment, it is possible to operate an AR object or the like with one hand and with small movements. [Example]

[0058] In this embodiment, an example will be described in which the image on the operation screen is simplified to make it easier to see.

[0059] FIG. 12 shows the display image of the MR space in this embodiment. In FIG. 12, the same components as those in FIG. 4(b) are assigned the same reference numerals, and their description will be omitted. FIG. 12(a) shows the same MR space as that shown in FIG. 4(b), but the image on the operation screen 107 is different. FIG. 12(b) is an enlarged view of the operation screen 107 in FIG. 12(a) for the purpose of explanation. As shown in FIG. 12(b), the image on the operation screen 107 displays a simplified image of the operation screen 107 in FIG. 4(b).

[0060] Fig. 13 is a flow chart of a process in this embodiment, which corresponds to step S208 of generating and displaying an operation screen in the display image generation process 84 in Fig. 8. In Fig. 13, the process starts at S250, and data of the camera-captured image and the captured object is received at S251.

[0061] In S252, a pattern image is assigned to the photographed object, and in S253, a color for drawing the pattern image is determined. The color does not have to be one color; the color may be different at the top and bottom of the pattern image, or different colors may be used for the frame and the inside of the pattern image. Similarly, in S254, a pattern image is assigned to the AR object, and in S255, a color for drawing is determined. Furthermore, in S256, the background shape is made into a pattern image, and in S257, a color for drawing is determined. In S258, the photographed object, AR object, and background pattern images are composited, and the operation screen shown by 107 in FIG. 12 is obtained and output. The process ends in S259.

[0062] As described above, according to this embodiment, the operation screen can be simplified, and an AR object can be easily selected on the operation screen. [Example]

[0063] In this embodiment, an example will be described in which only the AR object is displayed on the image of the operation screen.

[0064] Fig. 14 is an explanatory diagram showing only the AR object displayed on the operation screen image in this embodiment. In Fig. 14, the same components as those in Fig. 4(b) are denoted by the same reference numerals, and the description thereof will be omitted.

[0065] 14(a) shows how to select an AR object. The user selects the AR object by pointing at it. After selecting the object, the user moves (drags) the finger that was pointing at the AR object in the direction of the dashed arrow in the figure.

[0066] FIG. 14(b) displays the AR object selected in FIG. 14(a) as operation screen 107. After moving the finger downwards sufficiently to display it, when the palm 3a is opened, operation screen 107 is displayed in front of the palm 3a. The image on operation screen 107 is the selected AR object. Then, the user can operate the AR object on operation screen 107.

[0067] Note that the AR object may be selected by a method other than pointing. For example, the HMD 1 may be configured to include a means for detecting the line of sight of the user 3, and may select an AR object that is caught by the line of sight.

[0068] As described above, according to this embodiment, only the AR object to be operated is displayed on the operation screen, allowing the user to see the details of the AR object at hand. Furthermore, the AR object to be operated can be moved to a location where it is easy to operate, which has the effect of improving operability. [Example]

[0069] In this embodiment, an example of a method for operating the operation screen in which the user lifts his / her hand, holds his / her palm out in front of him / her, and then lowers his / her hand or head to a comfortable position will be described.

[0070] Fig. 15 is an explanatory diagram of an operation method in a comfortable position on the operation screen in this embodiment. In Fig. 15, the same components as those in Fig. 4 are given the same reference numerals, and their description will be omitted.

[0071] In the left diagram (a1) of FIG. 15(a), similar to Example 1, the user 3 has his / her hand raised and is looking forward. The HMD 1 captures the user's hand within the camera's shooting range and displays the operation screen on the palm of the hand. The user 3, for example, determines the AR object that can be controlled on the operation screen by changing his / her hand from an open state to a closed state. The user 3 then opens his / her hand again, and operates the AR object on the previously determined operation screen, changing from the position shown in the left diagram (a1) to a relaxed position with his / her hand and face facing downward, as shown in the right diagram (a2).

[0072] Figure 15(b) shows the change in the image in the MR space, corresponding to Figure 15(a). In the (a1) posture, the image in the MR space is 100, showing the real space in front and the AR object. The HMD1 stores this image in its internal memory for a few seconds, overwriting the previous image.

[0073] When the user's posture changes as shown in FIG. 15(a2), the image of the MR space changes to 100a, displaying the lower part of 100. The previously confirmed operation screen 107 is displayed on the palm 3a of the user 3. The image displayed on the operation screen 107 is the image for the posture of (a1), and the user 3 selects and operates the AR object for the posture of (a1). To cancel the confirmed operation screen 107 and return to the currently visible image 100a of the MR space, a gesture such as clenching the open palm or sliding the finger 3b out from a position where there is no AR object, as when the operation screen 107 was confirmed, can be registered as a cancel command, and the corresponding gesture can be executed to easily perform the operation.

[0074] As described above, in this embodiment, the user can operate the MR space in a comfortable position.

[0075] In this embodiment, the background of the real image is fixed on the operation screen 107, but it is not necessary to stop the movement of the moving AR object. By setting the processing range of the AR object to the MR space images 100 and 100a, the AR object on the operation screen 107 can be kept moving in the same way as the image displayed on the MR space image 100.

[0076] Furthermore, confirmation and cancellation of the operation screen are not limited to the above gestures, and the operation screen may be confirmed by, for example, other gestures, voice, or recognizing the palm of the hand for a few seconds. [Example]

[0077] In this embodiment, an example of an operation method in which the user does not need to lift his / her hand and hold his / her palm out in front of him / her will be described.

[0078] Fig. 16 is an explanatory diagram of an operation method in this embodiment that does not require the user to lift their hand and hold their palm out in front of them. In Fig. 16, the same components as in Fig. 15 are given the same reference numerals, and their description will be omitted.

[0079] In Figure 16(a), user 3 has his / her hands down and is looking forward. In this position, the HMD 1 does not capture the user's hands within the camera's range. From this position, user 3 turns his / her face downwards as shown in Figure 16(b) so that the palm 3a of his / her hand is within the camera's range of the HMD 1 to operate the MR space. In this case, the user does not need to move his / her hands.

[0080] The image in the MR space changes in the same way as in Figure 15(b). In the posture shown in Figure 16(a), the image in the MR space is 100, showing the real space and AR objects in front of you. At this time, the HMD1 stores this image in its internal memory for a few seconds, overwriting previous images in a circular fashion. When the user's posture changes to Fig. 16(b), the image in the MR space changes to 100a, displaying the lower part of 100 and capturing the user's palm 3a. This movement is detected by the sensor group 53 of the HMD 1, and the HMD 1 stops saving the image to its internal memory and reads out the image saved in Fig. 16(a) just before this, as the image on the operation screen. The image displayed on the operation screen 107 is the image in the posture of Fig. 16(a), and the user 3 selects and operates the AR object in the posture of Fig. 16(a).

[0081] As described above, in this embodiment, the user can operate the MR space simply by moving his / her head slightly downward. [Example]

[0082] In this embodiment, an example will be described in which a wide-angle camera is used to obtain the same effect as in embodiment 5 or 6, even when the viewing angle of the image display unit is relatively narrow, and the real space image that forms the background of the operation screen is converted into a real-time video.

[0083] Fig. 17 is an explanatory diagram showing the relationship between the imaging range of the wide-angle camera, the displayed image in the MR space, and the operation screen in this embodiment. In Fig. 17, the same components as those in Fig. 4(b) are assigned the same reference numerals, and their description will be omitted.

[0084] In Figure 17, 108 is the range optically photographed by wide-angle camera 10 in the attitude of (a1) in Figure 15, and wide-angle camera 10 performs wide-angle photography using an image sensor corresponding to the combined range of 100 and 100b from within the range of 108.

[0085] 15(a1), the MR space is in the range 100 (solid line). 100 is the background of the real space seen through the transmission optical system 13, and the MR space is an image of the AR object projected by the display optical system 12. The wide-angle camera 10 has a wider shooting range in the upper range 100b than the real space seen through the transmission optical system 13.

[0086] When the user moves from this state to the posture shown in FIG. 15(a2), the optical range of the wide-angle camera moves to 108a. At this time, the image of the MR space becomes 100a, capturing the user's palm 3a. The operation screen is displayed in front of the palm 3a, and this image is the image of the upper region 100 of the wide-angle imaging range. Therefore, the MR space operated by the user, which is the operation range of the MR space, does not change before and after the posture change. At this time, the HMD 1 generates and superimposes AR objects within the range of the MR space 100 and the MR space 100a.

[0087] Furthermore, the wide-angle camera continuously captures images of the operating range, providing real-time real-space images and enabling real-time tracking of captured AR objects.

[0088] As described above, in this embodiment, the user can operate the MR space simply by operating in a comfortable position or by slightly tilting their head downward, and can display and operate AR objects that match the changing image of the real space. [Example]

[0089] In the first embodiment, a transmission optical system is used to project and display an AR object onto the transmission optical system, and the user views the scenery and real objects in front of the user through the transmission optical system. In contrast, in the present embodiment, an example using a video-through type HMD will be described.

[0090] The video-through method is a configuration in which a video image of a scene or a real object in front of the user captured by a camera is combined with an AR object and displayed on a display device. A block diagram of the HMD configuration in this embodiment is omitted. Alternatively, the camera may be configured as a pair of cameras, and a 3D camera may be used to obtain not only camera-captured images with left-right parallax, but also positional information such as the distance of real objects and backgrounds in the camera-captured images from the parallax information.

[0091] When a transmissive optical system is used, there is a possibility that misalignment may occur due to the effects of parallax when pasting an AR object into real space. However, by using the video-through method as in this embodiment, it is possible to adjust for the effects of parallax when combining video footage with an AR object, and generate a composite image without misalignment.

[0092] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, the above-described configurations, functions, and processing units may be implemented in part or in whole in hardware, for example, by designing them as integrated circuits. Furthermore, hardware and software may be used in combination. [Explanation of symbols]

[0093] 1: Head-mounted display (HMD), 3: User, 3a: User's palm, 3b: User's finger, 10: Camera, 11: Ranging camera, 12, 12a, 12b: Display optical system (image projection unit), 13: Transmissive optical system, 15: Controller, 51: Feature extraction processing unit, 52: Distance calculation processing unit, 53: Sensor group, 54: Communication unit, 55: CPU, 56: RAM, 57: Image RAM, 58: Program FROM, 59: Data FROM, 81: Overall control process, 82: Object capture process, 83: AR object process, 84: Display image generation process, 85: Operation recognition process, 100, 100a: MR space and its image, 101, 102, 103, 104, 105, 106: AR object, 107: Operation screen.

Claims

1. A head-mounted display, A camera that captures real space and obtains captured images; A display unit; a control device that generates a display image of an MR space formed by displaying an AR object in the real space, detects a predetermined display object from the captured image, generates an image including the AR object in the MR space in front of the predetermined display object as an operation screen of the MR space, and controls the image to be displayed on the display unit; the control device converts the real objects, background, and AR objects in the real space in the captured image into patterned images, and synthesizes the patterned images to generate an image on the operation screen. A head-mounted display characterized by:

2. The head-mounted display according to claim 1, the control device recognizes a movement of a pointer on the operation screen, and operates an AR object in the MR space based on the movement of the pointer. A head-mounted display characterized by:

3. The head-mounted display according to claim 2, the pointing object is a finger, and the AR object and the operation screen are operated by the movement of the finger; A head-mounted display characterized by:

4. The head-mounted display according to claim 1, A head-mounted display, wherein the predetermined display object is the palm of a user of the head-mounted display.

5. The head-mounted display according to claim 4, the control device recognizes at least one of bending or opening of the user's finger on the operation screen as a movement of a pointing object pointing to the operation screen; A head-mounted display characterized by:

6. A head-mounted display, A camera that captures real space and obtains captured images; A display unit; a control device that generates a display image of an MR space formed by displaying an AR object in the real space, detects a predetermined display object from the captured image, generates an image including the AR object in the MR space in front of the predetermined display object as an operation screen of the MR space, and controls the image to be displayed on the display unit; The head-mounted display is characterized in that, when the control device recognizes that an AR object in the MR space is selected by a finger of a user of the head-mounted display and the operation of dragging the selected AR object by the finger, it performs processing to make the selected AR object an image on the operation screen.

7. A head-mounted display, A camera that captures real space and obtains captured images; A display unit; a control device that generates a display image of an MR space formed by displaying an AR object in the real space, detects a predetermined display object from the captured image, generates an image including the AR object in the MR space in front of the predetermined display object as an operation screen of the MR space, and controls the image to be displayed on the display unit; a sensor for detecting the movement of the head-mounted display; The control device is characterized in that, when the sensor detects that the head-mounted display has rotated downward and further detects the specified display object, the control device sets the image of the MR space before the head-mounted display rotated downward as the image of the operation screen.

8. A head-mounted display, A camera that captures real space and obtains captured images; A display unit; a control device that generates a display image of an MR space formed by displaying an AR object in the real space, detects a predetermined display object from the captured image, generates an image including the AR object in the MR space in front of the predetermined display object as an operation screen of the MR space, and controls the image to be displayed on the display unit; a sensor for detecting a movement of the head-mounted display; the camera that captures the real space and obtains the captured image is a camera that captures the captured image at a wider angle than the display range of the MR space, The control device is characterized in that when the sensor detects that the head-mounted display has rotated downward and further detects the specified display object, the control device displays an image of the MR space, including an image obtained by shifting the cut-out position of the wide-angle captured image upward, as the image on the operation screen.

Images