Information processing device, control method for information processing device, and program

Abstract

This invention provides an information processing device that can display virtual objects in a way that they do not overlap with the user's hands. [Solution] The information processing device of the present invention comprises a control means for displaying an image of a virtual object on a display means so as to be placed in real space, and a detection means for detecting a user's hand, wherein the control means changes the display position of the virtual object so as to the user, the virtual object does not overlap with the user's hand.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus that controls the display position of a virtual object, a control method for the information processing apparatus, and a program.

Background Art

[0002] In recent years, so-called VR (Virtual Reality) technology is known as a technology for rendering a virtual world in real time and seamlessly. Also, so-called MR (Mixed Reality) technology is known as a technology for seamlessly integrating the real world and the virtual world in real time. As these technologies, those using an HMD (Head Mounted Display) are known.

[0003] In MR technology, it has been proposed to control the position where a virtual object is displayed according to conditions. In Patent Document 1, it is disclosed to move a virtual object according to a change in the head position. In Patent Document 2, it is disclosed to control the display position of a virtual object so as not to overlap with real objects existing around a vehicle driven by a user.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] When a user wearing an HMD performs an operation using their hand while looking at a virtual object, there is a risk that the virtual object may hide the user's hands, even when using the technologies disclosed in Patent Document 1 and Patent Document 2.

[0006] The present invention aims to provide an information processing device capable of displaying virtual objects in a manner that does not overlap with the user's hand. [Means for solving the problem]

[0007] The information processing apparatus of the present invention comprises a control means for displaying an image of a virtual object on a display means so as to be placed in real space, and a detection means for detecting a user's hand, wherein the control means changes the display position of the virtual object so as to the user, the virtual object does not overlap with the user's hand. [Effects of the Invention]

[0008] According to the present invention, virtual objects can be displayed without overlapping the user's hand. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram of the information processing device in this embodiment. [Figure 2] This figure shows an example of an image displayed on the display unit in this embodiment. [Figure 3] This is a flowchart showing the operation of the information processing device in this embodiment. [Modes for carrying out the invention]

[0010] The embodiments for carrying out the present invention will be described in detail below with reference to the attached drawings.

[0011] <Configuration of the information processing device> Figure 1 is a block diagram showing the configuration of the information processing device in this embodiment. In this embodiment, the information processing device 100 will be described as a video see-through type HMD. The video see-through type HMD presents a composite image to the user wearing the HMD, which is created by superimposing an image of a virtual object created using computer graphics (CG) onto a real image captured by a camera (imaging unit).

[0012] The control unit 101 controls each part of the information processing device 100 according to the input signals and the program. Alternatively, instead of the control unit 101 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0013] Memory 102 consists of RAM (Random Access Memory) and the like. Memory 102 is used as a buffer memory for temporarily holding various data such as rendered images and composite images, which will be described later, and as working memory for the control unit 101.

[0014] The non-volatile memory 103 consists of a ROM (Read Only Memory) that can be electrically stored and erased. The non-volatile memory 103 stores programs that the control unit 101 reads and executes.

[0015] The recording medium 104 is a memory card or the like for recording various data such as images, and is composed of semiconductor memory or a magnetic disk. The recording medium 104 may be configured to be detachable from the information processing device 100, or it may be built into the information processing device 100. In the information processing device 100, it is sufficient that at least the control unit 101 can access the recording medium 104.

[0016] Battery 105 supplies power for the entire information processing device 100 to operate. Battery 105 is composed of a rechargeable secondary battery and can be charged by an external battery charger. Battery 105 may also be a power receiving unit that receives power from an external source.

[0017] The operation unit 106 can receive various operations from the user. For example, the operation unit 106 includes various operation members such as a power button and operation buttons. By using the operation unit 106, the user can, for example, instruct the power on / off of the information processing apparatus 100, the switching of the operation mode, the change of settings, and the like.

[0018] The imaging unit 107 is a camera or the like that converts light into an electrical video signal. The imaging unit 107 is composed of, for example, an optical system such as an optical lens unit and an imaging element for converting the light introduced through the optical lens unit into an electrical video signal. The imaging unit 107 captures an image (real image) of the real space by imaging the front of the user. The imaging unit 107 may be a binocular camera (stereo camera) having, for example, a camera provided corresponding to the user's left eye and a camera provided corresponding to the user's right eye. The imaging unit 107 may be able to acquire an image (stereo image) having two image regions with a parallax for the right eye and the left eye. In the information processing apparatus 100, the real image captured by the imaging unit 107 is used as the background image of the composite image.

[0019] The display unit 108 is a display element composed of an organic EL display, a liquid crystal display, or the like. The display unit 108 includes two display elements corresponding to the user's right eye and left eye respectively. The control unit 101 generates two images corresponding to the user's right eye and left eye respectively as a composite image in which an image of a virtual object by CG (Computer Graphics) is superimposed on the real image captured by the imaging unit 107. The control unit 101 causes the display unit 108 to display the composite image, thereby making the user see the virtual object as being arranged in the real space.

[0020] The communication unit 109 is composed of, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller. The communication unit 109 outputs the modulated wireless signal from the antenna and demodulates the wireless signal received by the antenna. The control unit 101 may perform wireless communication according to the IEEE802.11 standard by controlling the communication unit 109. The control unit 101 communicates with an external device by controlling the communication unit 109. The communication with the external device may or may not be communication via the Internet.

[0021] The position detection unit 110 is a sensor for acquiring information regarding the position and orientation of the information processing apparatus 100. The position detection unit 110 may be an inertial measurement unit (IMU: Inertial Measurement Unit) composed of, for example, an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like. The control unit 101 estimates the movement (position and orientation) of the user's head by specifying the position and orientation of the information processing apparatus 100 from the sensor values of the position detection unit 110.

[0022] FIGS. 2(A) to 2(C) are diagrams showing an example of an image output by the control unit 101 to the display unit 108 and displayed on the display unit 108 in the present embodiment. In FIGS. 2(A) to 2(C), only one of the two synthesized images for the right eye and the left eye output by the control unit 101 is illustrated. In FIGS. 2(A) to 2(C), the control unit 101 outputs only the image indicated by the solid line to the display unit 108 and does not output the image indicated by the dotted line to the display unit 108.

[0023] The display image 200 shown in FIG. 2(A) is an example of a synthesized image displayed on the display unit 108 when the user of the information processing apparatus 100 performs an operation using their hand while viewing a virtual object.

[0024] The tracking virtual object 201 is a virtual object that is displayed to follow the user's head movements. The tracking virtual object 201 can consist of, for example, videos, still images, and 3DCG models. The tracking virtual object 201 may also be a virtual object that assists the user in manual tasks, such as a video for instructions or work explanations.

[0025] The fixed virtual object 202 is a virtual object displayed on the display unit 108 so as to be fixed in a predetermined position in real space.

[0026] The hand 203 is the user's hand included in the real-world image captured by the imaging unit 107. The control unit 101 detects the hand 203 (its position and region) from the real-world image. The control unit 101 may detect the region of the hand 203 by performing at least one of the following processes: detecting the color of the user's skin (hand) from the real-world image, detecting joint points from the real-world image, and detecting a rectangular region corresponding to the region of the hand 203 from the real-world image. In addition to the region of the hand, the depth of the hand may also be determined. For example, the distance from the imaging unit to a representative coordinate of the hand may be calculated using triangulation or the like.

[0027] The control unit 101 may detect the region of the hand 203 using a machine learning model, or it may detect the region of the hand 203 using pattern matching with hand patterns pre-stored in the recording medium 104. Furthermore, the control unit 101 may detect the region of the hand by combining two or more processes, including a process for detecting skin color from a real image, a process for detecting joint points from a real image, and a process for detecting a rectangular region corresponding to the region of the hand 203 from a real image. That's fine.

[0028] The region 204 of the hand 203 is the region in the real image from the position of the hand 203 to a predetermined distance away (the surrounding region of the hand 203). For example, the control unit 101 detects the position of the hand 203 by performing at least one of the following processes: detecting the color of the user's skin (hand) from the real image, detecting joint points from the real image, and detecting a rectangular region corresponding to the region of the hand 203 from the real image. The control unit 10 then determines (calculates) the region from the position of the hand 203 to a predetermined distance away as the region 204 of the hand 203. However, the region 204 of the hand 203 is not limited to this; for example, it may be the region of the hand 203 itself obtained by detecting the contour of the hand 203.

[0029] The display image 210 shown in Figure 2(B) is an example of a composite image displayed on the display unit 108 when the user turns their face downwards from the state shown in Figure 2(A).

[0030] The tracking virtual object 201 is displayed in a region 211 similar to the region of the tracking virtual object 201 in Figure 2(A), so as to follow the user's head movements. The fixed virtual object 202 is displayed in a region 211 that reflects the user's head movements so as to be fixed in a predetermined position in real space. Region 211 is the region of the fixed virtual object 202 in Figure 2(A) moved upwards.

[0031] The display image 220 shown in Figure 2(C) is an example of a composite image displayed on the display unit 108 when the user turns their face downwards from the state shown in Figure 2(B).

[0032] The tracking region 221 indicates the region in which the tracking virtual object 201 moves when it is moved in accordance with the user's head movement in real space. In Figure 3(C), the tracking virtual object 201 is moving so as to overlap with the region 204 of the hand 203, and the tracking region 221 overlaps with the region 204 of the hand 203. When the tracking region 221 overlaps with the region 204 of the hand 203, the tracking virtual object 201 is displayed in the proximity region 222 so as to stop at a position close to the hand 203 (a position a predetermined distance from the hand 203). In this way, the control unit 101 changes the display position of the tracking virtual object 201 so that it does not overlap with the hand 203 from the user's perspective. Here, the position of the tracking virtual object 201 close to the hand 203 is, for example, the position where the region of the tracking virtual object 201 touches the region 204 of the hand 203.

[0033] The fixed virtual object 202 is displayed in a region 223 that reflects the user's head movements so that it is fixed in a predetermined position in real space. Region 223 is the region obtained by moving the region 211 of the fixed virtual object 202 in Figure 2(B) upwards.

[0034] As described above, the control unit 101 of this embodiment changes the display position of the tracking virtual object 201 so that it follows the movement of the user's head and does not overlap the virtual object with the area 204 of the hand 203. As a result, the virtual object does not overlap the user's hand, so for example, the user can perform tasks while displaying the virtual object for work assistance and checking their hands at the same time. In other words, the user can work while keeping the virtual object and their hands in the same field of view.

[0035] <Operation of the Information Processing Device> The operation of the information processing device 100 will be explained with reference to Figure 3. Figure 3 is a flowchart showing the operation of the information processing device 100 in this embodiment to output a composite image to the display unit 108. The process shown in Figure 3 involves the control unit 101 reading a program stored in the non-volatile memory 103 and processing the input signals received from each part of the information processing device 100. This is achieved by controlling each part of the information processing device 100. The control unit 101 repeats the process shown in Figure 3 according to a predetermined cycle.

[0036] In S301, the control unit 101 controls the imaging unit 107 to image the real space and acquire a real image.

[0037] In S302, the control unit 101 renders the real-world image acquired in S301 into a format that can be output to the display unit 108, and saves it as a rendered image in the memory 102.

[0038] In S303, the control unit 101 determines whether or not all virtual objects have been rendered. If the control unit 101 determines that all virtual objects have been rendered, the process proceeds to S313; otherwise, the process proceeds to S304.

[0039] In S304, the control unit 101 determines whether the virtual object to be rendered is a follow-me virtual object. The control unit 101 refers to memory 102 to determine whether the virtual object to be rendered is a follow-me virtual object or a fixed virtual object. Whether a virtual object is a follow-me virtual object or not may be predetermined depending on the type of virtual object, or it may be set by the user for each virtual object. If the control unit 101 determines that the virtual object to be rendered is a follow-me virtual object, it proceeds to S305; otherwise, it proceeds to S312.

[0040] In S305, the control unit 101 refers to the memory 102 to determine whether the setting to display virtual objects so that they do not overlap with the hand area (overlap avoidance setting) is enabled. The user may have previously selected a mode from a plurality of modes, including a mode that allows virtual objects to overlap with the user's hand and a mode that changes the display position of virtual objects so that they do not overlap with the user's hand. The user may select a mode by using a menu operation with the operation unit 106. If the control unit 101 determines that the overlap avoidance setting is enabled, it proceeds to S306; otherwise, it proceeds to S311.

[0041] In S306, the control unit 101 performs a hand detection process to detect the position of the user's hand from the real image captured by the imaging unit 107. If the control unit 101 can detect the position of the hand through the hand detection process, it proceeds to S307; otherwise, it proceeds to S311.

[0042] In S307, the control unit 101 determines (calculates) the user's hand area based on the position of the user's hand detected in S306. The control unit 101 determines the hand area to be the area from the hand position detected in S306 to a position at a predetermined distance away. The hand area determined in S307 corresponds to the area 204 shown in Figures 2(A) to 2(C).

[0043] In S308, the control unit 101 determines (calculates) the tracking region, which is the area in which the virtual object to be rendered will be positioned when the virtual object follows the user's head movement. The tracking region determined in S308 corresponds to the tracking region 221 shown in Figure 2(C).

[0044] In S309, the control unit 101 determines whether the tracking area determined in S308 overlaps with the hand area determined in S307. If the control unit 101 determines that the tracking area overlaps with the hand area, it proceeds to S310; otherwise, it proceeds to S311.

[0045] In S310, the control unit 101 renders the virtual object so that it stops at a position close to the hand region determined in S307 (in the proximity region). The control unit 101 then processes the memory The rendering image stored in 102 is rendered with a virtual object superimposed on it, and the composite image is stored in memory 102. The virtual object rendered by the control unit 101 in S310 corresponds to the tracking virtual object 201 located in the proximity region 222 shown in Figure 2(C).

[0046] In S311, the control unit 101 renders a virtual object in the tracking region determined in S308. The control unit 101 superimposes the virtual object onto the rendered image stored in memory 102 and renders the composite image, then stores it in memory 102. The virtual object rendered by the control unit 101 in S311 corresponds to the tracking virtual object 201 shown in Figures 2(A) and 2(B).

[0047] In S312, the control unit 101 renders a virtual object so that it is fixed in a predetermined position in real space. The control unit 101 superimposes the virtual object onto the rendered image stored in memory 102 and renders the composite image, then stores it in memory 102. The virtual object rendered by the control unit 101 in S312 corresponds to the fixed virtual object 202 shown in Figures 2(A) to 2(C).

[0048] In S313, the control unit 101 outputs the composite image stored in the memory 102 to the display unit 108, thereby displaying the composite image on the display unit 108.

[0049] In this manner, the control unit 101 of this embodiment controls the tracking virtual object to follow the user's head movements, while preventing the tracking virtual object from being overlaid on the user's hands. This allows the user to work with the tracking virtual object, which assists in their work, and their hands within the same field of view.

[0050] (modified version) In this embodiment, the user's hand was detected from a real-world image displayed as a background image, but the method of hand detection is not limited to this. For example, the information processing device 100 may have a dedicated imaging unit or sensor (e.g., an infrared sensor) for detecting the hand. The dedicated imaging unit or sensor for detecting the user's hand may be an external device connected to the information processing device 100 by wire or wireless. Furthermore, if the user is wearing a wearable device capable of acquiring positional information, the position of the wearable device may be further referenced to detect the user's hand. For example, if multiple hands are visible in the real-world image, and the user is wearing a wearable device, the user's hand can be detected from among the multiple hands in the real-world image.

[0051] In this embodiment, the hand region is detected, and control is implemented to prevent the tracking virtual object from being overlaid on the hand region. However, tracking virtual objects that meet certain conditions may be controlled to be overlaid on the hand region, and such specific conditions may be selected in detail by the user in the overlay avoidance setting (step S305) described above. For example, by acquiring depth information of the hand, tracking virtual objects that are behind the hand may be set to be overlaid on the hand region. In this case, the part of the tracking virtual object that overlaps with the hand is not displayed in order to represent that it is behind the hand. For example, when a virtual object is overlaid on a real image, the part of the virtual object that overlaps with the hand is excluded. In this way, when the user views it, it is expressed that part of the tracking virtual object is hidden by the hand, and the depth relationship in the virtual space (mixed reality space) can be expressed.

[0052] Furthermore, the overlap avoidance setting may also be configured based on the type of work performed. For example, when operating a GUI provided on a tracking virtual object, if the user's work is deemed not dangerous, or if the content displayed on the tracking virtual object is important, The tracking virtual object may be controlled to be displayed overlaid on the hand area. When displayed overlaid on the hand area in this way, the tracking virtual object may be displayed in a way that hides the hand, and as mentioned above, the part of the virtual object that overlaps with the hand may be excluded. Conversely, if the user's work is determined to be dangerous, such as when using a sharp object, the tracking virtual object may be controlled not to be displayed overlaid on the hand area. Alternatively, if the content displayed on the tracking virtual object is not important, such as a rewindable video, the tracking virtual object may be controlled not to be displayed overlaid on the hand area.

[0053] In this embodiment, the information processing device 100 is a video see-through type HMD, but the information processing device 100 is not limited to this. For example, the information processing device 100 may be an optical see-through type HMD (e.g., AR glasses). An optical see-through type HMD displays images of virtual objects without displaying real images. Therefore, if a dedicated imaging unit or sensor is used to detect the user's hands, the information processing device 100 does not need to have an imaging unit 107. The information processing device 100 may also be a computer connected to the HMD and controlling the HMD to display images.

[0054] In this embodiment, a composite image is generated by overlaying a tracking virtual object and a fixed virtual object onto a real image, but the composite image is not limited to this. For example, the control unit 101 may generate a composite image that clearly shows the tracking region 204 when the tracking region 221 overlaps with the region 204 of the hand 203. This allows the user to understand that the tracking virtual object 201 is controlling the system so that the hand is not obscured. The method for making the region 204 of the hand 203 identifiable is not particularly limited as long as it does not obstruct the view of the hand 203, and for example, the region 204 may be made identifiable by a semi-transparent mask or a frame. The control unit 101 may also notify the user that the system is controlling the system so that the hand is not obscured by the tracking virtual object 201 in a way different from the method for making the region 204 of the hand 203 identifiable. For example, the user may be notified by displaying an icon indicating that the overlap avoidance setting is enabled.

[0055] In this embodiment, a fixed virtual object is fixed to a predetermined position in real space, and it is permitted for the fixed virtual object to overlap with the user's hand. However, similar to the tracking virtual object, the display position of the fixed virtual object may be changed so that it does not overlap with the user's hand. For example, if the fixed virtual object should be displayed as if it actually exists in real space, it may be displayed in a way that allows it to overlap with the user's hand, while remaining fixed to a predetermined position in real space.

[0056] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0057] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0058] Furthermore, the embodiments described above (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0059] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0060] This embodiment includes the following configurations, methods, programs, and media. (Composition 1) A control means that displays an image of a virtual object on a display means so that it is placed in real space, A detection means for detecting the user's hand and It has, The control means changes the display position of the virtual object so that, from the user's perspective, the virtual object does not overlap with the user's hand. An information processing device characterized by the following: (Configuration 2) The detection means detects the hand region by performing at least one of the following processes: detecting a predetermined color from an image captured in real space; detecting joint points from the image; and detecting a rectangular region corresponding to the hand region from the image. The information processing device according to configuration 1, characterized by the above. (Composition 3) The detection means detects the area from the hand's position to a predetermined distance away as the hand's area. An information processing device according to configuration 1 or 2, characterized by the above. (Composition 4) The control means changes the display position of the virtual object so that it stops at a position close to the hand when the virtual object moves so that it overlaps the hand. An information processing device according to any one of configurations 1 to 3. (Composition 5) The control means changes the display position of the virtual object so that it does not overlap with the user's hand, only if the virtual object is a virtual object that follows the user's head movements. An information processing device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The system further includes setting means for setting one of a plurality of modes, including a first mode that allows the virtual object to overlap with the user's hand, and a second mode that changes the display position of the virtual object so that it does not overlap with the user's hand. An information processing device according to any one of configurations 1 to 5. (method) A control step that causes an image of a virtual object to be displayed on a display means so as to be placed in real space, A detection step to detect the user's hand and It has, In the control step described above, the virtual object is controlled by the user. The display position of the virtual object is changed so that it does not overlap with the virtual object. A control method for an information processing device characterized by the following features. (program) A program to cause a computer to function as one of the means of an information processing device described in any one of configurations 1 to 6. [Explanation of symbols]

[0061] 100: Information processing device 101: Control unit 108: Display unit

Claims

1. A control means that displays an image of a virtual object on a display means so that it is placed in real space, A detection means for detecting the user's hand and It has, The control means changes the display position of the virtual object so that, from the user's perspective, the virtual object does not overlap with the user's hand. An information processing device characterized by the following:

2. The detection means detects the hand region by performing at least one of the following processes: detecting a predetermined color from an image captured in real space; detecting joint points from the image; and detecting a rectangular region corresponding to the hand region from the image. The information processing apparatus according to feature 1.

3. The detection means detects the area from the hand's position to a predetermined distance away as the hand's area. The information processing apparatus according to feature 1.

4. The control means changes the display position of the virtual object so that it stops at a position close to the hand when the virtual object moves so that it overlaps the hand. The information processing apparatus according to feature 1.

5. The control means changes the display position of the virtual object so that it does not overlap with the user's hand, only if the virtual object is a virtual object that follows the user's head movements. The information processing apparatus according to feature 1.

6. The system further includes setting means for setting one of a plurality of modes, including a first mode that allows the virtual object to overlap with the user's hand, and a second mode that changes the display position of the virtual object so that it does not overlap with the user's hand. The information processing apparatus according to feature 1.

7. A control step that causes an image of a virtual object to be displayed on a display means so as to be placed in real space, A detection step to detect the user's hand and It has, In the control step, the display position of the virtual object is changed so that, from the user's perspective, the virtual object does not overlap with the user's hand. A control method for an information processing device characterized by the following features.

8. A program for causing a computer to function as one of the means of an information processing device described in any one of claims 1 to 6.

Images