Information processing device, information processing method and computer readable medium

By detecting the user's contact action and target area in the actual space, extracting the corresponding images to generate virtual images and displaying them synchronously, the problem of unclear virtual images in the virtual experience is solved, and the seamless connection between the actual space and the virtual space is achieved .

CN112840379BActive Publication Date: 2024-11-26SONY GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980066388.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-10-15
Filing Date
2019-10-02
Publication Date
2024-11-26
Estimated Expiration
2039-10-02

AI Technical Summary

Technical Problem

Existing virtual experience technologies are difficult to seamlessly connect the actual space and the virtual space, resulting in the display of virtual images being not clear and accurate enough.

Method used

By acquiring images in the actual space, detecting the user's contact action and target area, extracting corresponding partial images to generate virtual images, and controlling their display to achieve synchronous display with the user's actions.

Benefits of technology

It achieves seamless connection between real space and virtual space, provides clear and accurate virtual image display, and enhances the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112840379B_ABST
    Figure CN112840379B_ABST
Patent Text Reader

Abstract

An information processing device according to an embodiment of the present technology is equipped with an acquisition unit, an operation detection unit, an area detection unit, and a display control unit. The acquisition unit acquires one or more captured images that capture a real space. The operation detection unit detects a touch operation, which is a series of actions when a user touches a real object in the real space. The area detection unit detects a subject area including the real object based on the detected touch operation. The display control unit extracts a partial image corresponding to the subject area from one or more captured images, generates a virtual image of the real object, and controls the display of the virtual image based on the touch operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method and a program for providing a virtual experience. Background Art

[0002] Patent document 1 describes a system for providing a virtual experience using images of actual space. In this system, a wearable display and a wide-angle camera worn by a first user are used to generate an image representing the first user's field of view. The image is presented to a second user. The second user can input virtual objects such as text and icons into the presented image. In addition, the input virtual objects are presented to the first user. This enables a virtual experience of shared vision between users (Patent document 1, paragraphs

[0015] to

[0017] ,

[0051] ,

[0062] , Figure 1 , Figure 3 wait).

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-95802

[0006] Disclosure of the Invention

[0007] Technical issues

[0008] As described above, a technology for providing various virtual experiences using an image of a real space or the like has been developed, and a technology capable of seamlessly connecting the real space and the virtual space has been required.

[0009] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a program that can seamlessly connect the real space and the virtual space.

[0010] Solution to the problem

[0011] In order to achieve the above object, an information processing device according to an embodiment of the present technology includes an acquisition unit, a motion detection unit, a region detection unit, and a display control unit.

[0012] The acquisition unit acquires one or more captured images capturing the actual space.

[0013] The motion detection unit detects a contact motion which is a series of motions when a user contacts a real object in a real space.

[0014] A region detection unit detects a target region including an actual object corresponding to the detected contact action.

[0015] A display control unit generates a virtual image of the actual object by extracting a partial image corresponding to the target area from one or more captured images, and controls display of the virtual image corresponding to the contact action.

[0016] In the information processing device, a contact action of a user contacting an actual object is detected, and a target area including the actual object corresponding to the contact action is detected. A partial image corresponding to the target area is extracted from a captured image obtained by capturing an actual space where the actual object exists, and a virtual image of the actual object is generated. Then, display control of the virtual image corresponding to the user's contact action is performed. Therefore, it becomes possible to easily display a virtual image capturing the actual object, and the actual space and the virtual space can be seamlessly connected.

[0017] The display control unit may generate a virtual image representing an actual object that is not obscured by the obscuring object.

[0018] This makes it possible to introduce a clear image of an actual object that is not obscured by an obstructing object into the virtual space, and to seamlessly connect the actual space and the virtual space.

[0019] The display control unit may generate a partial image from one or more photographed images in which an object is not included in the target area.

[0020] This makes it possible to easily bring a virtual image representing an actual object into the virtual space without masking. As a result, it becomes possible to seamlessly connect the actual space and the virtual space.

[0021] The display control unit may superimpose the virtual image on the actual object and display it.

[0022] Therefore, a virtual image in which an actual object is replicated is displayed on the actual object. As a result, the virtual image can be easily handled and excellent usability can be demonstrated.

[0023] The acquisition unit may acquire one or more photographed images from at least one of a photographing device that photographs the actual space or a database that stores an output of the photographing device.

[0024] Therefore, for example, it becomes possible to easily generate a virtual image with high accuracy that represents an actual object without occlusion.

[0025] The contact action may include an action of bringing the user's hand closer to the actual object. In this case, the action detection unit may determine whether the state of the contact action is a pre-contact state predicting the contact of the user's hand with respect to the actual object. In addition, if it is determined that the state of the contact action is the pre-contact state, the acquisition unit may acquire one or more captured images by controlling the shooting device.

[0026] Therefore, for example, it becomes possible to capture the actual object immediately before the user touches the actual object. This makes it possible to sufficiently improve the accuracy of the virtual image.

[0027] If the state of the contact action is determined to be the pre-contact state, the acquisition unit may increase the photographing resolution of the photographing device.

[0028] This makes it possible, for example, to generate virtual images with high resolution.

[0029] The motion detection unit may detect a contact position between the actual object and the user's hand. In this case, the region detection unit may detect a target region based on the detected contact position.

[0030] Therefore, for example, it becomes possible to specify a shooting target, a range, and the like by a simple motion, and seamlessly connect the real space and the virtual space.

[0031] The region detection unit may detect a boundary of the actual object including the contact position as the target region.

[0032] Therefore, for example, it becomes possible to accurately separate the actual object and other areas, and generate a highly accurate virtual image.

[0033] The information processing device may further include a sight line detection unit for detecting a sight line direction of the user. In this case, the region detection unit may detect a boundary of the actual object based on the sight line direction of the user.

[0034] Therefore, it becomes possible to improve the separation accuracy between the actual object to be photographed and the target area. As a result, an appropriate virtual image can be generated.

[0035] The sight line detection unit may detect the gaze position based on the user's sight line direction. In this case, the region detection unit may detect a boundary of the actual object including the contact position and the gaze position as the target region.

[0036] Therefore, it becomes possible to greatly improve the separation accuracy between the actual object to be photographed and the target area, and the reliability of the device can be sufficiently improved.

[0037] The area detection unit may detect the boundary of the real object based on at least one of a shadow, a size, or a shape of the real object.

[0038] For example, this makes it possible to accurately detect the boundary of the actual object regardless of the state of the actual object, etc. As a result, it becomes possible to sufficiently improve the usability of the device.

[0039] The motion detection unit may detect the fingertip position of the user's hand. In this case, the region detection unit may detect the target region based on the trajectory of the fingertip position accompanying the movement of the fingertip position.

[0040] This makes it possible to easily set the shooting range, for example.

[0041] The display control unit may superimpose the area image representing the target area on the actual object and display it.

[0042] Therefore, for example, it becomes possible to confirm the target area as the photographing range, and a state such as generation of an unnecessary virtual image can be sufficiently avoided.

[0043] The area image may be displayed so that at least one of the shape, size or position may be edited. In this case, the area detection unit may change the target area based on the edited area image.

[0044] Therefore, it becomes possible to accurately set the photographing range, and, for example, a virtual image of a desired real object or the like can be easily generated.

[0045] The motion detection unit may detect a contact position between the actual object and the user's hand. In this case, the display control unit may control the display of a virtual image corresponding to the detected contact position.

[0046] Therefore, for example, it becomes possible to display a virtual image without a sense of discomfort corresponding to the contact position, and to seamlessly connect the real space and the virtual space.

[0047] The motion detection unit may detect a gesture of the user's hand contacting the actual object. In this case, the display control unit may control the display of a virtual image corresponding to the detected gesture of the user's hand.

[0048] Therefore, for example, it becomes possible to switch the display method of the virtual image corresponding to the gesture of the hand, and provide an easy-to-use interface.

[0049] The virtual image may be at least one of a two-dimensional image or a three-dimensional image of the actual object.

[0050] Therefore, it becomes possible to generate virtual images of various real objects existing in the real space, and to seamlessly connect the real space and the virtual space.

[0051] An information processing method according to an embodiment of the present technology is an information processing method executed by a computer system, and includes acquiring one or more captured images capturing an actual space.

[0052] A contact action is detected, which is a series of actions when a user contacts a real object in a real space.

[0053] A target area including an actual object corresponding to the detected contact action is detected.

[0054] A partial image corresponding to the target area is extracted from one or more captured images to generate a virtual image of the actual object and display of the virtual image corresponding to the contact action is controlled.

[0055] A program according to an embodiment of the present technology causes a computer system to execute the following steps:

[0056] A step of acquiring one or more captured images capturing the actual space;

[0057] The step of detecting a contact action, where the contact action is a series of actions when the user touches a real object in a real space;

[0058] detecting a target area of ​​the actual object corresponding to the detected contact action; and

[0059] The step of generating a virtual image of the actual object by extracting a partial image corresponding to the target area from one or more captured images, and controlling display of the virtual image corresponding to the contact action.

[0060] Advantageous Effects of the Invention

[0061] As described above, according to the present technology, the real space and the virtual space can be seamlessly connected. Note that the effects described here are not necessarily restrictive, and any effect described in the present disclosure can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] [ Figure 1 ] Figure 1 It is a schematic diagram for explaining an outline of the operation of the HMD according to the embodiment of the present technology.

[0063] [ Figure 2 ] Figure 2 It is a perspective view schematically showing the appearance of an HMD according to an embodiment of the present technology.

[0064] [ Figure 3 ] Figure 3 It is shown Figure 2 A block diagram of a configuration example of an HMD shown in FIG.

[0065] [ Figure 4 ] Figure 4 is a flowchart showing an example of the action of the HMD 100 .

[0066] [ Figure 5 ] Figure 5 is a schematic diagram showing an example of a contact action with respect to an actual object of a user.

[0067] [ Figure 6 ] Figure 6 : is a schematic diagram showing an example of detection processing of a photographing area in the area automatic detection mode.

[0068] [ Figure 7 ] Figure 7 2 is a schematic diagram showing another example of the detection process of the photographing area in the area automatic detection mode.

[0069] [ Figure 8 ] Figure 8 : is a schematic diagram showing an example of correction processing of a photographing area.

[0070] [ Fig. 9 ] Fig. 9 is a schematic diagram showing an example of a captured image used to generate a virtual image.

[0071] [ Fig.10 ] Fig.10 is a schematic diagram showing an example of display of a virtual image.

[0072] [ Fig.11 ] Fig.11 is a schematic diagram showing an example of display of a virtual image.

[0073] [ Fig.12 ] Fig.12 is a schematic diagram showing an example of display of a virtual image.

[0074] [ Fig.13 ] Fig.13 is a schematic diagram showing an example of display of a virtual image.

[0075] [ Fig.14 ] Fig.14 is a schematic diagram showing another example of display of a virtual image.

[0076] [ Fig.15 ] Fig.15 : is a schematic diagram showing an example of a detection process of a photographing area including an obstructed object.

[0077] [ Fig.16 ] Fig.16 It is shown by Fig.15 Schematic diagram of an example of a virtual image generated by the detection process shown.

[0078] [ Fig.17 ] Fig.17 is a flowchart showing another example of the operation of the HMD.

[0079] [ Fig.18 ] Fig.18 is a schematic diagram showing an example of a photographing area designated by a user.

[0080] [ Fig.19 ] Fig.19 is a perspective view schematically showing the appearance of an HMD according to another embodiment.

[0081] [ Fig. 20 ] Fig. 20 is a perspective view schematically showing an appearance of a mobile terminal according to another embodiment.

[0082] Mode for carrying out the invention

[0083] Embodiments according to the present technology will now be described below with reference to the drawings.

[0084] HMD Configuration

[0085] Figure 1 1 is a schematic diagram for explaining an outline of the operation of the HMD according to the embodiment of the present technology. The HMD 100 (head mounted display) is a glasses-type device having a transmissive display, and is used by being worn on the head of the user 1 .

[0086] The user 1 wearing the HMD 100 will be able to visually recognize the actual scene and at the same time visually recognize the image displayed on the transmissive display. That is, by using the HMD 100, a virtual image or the like can be superimposed and displayed on the real space (actual space) around the user 1. Therefore, the user can experience augmented reality (AR) or the like.

[0087] Figure 1 A is a schematic diagram showing an example virtual space (AR space) visually seen by the user 1. The user 1a wearing the HMD 100 sits Figure 1 A. For example, an image of another user 1b sitting on the other side of the table is displayed on the display of the HMD 100. As a result, the user 1a wearing the HMD 100 can experience augmented reality as if the user 1a and the other user 1b were sitting face to face.

[0088] Note that the parts shown by the solid lines in the figure (e.g., the chair on which the user 1a sits, the table, the document 2 on the table) are real objects 3 arranged in the real space where the user actually exists. In addition, the parts shown by the dotted lines in the figure (e.g., other users 1b and their chairs) are images displayed on the transmissive display and become virtual images 4 in the AR space. In the present disclosure, the virtual image 4 is an image for displaying various objects (virtual objects) displayed in a virtual space, for example.

[0089] By wearing the HMD 100 in this manner, for example, even when the other user 1b is located at a remote location, a conversation using gestures or the like can be performed naturally, and good communication can be performed. Of course, the present technology can be applied even when the user 1a and another user 1b are in the same space.

[0090] The HMD 100 includes a shooting function that generates a virtual image 4 of a real object 3 in a real space and displays it in an AR space. For example, it is assumed that a user 1a wearing the HMD 100 reaches his hand toward a document 2 on a table and touches the document 2. In this case, in the HMD 100, a virtual image 4 of the document 2 touched by the user 1a is generated. In the present embodiment, the document 2 is an example of a real object 3 in a real space.

[0091] Figure 1 B schematically shows an example contact action of the user 1a contacting the document 2. For example, when the user 1a contacts the document 2, the area of ​​the document 2 to be photographed (the boundary of the document 2) is detected. Based on the detection result, a virtual image 4 (the shaded area in the figure) representing the document 2 contacted by the user 1a is generated and displayed on the HMD 100 display (AR space). A method of detecting the area to be photographed, a method of generating the virtual image 4, etc. will be described in detail later.

[0092] For example, Figure 1 As shown in FIG. 2B , when the user 1a manually rubs the document 2 on the table, the photographed document 2 (virtual image 4) is displayed as if it turns over the actual document 2. That is, the generated virtual image 4 is superimposed on the actual document 2 and displayed as if the actual document 2 is turned over. Note that the user 1a does not need to actually turn over the document 2, and the virtual image 4 can be generated by only performing a gesture of turning over the document 2, for example.

[0093] Therefore, in the HMD 100, the real object 3 (document 2) to be photographed is designated by the hand of the user 1a, and a target virtual image 4 is generated. The photographed virtual image 4 is superimposed and displayed on the target real object. The virtual image 4 of the document 2 displayed in the AR space can be freely displayed in the AR space corresponding to various gestures of the user 1a (e.g., grabbing, moving the virtual image 4, or deforming the virtual image 4).

[0094] Furthermore, the document 2 brought into the AR space as a virtual image 4 can be freely moved in the virtual AR space. For example, Figure 1 C shows that the user 1a grabs the virtual object document 2 (virtual image 4) and hands it to another user 1b at a remote location displayed on the display of the HMD 100. Such communication is possible by using the virtual image 4, for example.

[0095] As described above, in the HMD 100, the actual object 3 existing in the actual space (real world) is simply photographed and presented in the virtual space (virtual world). That is, it can be said that the HMD 100 has a function of simply photographing the actual space. This makes it possible to easily introduce objects in the actual space into a virtual space such as an AR space, and to seamlessly connect the actual space and the virtual space. Hereinafter, the configuration of the HMD 100 will be described in detail.

[0096] Figure 2 : is a perspective view schematically showing the appearance of the HMD 100 according to the embodiment of the present technology. Figure 3 It shows Figure 2 00 is a block diagram of an example configuration of an HMD 100 shown in FIG.

[0097] The HMD 100 includes a frame 10 , left-eye and right-eye lenses 11 a and 11 b , left-eye and right-eye displays 12 a and 12 b , left-eye and right-eye cameras 13 a and 13 b , and an outward-facing camera 14 .

[0098] The frame 10 has a glasses-like shape and includes a frame portion 15 and a temple portion 16. The frame portion 15 is a portion provided in front of the left and right eyes of the user and supports each of the left-eye lens 11a and the right-eye lens 11b. The temple portion 16 extends backward from both ends of the frame portion 15 toward the ears of the user 1, and both ears are worn with a tip. For example, the frame portion 15 and the temple portion 16 are formed of a material such as a synthetic resin and a metal.

[0099] The left eye lens 11a and the right eye lens 11b are respectively arranged in front of the user's left eye and right eye to cover at least a portion of the user's field of view. Generally, each lens is designed to correct the user's vision. Of course, the present invention is not limited to this, and so-called non-power lenses can also be used.

[0100] The left-eye display 12a and the right-eye display 12b are transmissive displays and are arranged to cover partial areas of the left-eye lens 11a and the right-eye lens 11b, respectively. That is, the left-eye lens 11a and the right-eye lens 11b are respectively disposed in front of the left eye and the right eye of the user.

[0101] Images for the left eye and the right eye and the like are displayed on the left eye display 12a and the right eye display 12b, respectively. A virtual display object (virtual object) such as a virtual image 4 is displayed on each of the displays 12a and 12b. Therefore, the user 1 wearing the HMD 100 visually sees a real space scene such as an actual object 3 on which the virtual image 4 displayed on the displays 12a and 12b is superimposed.

[0102] For example, a transmissive organic electroluminescent display, an LCD (Liquid Crystal Display) display, or the like is used as the left-eye display 12a and the right-eye display 12b. In addition, the specific configurations of the left-eye display 12a and the right-eye display 12b are not limited, and for example, any method of transmissive display, such as a method of projecting and displaying an image on a transparent screen or a method of displaying an image using a prism or the like, can be appropriately used.

[0103] The left eye camera 13a and the right eye camera 13b are appropriately placed in the frame 10 so that the left eye and the right eye of the user 1 can be imaged. For example, the line of sight of the user 1, the gaze point at which the user 1 is gazing, etc. can be detected based on the images of the left eye and the right eye taken by the left eye camera 13a and the right eye camera 13b.

[0104] For example, digital cameras including image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors are used as left eye camera 13a and right eye camera 13b. In addition, for example, infrared cameras equipped with infrared illumination (e.g., infrared LEDs) can be used.

[0105] Hereinafter, both left-eye lens 11a and right-eye lens 11b are referred to as lens 11, and left-eye display 12a and right-eye display 12b are referred to as transmissive display 12 in some cases. Left-eye camera 13a and right-eye camera 13b are referred to as inward camera 13 in some cases.

[0106] The outward camera 14 is provided at the center of the frame 10 (frame portion 15) facing outward (on the side opposite to the user 1). The outward camera 14 captures the actual space around the user 1, and outputs a captured image capturing the actual space. For example, the capturing range of the outward camera 14 is set to be substantially the same as the field of view of the user 1, or is set to a wider range than the field of view of the user 1. That is, it can be said that the outward camera 14 captures the field of view of the user 1. In the present embodiment, the outward camera 14 corresponds to a capturing device.

[0107] For example, a digital camera including an image sensor such as a CMOS sensor or a CCD sensor is used as the outward-facing camera 14. In addition, for example, a stereo camera capable of detecting depth information of a real space, a camera equipped with a TOF (Time of Flight) sensor, or the like may be used as the outward-facing camera 14. The specific configuration of the outward-facing camera 14 is not particularly limited, and for example, any camera capable of photographing a real space with desired accuracy may be used as the outward-facing camera 14.

[0108] like Figure 3 As shown, the HMD 100 further includes a sensor unit 17 , a communication unit 18 , a storage unit 20 , and a controller 30 .

[0109] The sensor unit 17 includes various sensor elements for detecting the state of the surrounding environment, the state of the HMD 100, the state of the user 1, etc. In the present embodiment, a distance sensor (depth sensor) for measuring the distance to the target is installed as a sensor element. For example, the above-mentioned stereo camera, etc. are examples of distance sensors. In addition, a LiDAR sensor, various radar sensors, etc. can be used as a distance sensor.

[0110] In addition, as the sensor element, for example, a 3-axis acceleration sensor, a 3-axis gyro sensor, a 9-axis sensor including a 3-axis compass sensor, a GPS sensor for acquiring information on the current position of the HMD 100, etc. can be used. In addition, a biosensor (heart rate) for detecting bio-information of the user 1, such as an electroencephalogram sensor, an electromyogram sensor, or a pulse sensor can be used.

[0111] The sensor unit 17 includes a microphone for detecting the user's voice or sound information of surrounding sounds. For example, the voice uttered by the user is appropriately detected. Therefore, for example, the user can experience AR while making a voice call and perform operation input of the HMD 100 using voice input. In addition, there is no limitation on the sensor element or the like provided as the sensor unit 17.

[0112] The communication unit 18 is a module for performing network communication with other devices, short-range wireless communication, etc. For example, a wireless LAN module such as WiFi and a communication module such as Bluetooth (registered trademark) are provided.

[0113] The storage unit 20 is a nonvolatile storage device, and, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like is used.

[0114] The storage unit 20 stores a captured image database 21. For example, the captured image database 21 is a database storing images of the actual space captured by the outward camera 14. Images of the actual space captured by other cameras other than the outward camera 14 and the like may be stored in the captured image database 21.

[0115] For example, the captured image database 21 stores the captured images of the actual space and the captured information related to the captured state of each captured image in association with each other. For example, when the image is captured, the captured time, the position of the HMD 100 at the time of the capture, the captured direction (HMD 100 posture, etc.), the captured resolution, the captured magnification, the exposure time, etc. are stored as the captured information. In addition, the specific configuration of the captured image database 21 is not limited. In the present embodiment, the captured image database corresponds to a database storing the output of the capturing device.

[0116] Furthermore, the storage unit 20 stores a control program 22 for controlling the overall action of the HMD 100. The method of installing the captured image database 21 and the control program 22 to the HMD 100 is not limited.

[0117] The controller 30 corresponds to the information processing device according to the present embodiment, and controls the actions of the various blocks of the HMD 100. The controller 30 includes a hardware configuration required for a computer, such as a CPU and a memory (RAM, ROM). When the CPU loads the control program 22 stored in the storage unit 20 into the RAM and executes it, various processes are performed.

[0118] For example, a device such as a PLD (Programmable Logic Device) of an FPGA (Field Programmable Gate Array) and other ASICs (Application Specific Integrated Circuits) may be used as the controller 30 .

[0119] In the present embodiment, the CPU of the controller 30 executes the program according to the present embodiment, whereby the image acquisition unit 31, the contact detection unit 32, the line of sight detection unit 33, the area detection unit 34, and the AR display unit 35 are implemented as functional blocks. The information processing method according to the present embodiment is performed by these functional blocks. Note that in order to implement each functional block, dedicated hardware such as an IC (integrated circuit) can be appropriately used.

[0120] Image acquisition unit 31 acquires one or more captured images capturing the actual space. For example, image acquisition unit 31 reads the captured images captured by outward camera 14 by appropriately controlling outward camera 14. In this case, image acquisition unit 31 may acquire images captured in real time.

[0121] For example, when receiving a notification that the user 1 and the actual object 3 are about to contact each other from the contact detection unit 32 to be described later, the image acquisition unit 31 controls the outward camera 14 to start photographing the actual object 3 to be photographed. In addition, in the case where the outward camera 14 is performing continuous photographing, the photographing parameters of the outward camera 14 are changed and switched to photographing a higher resolution image. That is, the image acquisition unit 31 controls the outward camera 14 to switch to a mode for photographing the actual object 3 to be photographed. Figure 5 etc. describe this in detail.

[0122] Furthermore, for example, the image acquisition unit 31 appropriately accesses the storage unit 20 to read the captured image 40 stored in the captured image database 21. That is, the image acquisition unit 31 can appropriately refer to the captured image database 21 and acquire a captured image captured in the past.

[0123] Therefore, in the present embodiment, the image acquisition unit 31 acquires one or more captured images from at least one of the outward camera 14 for capturing the actual space or the captured image database 21 storing the output of the outward camera 14. For example, the acquired captured images are appropriately provided to other functional blocks. In addition, the captured images acquired from the outward camera 14 are appropriately stored in the captured image database 21. In this embodiment, the image acquisition unit 31 corresponds to the acquisition unit.

[0124] The contact detection unit 32 detects a series of contact actions when the user 1 contacts the actual object 3 in the actual space. As detection of the contact action, for example, depth information detected by a distance sensor installed as the sensor unit 17, an image (captured image) of the field of view of the user 1 captured by the outward camera 14, etc. are used.

[0125] In the present disclosure, a contact action is a series of actions (gestures) performed when user 1 contacts an actual object 3, and is generally an action performed by user 1 so that the hand (fingers) of user 1 contacts the actual object 3. For example, a gesture of the user's finger when the hand of user 1 contacts the actual object 3 is a contact action. For example, gestures such as pinching, flipping, grabbing, tapping, and moving document 2 (actual object 3) are included in the contact action. Incidentally, gestures are not limited to gestures performed when contacting the actual object 3. For example, gestures performed in a state where user 1 is not in contact with the actual object 3 (for example, spreading or gathering fingers to pinch the actual object 3) are also contact actions.

[0126] The contact action includes an action of bringing the hand of the user 1 closer to the actual object 3. That is, in order to contact the actual object 3, the action of the user 1 extending the hand toward the actual object 3 to be targeted is also included in the contact action. For example, the action (approach action) of the user 1 moving the hand to approach the document 2 (actual object 3) is a contact action. Therefore, it can be said that the contact detection unit 32 detects a series of actions performed when the user contacts the actual object 3 (such as approach actions and gestures at the time of contact) as the contact action of the user 1.

[0127] The contact detection unit 32 determines the state of the contact action. For example, the contact detection unit determines whether the state of the contact action is a pre-contact state for predicting the contact of the hand of the user 1 with respect to the actual object 3. That is, it is determined whether the hand of the user 1 is likely to contact the actual object 3. For example, when the distance between the finger of the user 1 and the surrounding actual object 3 is less than a certain threshold, it is determined that the hand of the user 1 is likely to contact the actual object 3, and the contact action of the user 1 is in the pre-contact state (see Figure 4 In this case, the state in which the distance between the finger and the actual object 3 is less than the threshold value and the finger is not in contact with the actual object 3 is the pre-contact state.

[0128] In addition, the contact detection unit 32 determines whether the state of the contact action is a contact state in which the hand of the user 1 and the real object 3 are in contact with each other. That is, the contact detection unit 32 detects the contact of the finger of the user 1 with the surface (plane) of the real object 3.

[0129] When contact between the user 1 and the real object 3 is detected, the contact detection unit 32 detects a contact position P between the hand of the user 1 and the real object 3. For example, the coordinates of a position where the hand of the user 1 and the real object 3 contact each other in a predetermined coordinate system set in the HMD 100 are detected as the contact position P.

[0130] The method of detecting the contact action and the like is not limited. For example, the contact detection unit 32 appropriately measures the position of the hand of the user 1 and the position of the surrounding actual object 3 using a distance sensor or the like attached to the HMD 100. Based on the measurement results of the corresponding positions, for example, it is determined whether the state is a pre-contact state, and it is detected whether the hand of the user 1 may contact the actual object 3. In addition, it is determined whether the state is a contact state, and it is determined whether the hand contacts the actual object 3, for example.

[0131] In order to detect whether contact is possible, for example, prediction processing by machine learning, prediction processing using the fact that the distance between the hand of user 1 and the actual object 3 is shortened, etc. are used. Alternatively, processing for predicting the contact of user 1 with the actual object 3 may also be performed based on the moving direction, moving speed, etc. of the hand of user 1.

[0132] In addition, the contact detection unit 32 detects a gesture of the user 1 based on a captured image or the like captured by the outward camera 14. For example, a method of detecting a gesture by detecting the area of ​​a finger in a captured image, a method of detecting the tip of each finger and detecting a gesture, etc. can be appropriately used. A process of detecting a gesture using machine learning or the like can be performed. In addition, the method of detecting a gesture or the like is not limited.

[0133] The sight line detection unit 33 detects the sight line direction of the user 1. For example, the sight line direction of the user 1 is detected based on the images of the left eye and the right eye of the user 1 captured by the inward camera 13. The sight line detection unit 33 detects the gaze position Q based on the user's sight line direction. For example, in the case where the user 1 is looking at a certain real object 3 in the real space, the position where the real object 3 intersects with the sight line direction of the user 1 is detected as the gaze position Q of the user 1.

[0134] The method of detecting the sight line direction and gaze position Q of the user 1 is not limited. For example, in a configuration in which an infrared camera (inward-facing camera 13) and an infrared light source are installed, an image of an eyeball in which a reflection (bright spot) of infrared light emitted from the infrared light source is reflected is captured. In this case, the sight line direction is estimated based on the bright spot of infrared light and the pupil position, and the gaze position Q is detected.

[0135] In addition, a method of estimating the gaze direction and gaze position Q based on feature points such as the corners of the eye may be used based on an image of the eyeball. In addition, the gaze direction or gaze position Q may be detected based on changes in the eye potential or the like generated by the charging of the eyeball. In addition, any algorithm or the like capable of detecting the gaze direction, gaze position Q, etc. of the user 1 may be used.

[0136] The area detection unit 34 detects a shooting area including the actual object 3 corresponding to the contact action detected by the contact detection unit 32. For example, the shooting area is an area for generating a virtual image 4 in which the actual object 3 is shot. That is, it can be said that the area including the actual object 3 to be shot as the virtual image 4 is the shooting area. In the present embodiment, the shooting area corresponds to the target area.

[0137] For example, a captured image capturing a state in which the user 1 is in contact with the actual object 3 (hereinafter, referred to as a contact image) is acquired. The area detection unit 34 analyzes the contact image, and detects a range in the contact image to be captured as the virtual image 4. Note that this is not limited to the case where the captured area is detected from the contact image. For example, the captured area may be detected from a captured image other than the contact image based on the contact position of the user 1 or the like.

[0138] In the present embodiment, an area automatic detection mode for automatically detecting a shooting area is performed. In the area automatic detection mode, for example, the actual object 3 contacted by the user 1 is automatically identified as a shooting target. Then, the extended area representing the surface of the actual object 3 to be photographed, that is, the boundary (periphery) of the actual object 3 contacted by the user 1 can be detected as the shooting area. In addition, the area representing the boundary (periphery) of the actual object 3 related to the actual object 3 contacted by the user 1 can be detected as the shooting area. For example, the boundary of a document on the top surface, back surface, etc. of the document contacted by the user 1 can be detected as the shooting area. Alternatively, when the contact boundary has a document of a binder, a shooting area such as containing another document can be detected.

[0139] In this way, in the area automatic detection mode, it is detected which surface the user 1 is about to touch and to what extent the surface extends. This makes it possible to recognize the range of the surface that the user 1 touches (the range of the document 2, the whiteboard, etc.). The method of automatically detecting the shooting area is not limited, and for example, any image analysis processing capable of detecting an object, recognizing a boundary, etc., or detection processing by machine learning, etc. can be appropriately used.

[0140] Furthermore, in the present embodiment, a region manual designation mode for detecting a photographing region designated by the user 1 is performed. In the region manual designation mode, for example, an action of the user 1 tracking the actual object 3 is appropriately detected, and the range designated by the user 1 is detected as the photographing region. The region automatic detection mode and the region manual designation mode will be described in detail later.

[0141] The AR display unit 35 generates an AR image (virtual image 4) displayed on the transmissive display 12 of the HMD 100 and controls the display thereof. For example, the position, shape, posture, etc. of the displayed AR image are calculated according to the state of the HMD 100, the state of the user 1, etc.

[0142] The AR display unit 35 extracts a partial image corresponding to the photographing area from one or more photographed images to generate a virtual image 4 of the actual object 3. For example, the partial image is an image generated by cutting out a portion corresponding to the photographing area in the photographed image. Based on the cut-out partial image, a virtual image 4 for display in the AR space is generated. Therefore, it can be said that the virtual image 4 is a partial image corresponding to the AR space processing.

[0143] For example, if a real object 3 having a two-dimensional extension, such as a document 2 and a whiteboard, is photographed, a virtual image 4 having a two-dimensional extension is generated for displaying the content written on the surface of the real object 3. In this case, the virtual image 4 is a two-dimensional image of the real object 3.

[0144] In addition, in the HMD 100, the real object 3 having a three-dimensional shape can be photographed. For example, the virtual image 4 is generated so that the stereoscopic shape of the real object 3 can be represented in the AR space. In this case, the virtual image 4 is a three-dimensional image of the real object 3. In this way, the AR display unit 35 generates the virtual image 4 corresponding to the shape of the real object 3.

[0145] In addition, the AR display unit 35 generates a virtual image representing the actual object that is not obscured by the obstructing object. Here, the state of being obscured by the obstructing object (other object) is a state in which a part of the actual object 3 is hidden by the obstructing object. For example, in a contact image captured in a state in which the hand of the user 1 is in contact with the actual object 3, it can be thought that a part of the actual object 3 is obscured by the hand of the user 1. In this case, the hand of the user 1 becomes the obscuration object that obscures the actual object 3.

[0146] In the present embodiment, the AR display unit 35 generates a virtual image 4 in which the real object 3 is not blocked but the entire real object 3 is displayed. Therefore, the virtual image 4 is a clear image representing the entire real object 3 of the photographed object (refer to Fig. 9 For such a virtual image 4, a partial image can be generated from the captured image, for example, in which the actual object 3 is captured without being blocked. Incidentally, a virtual image 4 in which a portion of the actual object 3 is blocked can be generated (see Fig.16 A, etc.).

[0147] The AR display unit 35 displays the generated virtual image 4 on the transmissive display 12 so as to overlap with the actual object 3. That is, a clear image of the actual object 3 (virtual image 4) is superimposed on the actual object 3 and displayed. In addition, the virtual image 4 is displayed corresponding to the hand action (gesture) of the hand of the user 1 that contacts the actual object 3, etc. For example, for each type of action of contacting the actual object 3 (for example, tapping or rubbing the actual object 3), the display type of the virtual image 4 is changed. In this way, the AR display unit 35 controls the display of the virtual image 4 corresponding to the contact action of the user 1.

[0148] A method of generating the virtual image 4 of the real object 3, a method of displaying the virtual image 4, etc. will be described in detail later. In the present embodiment, the AR display unit 35 corresponds to a display control unit.

[0149] HMD Actions

[0150] Figure 4 is a flowchart showing an example of the action of the HMD 100 . Figure 4 The illustrated processing is processing performed in the area automatic detection mode, and is, for example, a loop processing that is repeatedly performed during the operation of the HMD 100 .

[0151] The contact detection unit 32 measures the position of the finger of the user 1 and the surface position of the actual object 3 existing around the finger of the user 1 (step 101). Here, for example, the position of the surface of any actual object 3 existing around the finger is measured. By the way, at this moment, it is not necessary to identify the actual object 3 that the user 1 is about to contact.

[0152] For example, based on the depth information detected by the distance sensor, the position of the finger of the user 1 and the surface position of the actual object 3 in the coordinate system set on the HMD 100 (distance sensor) are measured. In this case, it can be said that the spatial arrangement relationship between the finger of the user 1 and the actual object 3 around the finger is measured. For example, each fingertip of the user 1 pointing to the actual object 3 is detected as a finger position. In addition, for example, the shape of the surface of the actual object 3 near the finger of the user 1 is detected as a surface position.

[0153] Furthermore, in the case where the field of view of the user 1 is captured by the outward-facing camera 14 or the like, the finger position and the surface position (the arrangement of the finger and the actual object) can be appropriately detected based on the depth information and the captured image. By using the outward-facing camera 14, the detection accuracy of each position can be improved. In addition, the method of detecting the finger position and the surface position is not limited.

[0154] The contact detection unit 32 determines whether the finger of the user 1 is likely to contact the surface of the real object 3 (step 102). That is, it is determined whether the state of the contact motion of the user 1 is a pre-contact state of predicted contact.

[0155] For example, a threshold determination of the distance between the finger position and the surface position is performed as the determination of the pre-contact state. That is, it is determined whether the distance between the finger position and the surface position is greater than a predetermined threshold. For example, the predetermined threshold is appropriately set so that the photographing process of the actual object 3 can be appropriately performed.

[0156] For example, if the distance between the finger position of user 1 and the surface position of actual object 3 is greater than a predetermined threshold, it is determined that the finger of user 1 is sufficiently far away from actual object 3 and is not in a pre-contact state ("No" in step 102). In this case, return to step 101, measure the finger position and the surface position at the next moment, and determine whether the state is a pre-contact state.

[0157] If the distance between the finger position and the surface position is equal to or less than the predetermined threshold, it is determined that the finger of the user 1 is in a state close to the actual object 3 and is in a pre-contact state of predicted contact ("Yes" in step 102). In this case, the image acquisition unit 31 controls the outward camera 14 and starts shooting the actual space with a setting suitable for shooting (step 103). That is, when it is predicted that an interaction occurs between the actual object 3 and the user 1, the shooting mode is switched and specific shooting is started.

[0158] Specifically, various photographing parameters such as photographing resolution, exposure time, and photographing interval of the outward camera 14 are set as values ​​for photographing by the image acquisition unit 31. For example, the values ​​for photographing are appropriately set so that a desired virtual image 4 can be generated.

[0159] For example, in a configuration in which the outward camera 14 always captures the field of view of the user 1, the capture resolution for monitoring is set to suppress the amount of image data. The capture resolution for monitoring is changed to a capture resolution for more detailed capture. That is, in a case where the state of the contact action is determined to be a pre-contact state, the image acquisition unit 31 increases the capture resolution of the outward camera 14. For example, this makes it possible to generate a detailed captured image (virtual image 4) with high resolution.

[0160] Furthermore, for example, the exposure time of the outward camera 14 is appropriately set so that an image with desired brightness and contrast is captured. Alternatively, as will be described later, the capturing interval is appropriately set so that a sufficient number of captured images can be captured.

[0161] When each shooting parameter of the outward camera 14 is set to a value for shooting and the shooting mode is switched, shooting of the actual space by the outward camera 14 (shooting of the field of view of the user 1) is started. The shot image shot by the outward camera 14 is appropriately read by the image acquisition unit 31. For example, the shooting process is repeatedly performed until a predetermined condition for generating the virtual image 4 is satisfied.

[0162] Figure 5 is a schematic diagram showing an example of a contact action of the user 1 with respect to the actual object 3 . Figure 5 A schematically shows the finger 5 of the user 1 and the actual object 3 (document 2) at the moment determined to be in the pre-contact state. Figure 5 In the state shown in A, there is no recognition Figure 5 Whether the document 2 shown in A is the target of the contact action (target to be photographed).

[0163] exist Figure 5 In the state shown in A, the shooting area of ​​the outward camera 14 ( Figure 5 The dotted line in A) includes the finger 5 of the user 1 and a part of the document 2. For example, a photographic image with high resolution is photographed within such a photographing range. In this case, the photographed image is an image in which only a part of the document 2 is photographed.

[0164] Figure 5 B shows a diagram in which the finger 5 of the user 1 is Figure 5 The state shown in A is closer to the pre-contact state of the actual object 3. Figure 5 In the state shown in B, the entire document 2 is included in the photographing area of ​​the outward camera 14. The finger 5 of the user 1 does not touch the document 2, and the document 2 is photographed without being blocked by the blocking object. Figure 5The captured image captured in the state shown in B becomes an image in which the document 2 (actual object 3) which is not shielded by the shielding object is captured.

[0165] Figure 5 C shows a contact state in which the finger 5 of the user 1 and the actual object 3 are in contact with each other. Even in the contact state, the photographing process of the outward camera 14 can be continued. In this case, the entire document 2 is included in the photographing range of the outward camera 14, but a part of the document 2 is blocked by the finger of the user 1. In this case, the captured image is an image in which a part of the document 2 is blocked.

[0166] In the shooting process of the outward camera 14, for example Figure 5 A to Figure 5 C, and appropriately read the captured images in each state. Therefore, in the case where the state of the contact action is determined to be the pre-contact state, the image acquisition unit 31 controls the outward camera 14 to acquire one or more captured images. That is, it can be said that the image acquisition unit 31 acquires an image (captured image) captured by the capture setting.

[0167] The period of time during which the photographing process for photographing by the outward camera 14 is performed is not limited. For example, the photographing process may be continued until the virtual image 4 is generated. Alternatively, the photographing process may be ended when a predetermined number of photographing processes are performed. Furthermore, for example, after a predetermined number of photographing processes, if there is no photographic image required to generate the virtual image 4, the photographing process may be restarted. In addition, the number of times, time, etc. of the photographing process may be appropriately set so that the virtual image 4 may be appropriately generated.

[0168] return Figure 4 , when the photographing process for photographing is started, it is determined in step 104 whether the finger 5 of the user 1 contacts the surface of the real object 3. That is, it is determined whether the state of the contact action of the user 1 is the contact state.

[0169] , for example, a threshold determination of the distance between the finger position and the surface position is performed as the determination of the contact state. For example, when the distance between the finger position and the surface position is greater than the contact detection threshold, it is determined that there is no contact state, and when the distance is equal to or less than the contact detection threshold, it is determined that there is a contact state. The method of determining the contact state is not limited.

[0170] For example, in Figure 5 A and Figure 5In step B, the distance between the finger 5 of the user 1 and the actual object 3 (document 2) is greater than the threshold value of contact detection. In this case, it is determined that the finger 5 of the user 1 is not in contact with the surface of the actual object 3 ("No" in step 104), and the determination of the contact state is performed again.

[0171] In addition, for example, in Figure 5 C, the distance between the finger 5 of the user 1 and the actual object 3 is equal to or less than the threshold value for detecting contact. In this case, it is determined that the finger 5 of the user 1 is in contact with the surface of the actual object 3 ("Yes" in step 104), and the area detection unit 34 performs detection processing of the range (photographing area) of the surface contacted by the finger 5 of the user 1 (step 105).

[0172] Figure 6 : is a schematic diagram showing an example of detection processing of a photographing area in the area automatic detection mode. Figure 6 A captured image 40 (contact image 41) captured at the moment when the finger 5 of the user 1 is in contact with the document 2 (actual object 3) is schematically shown. Incidentally, the finger 5 of the user 1 is schematically shown using a dotted line.

[0173] exist Figure 6 In the example shown, the finger 5 of the user 1 is in contact with the document 2 placed on the uppermost of the plurality of documents 2 arranged in an overlapping manner. Therefore, the uppermost document 2 is the target of the contact action of the user 1, that is, the photographic subject.

[0174] In this embodiment, when contact is detected, the contact position P between the real object 3 and the hand of the user 1 is detected by the contact detection unit 32. Figure 6 , the position where the fingertip of the index finger of the user 1 contacts the uppermost document 2 is detected as the contact position P. Note that when the user 1 contacts the actual object 3 with multiple fingers, the position of the fingertip of each finger contacting the actual object 3 or the like may be detected as the contact position P.

[0175] exist Figure 6 In the illustrated process, the photographing area 6 is detected based on the contact position P detected by the contact detection unit 32. Specifically, the area detection unit 34 detects the boundary 7 of the actual object 3 including the contact position P as the photographing area 6. Here, the boundary 7 of the actual object 3 is the outer edge of the surface of a single actual object 3, and is a boundary representing the range of the continuous surface of the actual object 3, for example.

[0176] For example, in the contact image 41, the contact position P is detected on the top document 2. That is, the top document 2 becomes the actual object 3 including the contact position P. The area detection unit 34 performs predetermined image processing to detect the boundary 7 of the top document 2. That is, using the contact point (contact position P) of the surface contacted by the finger 5 of the user 1 as a clue, the continuous surface area (photographing area 6) is automatically detected by image processing. Figure 6 In the example shown, a rectangular photographing area 6 corresponding to the boundary 7 of the uppermost document 2 is detected.

[0177] For example, a region where the color in the contact image 41 changes discontinuously is detected as the boundary 7. Alternatively, the boundary 7 may be detected by detecting a continuous line (e.g., a straight line or a curved line) in the contact image 41. When the object to be photographed is the document 2 or the like, the boundary 7 may be detected by detecting the arrangement of characters on the surface of the document or the like.

[0178] In addition, for example, in the case of a thick document 2, a flipped document 2, etc., a shadow may be generated at its outer edge. The boundary 7 of the actual object 3 can be detected based on the shadow of the actual object 3. As a result, the photographing area 6 of the actual object 3 having the same color as the background color can be correctly detected.

[0179] In addition, the boundary 7 of the actual object 3 can be detected based on the size of the actual object 3 to be photographed. For example, the size of the actual object 3 is a size in the actual space, and is appropriately estimated based on the size of the hand of the user 1, depth information, and the like. For example, a range of the size held by the user 1 is appropriately set, and the boundary 7 of the actual object 3, etc. is detected so that it falls within the range. Therefore, for example, when the hand contacts the document 2 (actual object 3) placed on the table, the boundary 7 of the document 2 is detected instead of the table. As a result, detection of unnecessary large or small-sized boundaries, etc. is prevented, and the shooting area 6 can be appropriately detected.

[0180] In addition, for example, for an actual object 3 having a fixed shape, such as a document 2, etc., the boundary 7 of the actual object 3 can be detected based on the shape. For example, the shape of the actual object 3 is a shape in the actual space. For example, by performing a correction process such as a keystone correction on the contact image 41 shot obliquely, the shape observed from the front can be estimated. For example, the boundary 7 of the document 2 having an A4 shape, a postcard shape, etc. is detected based on information about the shape such as the aspect ratio. Incidentally, information about the size and shape of the actual object 3 can be acquired, for example, via an external network, etc., or can be acquired based on a past shot image 40 stored in a shot image database 21, etc. In addition, any method capable of detecting the boundary 7 of the actual object 3 can be used.

[0181] Figure 7is a schematic diagram showing another example of detection processing of a photographing area in the area automatic detection mode. Figure 7 In the illustrated process, the photographing area 6 is detected based on the contact position P and gaze position Q of the user 1. That is, the sight line of the user 1 is used to detect the spread of the surface that the finger 5 of the user 1 is about to touch.

[0182] For example, the sight line detection unit 33 detects the gaze position Q of the user 1 in the contact image 41 based on the sight line direction of the user 1 detected at the time of capturing the contact image 41. Figure 7 As shown, since the user 1 is likely to simultaneously view the selected actual object 3 (the topmost document 2 ) through the line of sight, it is likely that the gaze position Q of the user 1 is detected on the actual object 3 .

[0183] exist Figure 7 In the process shown, the boundary 7 of the actual object 3 including the contact position P and the gaze position Q is detected by the area detection unit 34 as the photographing area 6. That is, the boundary 7 of the continuous surface where the contact position P and the gaze position Q exist is detected. As a method of detecting the boundary 7, for example, using a reference Figure 6 The various methods described above can significantly improve the detection accuracy of the photographing area 6 (the boundary 7 of the target actual object 3).

[0184] Note that it is not limited to the case of using the gaze position Q. For example, a process may be performed to calculate the gaze area of ​​the user 1 based on the gaze direction of the user 1 and detect the boundary 7 of the actual object 3 including the contact position P and the gaze area in the contact image 41. In addition, the boundary 7 of the actual object 3 may be detected by any method using the gaze direction of the user 1 or the like.

[0185] In this way, the area detection unit 34 detects the boundary 7 of the real object 3 based on the line of sight direction of the user 1. It thus becomes possible to determine the target that the user 1 is trying to contact with high accuracy, and to appropriately detect the boundary 7. As a result, it becomes possible to accurately photograph the real object 3 desired by the user 1, and the reliability of the device is improved.

[0186] Note that, in the case where the user 1 is observing at a place other than the contact object, etc., the contact position P and the gaze position Q cannot be detected on the same real object 3. In such a case, the boundary 7 of the real object 3 including the contact position P is detected as the photographing area 6. Therefore, the state in which the wrong area is detected can be fully avoided.

[0187] pass Figure 6 , Figure 7 The information on the photographing area 6 (the boundary 7 of the actual object 3 ) detected by the processes shown in FIG. 1 and FIG. 2 is output to the AR display unit 35 .

[0188] In the present embodiment, the AR display unit 35 superimposes and displays each area image 42 representing the photographing area 6 on the real object 3. For example, Figure 6 and Figure 7 In the example shown, each area image 42 representing the border 7 of the top document 2 is generated and displayed on the transmissive display 12 so as to overlap with the border 7 of the top document 2. As a result, the user 1 will be able to visually see the area on the actual space to be photographed.

[0189] The specific configuration of the area image 42 is not limited. For example, the shooting area 6 may be represented by a line displayed in a predetermined color, etc. Alternatively, the line representing the shooting area 6 may be displayed by an animation such as blinking, etc. In addition, the entire shooting area 6 may be displayed using a predetermined pattern with transparency, etc.

[0190] Note that, for example, even if the viewpoint of the user 1 (HMD 100) changes, the area image 42 can be displayed by appropriately adjusting the shape, display position, etc. so as to overlap the actual object 3. Therefore, the photographing area 6 (rectangular area frame, etc.) that can be seen through the AR display is corrected by manual operation as described below.

[0191] return Figure 4 When the photographing area 6 is detected, the input operation of the user 1 for modifying the photographing area 6 is accepted (step 106). That is, in step 106, the user 1 will be able to modify the photographing area 6 manually.

[0192] Figure 8 is a schematic diagram showing an example of correction processing of the imaging area 6 . Figure 8 is with reference Figure 6 and Figure 7 An image similar to the illustrated contact image 41. In the border 7 of the uppermost document 2 (actual object 3), an area image 42 for correction is schematically shown.

[0193] In the present embodiment, the area image 42 is displayed so that at least one of the shape, size, or position can be edited. For example, in the HMD 100, the input operation of the user 1 on the display screen (transmissive display 12) is detected by detecting the position of the finger 5 of the user 1. The area image 42 is displayed so as to be editable corresponding to the input operation (correction operation) of the user 1.

[0194] exist Figure 8In the example shown, the fingertip of the left hand of the user 1 is arranged at a position overlapping with the left side of the photographing area 6. In addition, the fingertip of the right hand of the user 1 is arranged at a position overlapping with the right side of the photographing area 6. In this case, the AR display unit 35 receives an operation input from the user 1 for selecting the left and right sides of the photographing area 6. Figure 8 In FIG. 6 , the selected left and right sides are shown using dotted lines. In this way, the display of the photographing area 6 can be appropriately changed to indicate that each portion is selected.

[0195] For example, if the user 1 moves the left hand to the left and the right hand to the right, the left side of the photographing area 6 is dragged to the left and the right side is dragged to the right. As a result, the user 1 expands the visible photographing area 6 in the left-right direction by spreading it with the hands, and modifies the size and shape. Of course, the photographing area 6 can also be expanded in the up-down direction.

[0196] In addition, the position of the photographing area 6 may also be modifiable. For example, if the user 1 places the finger 5 within the photographing area 6 and moves the finger 5, a correction operation may be accepted, for example, to move the photographing area 6 corresponding to the movement direction of the finger or the movement amount of the finger. In addition, the area image 42 is displayed so as to be able to accept any correction operation corresponding to the hand operation of the user 1.

[0197] In this way, the range of the actual object 3 to be photographed is automatically determined by the detection processing of the photographing area 6, but the range can be further manually corrected. This makes it possible to easily perform fine adjustment of the photographing area 6, and to generate a virtual image 4, etc. in which the range desired by the user 1 is appropriately photographed. After the modification operation of the user 1 is completed, the photographing area 6 is changed based on the edited area image 42.

[0198] Note that while the modification (editing) of the photographing area 6 is being performed, the photographing process of the photographing image 40 for photographing described in step 103 may be continued. In this case, the process of changing the settings of the outward camera 14 for photographing to the photographing parameters that are optimal for photographing the edited photographing area 6 is performed.

[0199] For example, if the outward camera 14 has an optical zoom function or the like, the optical zoom ratio or the like of the outward camera 14 is appropriately adjusted corresponding to the edited photographing area 6. Therefore, for example, even when the size of the photographing area 6 is small, a virtual image 4 with a high resolution or the like can be generated. Of course, other photographing parameters can be changed.

[0200] Incidentally, the process of manually correcting the photographing region 6 may not be performed. In this case, it is possible to shorten the time for displaying the virtual image 4. Also, a mode for modifying the photographing region 6 may be selected.

[0201] return Figure 4 , a virtual image 4 is generated based on the captured image 40 captured by the outward camera 14 (step 107). Specifically, a clear partial image of the capture area 6 is extracted from the captured image 40 (captured video) captured in step 103. Then, using this partial image, a virtual image 4 of the captured actual object 3 is generated.

[0202] In the present embodiment, the AR display unit 35 generates a partial image based on a captured image 40 that does not include an obstructing object in the capturing area 6 among one or more captured images 40 captured by the outward camera 14. That is, a partial image corresponding to the capturing area 6 is generated by using a frame of a captured image that is not obstructed by the obstructing object (the finger of the user 1).

[0203] For example, the actual object 3 to be photographed is detected from each photographed image 40 photographed after the pre-contact state is detected. The actual object 3 to be photographed is appropriately detected by using matching processing such as feature point matching. The method of detecting the photographing target from each photographed image 40 is not limited.

[0204] It is determined whether the actual object 3 to be photographed included in each photographed image 40 is shielded. That is, it is determined whether the photographing area 6 in each photographed image 40 includes a shielded object. For example, if the boundary 7 of the actual object 3 to be photographed is discontinuously cut, it is determined that the actual object 3 is shielded. In addition, for example, if each finger 5 of the user 1 is detected in each photographed image 40 and each finger 5 is included in the photographing area 6, it is determined that the actual object 3 is shielded. The method of determining the presence or absence of shielding is not limited.

[0205] Among the captured images 40, the captured image 40 in which the actual object 3 to be captured is determined to be not blocked is selected. Therefore, the captured image 40 in which the actual object 3 to be captured is not blocked, that is, the captured image 40 in which the actual object 3 to be captured is captured in a clear manner is used as an image for generating the virtual image 4.

[0206] Fig. 9 is a schematic diagram showing an example of a captured image used to generate a virtual image. Fig. 9 The captured image 40 shown is a representation of Figure 5 B is a schematic diagram of a captured image 40 captured in the pre-contact state.

[0207] exist Fig. 9In the captured image 40 shown, the entire document 2 as the actual object 3 to be captured is captured. The document 2 includes a clear image of the document 2 that is not hidden by the finger 5 of the user 1 and is not blocked by the blocking object. The AR display unit 35 generates a partial image 43 corresponding to the captured area 6 from such a captured image 40. Fig. 9 , the partial image 43 (document 2) to be generated is indicated by the shaded area.

[0208] Note that the captured image 40 may include an image in which a portion of the captured area 6 (actual object 3) is cut out (see Figure 5 A), an image in which a part of the imaged area 6 (actual object 3) is blocked (see Figure 5 C) etc. For example, the partial image 43 can be generated by supplementing the clear portion of the photographing area 6 in these images. For example, such processing is also possible.

[0209] When the partial image 43 is generated, correction processing such as keystone correction is performed. For example, if the captured image 40 is captured from an oblique direction, even a rectangular document may be captured so as to be deformed into a trapezoidal shape. Such deformation is corrected by the keystone correction processing, and, for example, a rectangular partial image 43 is generated. In addition, a noise removal processing for removing a noise component of the partial image 43, a processing for correcting the color, brightness, etc. of the partial image 43, etc. may be appropriately performed.

[0210] A virtual image 4 for displaying the partial image 43 (the actual object 3 to be photographed) in the AR space is generated based on the partial image 43. That is, a virtual image 4 for displaying the planar partial image 43 in the three-dimensional AR space is appropriately generated.

[0211] Therefore, in the present embodiment, when contact between the real object 3 and each finger 5 of the user 1 is predicted, the shooting mode of the outward camera 14 is switched, and a detailed shot image 40 is continuously shot. Then, when the real object 3 (shooting target) brought into the virtual world is specified by the contact of each finger 5, the shot image is backtracked, and a clear virtual image 4 of the real object 3 is generated using an image (shot image 40) in which each finger 5 of the user 1 does not overlap. Therefore, the user 1 will be able to easily create a high-quality copy (virtual image 4) of the real object 3 with a simple operation.

[0212] The AR display unit 35 displays the virtual image 4 superimposed on the actual object 3 (step 108). That is, the user 1 will be able to visually see the virtual image 4 displayed as superimposed on the actual object 3 photographed in reality. For example, by displaying the photographed image of the actual object 3 (virtual image 4) on the actual object 3, the user 1 can intuitively understand that the actual object 3 is copied into the AR space.

[0213] The virtual image 4 of the real object 3 copied from the real space can be freely processed in the AR space. Therefore, for example, the user 1 can perform operations such as grabbing the copied virtual image 4 and passing it to the remote partner (see Figure 1 ) action. As described above, by using this technology, information in the real space will be able to be easily brought into the virtual space.

[0214] Figures 10 to 13 are schematic diagrams each showing an example of display of the virtual image 4. In the present embodiment, the contact detection unit 32 detects a gesture of the hand of the user 1 contacting the actual object 3. The AR display unit 35 controls the display of the virtual image 4 corresponding to the gesture of the hand of the user 1 detected by the contact detection unit 32.

[0215] That is, when the user 1 specifies a photographing target, the virtual image 4 is superimposed on the actual object 3 corresponding to the specified operation. Figures 10 to 13 , a modification of the overlapping display of the captured image (virtual image 4 ) corresponding to the posture (gesture) of the hand of the user 1 will be described.

[0216] exist Fig.10 In the example shown, a gesture is performed in which a user 1 flips a document 2 (actual object 3). Fig.10 As shown in the figure above, assume that user 1 touches the corner of document 2 with the thumb and index finger open. In this case, as Fig.10 As shown in the lower figure of , the display of the virtual image 4 is controlled so as to display the corner of the document 2 being flipped between the thumb and index finger of the user 1 . Fig.10 The display example shown is the same as Figure 1 The display example shown in B is the same.

[0217] For example, in a state where the periphery of the contact position P is flipped, the virtual image 4 is superimposed and displayed on the actual document 2. Therefore, the virtual image 4 is displayed in the same manner as the actual paper, and a visual effect is presented. As a result, even in the AR space, a natural virtual experience in which the actual document 2 is flipped can be provided.

[0218] Furthermore, for example, the virtual image 4 may be displayed only near the position (corner of the document 2) touched by each finger of the user 1. In this case, when the user 1 performs an action of grabbing the virtual image 4, processing such as displaying the entire virtual image 4 is performed.

[0219] In this way, the display of the virtual image 4 can be controlled corresponding to the contact position P detected by the contact detection unit 32. Therefore, immediately after the user 1 contacts the actual object 3 (document 2), the virtual image 4 is displayed only in the vicinity of the contact position P, so that the processing amount of image processing and the like can be suppressed. This makes it possible to display the virtual image 4 smoothly without a sense of incongruity. In addition, unnecessary processing is avoided, so that the power consumed by the HMD 100 can be suppressed.

[0220] exist Fig.11 In the example shown, a gesture is performed in which the user 1 pinches and lifts the central portion of the document 2 (actual object 3). Fig.11 As shown in the above figure, when the user 1 performs an operation of pinching the document 2 with the thumb and index finger, the document 2 of the virtual image 4 (virtual paper) is superimposed and displayed on the actual document 2 in the pinched shape.

[0221] like Fig.11 As shown in the figure below, when the user 1 removes the hand from the virtual image 4, the virtual image 4 remains at that position. At this time, the virtual image 4 is displayed in a manner that is restored from the pinched shape to a flat shape and remains floating above the actual document 2. In this case, for example, the user 1 can grab and move the virtual image 4 displayed floating in the air. By the way, after the user 1 releases his hand, the virtual image 4 can be gradually lowered to a position directly above the actual document 2.

[0222] In addition, in a pinch gesture, when an actual object 3 such as a document 2 is brought into the AR space, the photographed actual object 3 existing in the actual space may be grayed out. That is, a process of filling the actual object 3 as a copy source with gray may be performed. By graying out the actual object 3 in this way, it becomes possible to easily present that a clone of the actual object 3 is generated in the AR space.

[0223] By the way, the object after shooting, that is, the copied virtual image 4, can be marked so as to be called a virtual object on AR. Therefore, it becomes easy to distinguish the virtual image 4 from the actual object 3. In the case of performing other gestures, graying processing, AR marker adding processing, etc. can be appropriately applied.

[0224] exist Fig.12 In the example shown, a gesture of user 1 tapping document 2 (actual object 3) is performed. Fig.12 As shown in the figure above, it is assumed that user 1 taps the surface of actual document 2 with his fingertips. In this case, Fig.12 As shown in the figure below, the virtual image 4 is superimposed and displayed on the actual document 2 as if it were floating. At this time, an effect can be added so that the two-dimensional virtual image 4 is curved and floats like an actual paper.

[0225] Furthermore, processing may be performed so that the virtual image 4 is gradually raised and displayed from the position tapped by the user 1. Furthermore, for example, when a gesture of the user 1 momentarily rubbing the actual document 2 is performed, processing may be performed in which the virtual image 4 is raised in the rubbing direction.

[0226] exist Fig.13 In the example shown, the user 1 performs a gesture of holding a cylindrical real object 3. Such a three-dimensional real object 3 can also be photographed. For example, Fig.13 As shown in the above figure, it is assumed that user 1 grasps or holds actual object 3. For example, the state of force applied to actual object 3 is detected from the arrangement of fingers 5 of user 1, etc. In this case, as Fig.13 As shown in the following figure, a virtual image 4 that replicates the cylindrical actual object 3 is appropriately generated, and is gradually displayed near the actual object 3 in such a manner that the virtual image 4 is extruded.

[0227] In this case, the virtual image 4 is a three-dimensional image representing a stereoscopic real object 3. For example, a three-dimensional image is generated by 3D shooting of shooting a three-dimensional real object 3 (stereoscopic object). For example, in 3D shooting, other cameras other than the outward camera 14 are also used in combination to shoot the real object 3. Then, based on the shot image 40 shot by each camera and the depth information detected by the distance sensor, etc., 3D modeling of the real object 3 is performed. By the way, even when shooting a planar real object 3, other cameras can be used in combination therewith.

[0228] When presenting a captured image (virtual image 4 representing a 3D model), it may take longer to display in order to perform modeling, etc. In such a case, a rough virtual image 4 (3D model) may be initially presented and replaced with gradually highly accurate data. This enables the virtual image 4 to be displayed at high speed even in the case where a stereoscopic real object 3, etc. is captured.

[0229] Fig.14 is a schematic diagram showing another example of display of a virtual image. Fig.14 In the example shown, a virtual image 4 is displayed corresponding to a gesture of a user 1 tapping a document 2 (real object 3). Fig.14 In the example shown, a virtual image 4 displaying an icon 44 indicating that processing is in progress is generated in a frame in which the shape of the document 2 (the shape of the photographing area 6) is replicated.

[0230] For example, in the case where a virtual image 4 of an actual object 3 is generated, as described above, processing such as noise removal and keystone correction is performed on a partial image 43. It may take some time to perform processing so that the actual object 3 generates a captured virtual image 4. Therefore, an icon 44 indicating that processing is in progress or the like is displayed instead of the captured image until the final virtual image 4 is generated.

[0231] Incidentally, when the final virtual image 4 is generated, the display is switched from the icon 44 indicating that the process is in progress to the final virtual image 4 replicating the actual object 3. The type of the icon 44, the method of switching the display, etc. are not limited. For example, a fade-in process may be performed so that the final virtual image 4 gradually becomes darker.

[0232] In the above description, the photographing process is performed on the document 2 which is arranged at the uppermost part and is not shielded as an example of the real object 3. For example, the present technology is also applicable to the real object 3 shielded by other real objects 3 and the like.

[0233] Fig.15 : is a schematic diagram showing an example of a detection process of a photographing area including an obstructed object. Fig.16 It is shown by Fig.15 Schematic diagram of an example of a virtual image generated by the detection process shown.

[0234] Fig.15 The first to third files 2a to 2c of partially overlapping arrangement are schematically shown. The first file 2a is the rearmost file and is partially shielded by the second file 2b. The second file 2a is arranged between the first file 2a and the third file 2c and is partially shielded by the third file 2c. The third file 2c is the topmost file and is not shielded.

[0235] For example, assume that the finger 5 of the user 1 touches the surface of the second document 2b. In this case, the area detection unit 34 detects the boundary 7 of the second document 2b. Fig.15 As shown, a portion of the boundary 7 of the second document 2b (dashed line in the figure) is obscured by the third document 2c. The obscured boundary 7 is detected by appropriate supplementation, for example, based on the unobscured boundary 7 (thick solid line in the figure).

[0236] Therefore, the area to be cut (shooting area 6) is determined by automatically detecting the shooting area 6, but the actual object 3 to be cut (the second document 2b) may be partially hidden. In this case, in the shooting image 40 shot by the outward camera 14, it can be imagined that other shielding objects are located on top of the expected actual object 3, and a certain part cannot be shot.

[0237] In the AR display unit 35, for example, by Fig.16 A to Fig.16The method shown in C generates a virtual image 4 of the actual object 3 (second document 2b) obscured by the obscuring object.

[0238] exist Fig.16 In the example shown in A, a virtual image 4 representing a state blocked by a blocking object is generated as it is. For example, a captured image 40 including a captured area 6 is appropriately selected from the captured images 40 captured by the outward camera 14. Then, a partial image 43 corresponding to the captured area 6 is generated from the selected captured image 40, and a virtual image 4 using the partial image 43 is generated.

[0239] therefore, Fig.16 The virtual image 4 shown in A is an image showing a case where a portion of the second document 2b is shielded by the third document 2c. Therefore, by using the partial image 43 as it is, it becomes possible to shorten the process of generating the virtual image 4 and improve the response speed to the interaction of the user 1.

[0240] exist Fig.16 In the example shown in B, a virtual image 4 is generated in which the portion blocked by the blocking object is grayed out. Fig.16 A detects the boundary 7 of the actual object 3 in the partial image 43 generated in the same manner as in FIG. 1A. That is, the boundary 7 of the shielding object (third document 2c) included in the partial image 43 is detected. Then, a virtual image 4 is generated in which the inside of the boundary 7 of the shielding object is filled with grayscale. By filling unnecessary information in this way, it becomes possible to clearly present the missing portion.

[0241] exist Fig.16 In the example shown in C, a virtual image 4 is generated in which the portion blocked by the blocking object is supplemented by other data. For example, based on the description of the front side of the second document 2b, the captured image database 21 is referenced, and a captured image 40 of a document 2 similar to the second document 2b is searched, etc. A similar document 2 is searched using a predetermined matching process, etc.

[0242] In the case of searching for a captured image 40 including a similar document 2, a partial image 43b of a missing portion shielded by the third document 2c is generated from the captured image 40. Then, a virtual image 4 of the second document 2b is generated using the partial image 43a of the unshielded area and the partial image 43b of the missing portion. Therefore, the virtual image 4 is an image in which the two partial images 43a and 43b are combined.

[0243] In this way, by querying the captured image database 21 or the like, the missing portion is supplemented from a similar document of the target document 2. Therefore, even when the actual object 3 shielded by the shielding object becomes the captured target, it becomes possible to generate a virtual image 4 representing the actual object 3 that is not shielded. Note that since there is a possibility that the searched similar document is different from the target document 2, the supplement area is clearly displayed by using a frame line (a dotted line in the figure) or the like. Therefore, it becomes possible to notify that the virtual image 4 is supplemented and generated.

[0244] Fig.17 is a flowchart showing other example actions of the HMD 100 . Fig.17 The illustrated processing is processing performed in the area manual designation mode, and is, for example, a loop processing repeatedly performed during the action of the HMD 100. The following describes processing when the user 1 manually designates the photographing area 6 (area manual designation mode).

[0245] exist Fig.17 In steps 201 to 203 shown, for example, the execution Figure 4 The same processing as steps 101 to 103 in the area automatic detection mode shown in FIG. 206 to 208 is performed using, for example, the photographing area 6 manually specified by the user 1. Figure 4 The same processing is performed in steps 206 to 208 as shown.

[0246] The finger position of user 1 and the surface position of actual object 3 are measured (step 201), and it is determined whether finger 5 of user 1 is likely to contact the surface of actual object 3 (step 202). If it is determined that finger 5 of user 1 is unlikely to contact the surface (not a pre-contact state predicting contact) ("No" in step 202), step 201 is performed again.

[0247] If it is determined that the finger 5 of the user 1 may be in contact with the surface (pre-contact state predicting contact) ("Yes" in step 202), the photographing process is started using the outward camera 14 with a setting suitable for photographing (step 203). This photographing process is repeatedly performed, for example, until the virtual image 4 is generated.

[0248] When the photographing process is started, the detection process of the photographing area 6 designated by the user 1 is performed (step 204). More specifically, the fingertip position R of the user 1 is tracked, and information of range designation is acquired. The designated range is appropriately displayed on the AR space.

[0249] Fig.18 1 is a schematic diagram showing an example of the imaging area 6 designated by the user 1 . Fig.18 A state in which the user 1 moves the index finger 5 so as to trace the outer periphery of the document 2 as the actual object 3 is schematically shown.

[0250] When the area manual designation mode is executed, the fingertip position R of the hand of the user 1 is detected by the contact detection unit 32. For example, the fingertip position of the finger 5 of the user 1 at the position closest to the actual object 3 is detected as the fingertip position R. Note that the finger 5 of the user 1 may be in contact with or away from the surface of the actual object 3. That is, regardless of whether the state of the contact action of the user 1 is the contact state or the pre-contact state, the fingertip position R of the user 1 is appropriately detected.

[0251] The information of the fingertip position R of the user 1 is sequentially recorded by the user 1 as the range specifying information. Fig.17 As shown, step 204 is a loop process, and for example, every time step 204 is executed, information of the fingertip position R of the user 1 is recorded. That is, it can be said that tracking processing of the fingertip position R for recording the trajectory 8 of the fingertip position R of the user 1 is performed.

[0252] Fig.18 The fingertip position R of the user 1 is schematically shown using a black circle. In addition, a trajectory 8 of the fingertip position R detected by tracking the fingertip position R is schematically shown using a thick black line. Information on the trajectory 8 of the fingertip position R is range specification information of the user 1 .

[0253] In addition, the AR display unit 35 displays a frame line or the like at the position that the user 1 tracks with the fingertip through AR. That is, the trajectory 8 of the fingertip position R of the user 1 is displayed on the AR space. Therefore, for example, Fig.18 As shown, it becomes possible for the user 1 to visually see a state in which the trajectory of his / her own fingertip (finger 5) is displayed in a superimposed manner on the actual object 3. As a result, it becomes possible to easily perform designation of the photographing area 6, and usability is improved.

[0254] Return to Fig.17 , it is determined whether the manual range designation of user 1 is completed (step 205). For example, it is determined whether the range input by user 1 (the track 8 of the fingertip position R) is a closed range. Alternatively, it is determined whether the fingertip (finger 5) of user 1 is separated from the surface of the actual object 3. In addition, the method for determining the completion of range designation is not limited. For example, the operation of designating the range can be terminated based on the gesture of user 1 or other input operation.

[0255] If it is determined that the manual range designation is not completed ("No" in step 205), step 204 is executed, and the tracking of the fingertip position R and the like is continued.

[0256] If it is determined that the manual range designation is completed (Yes in step 205), the region detection unit 34 detects the range designated by the user 1 as the photographing region 6. That is, it can also be said that the trajectory 8 of the fingertip position R of the user 1 is set in the photographing region 6.

[0257] Therefore, in the area manual designation mode, the area detection unit 34 detects the photographing area 6 based on the trajectory 8 of the fingertip position R associated with the movement of the fingertip position R. Therefore, it becomes possible to manually designate the photographing area 6 and photograph an arbitrary area in the actual space. As a result, for example, it becomes possible to easily provide a virtual experience with a high degree of freedom, for example.

[0258] When the range designation is completed and the photographing area 6 is detected, a process of accepting manual correction of the photographing area 6 is performed (step 206). When the photographing area 6 is corrected, a partial image 43 in which the photographing area 6 is clearly photographed is appropriately extracted from the photographed image 40, and a virtual image 4 of the actual object 3 is generated based on the partial image 43 (step 207). The generated virtual image 4 is superimposed on the actual object 3, and is appropriately displayed corresponding to the gesture of the user 1, etc.

[0259] Note that the method of generating and displaying the virtual image 4 based on the manually specified photographing area 6 is not limited, and for example, refer to Figures 10 to 16 The described method is applicable. That is, the description about the above-mentioned automatically detected photographing area 6 can be appropriately replaced with the description about the manually specified photographing area 6.

[0260] Note that each of the area automatic detection mode and the area manual designation mode may be executed separately, or they may be appropriately switched and executed in parallel. For example, if the gesture of the user 1 is a gesture for designating an area, the area manual designation mode is executed, and if it is another gesture such as tapping the actual object 3, the area automatic detection mode is executed. For example, such a configuration may be adopted.

[0261] As described above, in the controller 30 according to the present embodiment, a contact action as a series of operations when the user contacts the actual object 3 is detected, and a shooting area 6 including the actual object 3 corresponding to the contact action is detected. From the shot image 40 shot by the actual space where the actual object 3 exists, a partial image 43 corresponding to the shooting area 6 is extracted, and a virtual image 4 of the actual object 3 is generated. Then, display control of the virtual image 4 is performed corresponding to the contact action of the user 1. This makes it possible to easily display the virtual image 4 in which the actual object 3 is shot, and seamlessly connect the actual space and the virtual space.

[0262] As a method of photographing the real world, for example, a method of automatically photographing the real world in response to a predetermined input operation can be conceived. For example, this method requires an action of specifying a range to be photographed, and the photographing process may be troublesome. In addition, since the photographing is automatically performed corresponding to the timing of performing the input operation, there may be a case where, for example, an obstructing object or the like is included in the photographing range. In this case, it is necessary to re-photograph the image, etc., which may interfere with the user's experience, etc.

[0263] In the present embodiment, the photographing area 6 is detected corresponding to the contact action of the user 1 with respect to the real object 3. Therefore, for example, when the user 1 contacts the real object 3, the photographing area 6 for photographing the real object 3 is automatically detected.

[0264] That is, even in a case where the user 1 does not explicitly set the shooting area 6, etc., it is possible to easily generate a virtual image 4 that shoots a desired real object 3, etc. As a result, the user 1 can easily bring an appropriate shot image (virtual image 4) into the virtual space without inputting the shooting area 6. As a result, it becomes possible to seamlessly connect the real space and the virtual space.

[0265] Furthermore, in the present embodiment, a partial image corresponding to the photographing area 6 is extracted from one or more photographed images 40 that photograph the actual space, and a virtual image 4 is generated. Therefore, for example, it becomes possible to acquire a partial image that is not masked in time, and it becomes possible to generate a clear virtual image 4 of the actual object 3 that is not masked, etc. As a result, it becomes possible to appropriately generate a desired virtual image 4 through a one-time photographing process, and to sufficiently avoid the occurrence of re-shooting, etc.

[0266] In addition, the generated virtual image 4 is superimposed and displayed on the actual object 3 in accordance with the contact action of the user 1. Therefore, in the HMD 100, when a contact action (interaction) occurs, a high-precision virtual image 4 generated based on the image just captured before is presented. The display of the virtual image 4 is appropriately controlled in accordance with the type of contact action, etc. This makes it possible to naturally bring the actual object 3 of the real world into the AR space, etc. As a result, the movement of the object from the real world (real space) to the virtual world (virtual space) becomes easy, and it becomes possible to achieve a seamless connection between the real world and the virtual world.

[0267] <Other Implementation Methods>

[0268] The present technology is not limited to the above-described embodiments, and various other embodiments can be implemented.

[0269] In reference Figure 4 and Fig.17In the described process, after the pre-contact state predicting contact between the user 1 and the actual object 3 is detected, the photographing process (step 103 and step 203) is started by the outward camera 14 using the setting for photographing. The timing of executing the photographing process is not limited.

[0270] For example, the photographing process may be performed in a state where the pre-contact state is not detected. For example, the photographing process may be performed in which objects that may be contacted around the user 1 are sequentially photographed in preparation for contact.

[0271] In addition, in the case where the actual object 3 that the user 1 attempts to contact cannot be specified, the actual object 3 that the user 1 may contact can be photographed in a presumed manner. For example, the user 1 wearing the HMD 100 points the line of sight in various directions, and various actual objects 3 around the user 1 can be photographed. For example, when the actual object 3 existing around the user 1 is included in the photographing range of the outward camera 14, the photographing process for photographing is performed in a presumed manner.

[0272] This makes it possible to configure a library or the like in which the actual objects 3 around the user 1 are photographed in the photographed image database 21. As a result, for example, even in a state where it is difficult to immediately photograph the object of the contact action of the user 1, it becomes possible to appropriately generate the virtual image 4 of the actual object 3 contacted by the user 1. Alternatively, it is also possible to perform the photographing process at an arbitrary time before the virtual image 4 is generated.

[0273] When photography fails, for example, photographed object data or the like on a cloud connectable by the HMD 100 via the communication unit 18 or the like may be searched. This makes it possible to generate a virtual image 4 even when an appropriate photographed image 40 is not included in the photographed image database 21 or the like.

[0274] exist Fig.13 In the embodiment, the user 1 grabs the stereoscopic real object 3 to generate a three-dimensional image (virtual image 4) representing the three-dimensional shape of the real object 3. For example, the shooting method may be switched to either 2D shooting or 3D shooting corresponding to the type of gesture. For example, when the user 1 performs a gesture for pinching the real object 3, 2D shooting is performed, and when the user 1 performs a gesture for grabbing the real object 3, 3D shooting is performed. For example, such processing may be performed.

[0275] In the above embodiment, the transmissive HMD 100 on which the transmissive display is mounted is used. For example, the present technology can be applied to a case where an immersive HMD that covers the visual field of the user 1 is used.

[0276] Fig.19200 is a perspective view schematically showing the appearance of an HMD according to another embodiment. The HMD 200 includes a mounting portion 210 of a body portion 220 that is worn on the head of the user 1 and located in front of both eyes of the user 1. The HMD 200 is an immersive head mounted display configured to cover the field of view of the user 1.

[0277] The main body section 220 includes a display (not shown) arranged to face the left and right eyes of the user 1. An image for the left eye and an image for the right eye are displayed on the display, which allows the user 1 to visually recognize a virtual space.

[0278] In addition, on the outside of the main body part 220, an outward camera 221 is installed. By displaying the image captured by the outward camera 221 on the internal display, the user 1 can visually recognize the video of the real world. On the display, various virtual images 4 are superimposed and displayed on the image captured by the outward camera. As a result, it is possible to provide a virtual experience using augmented reality (AR).

[0279] For example, using the reference Figure 3 The controller 30 described above is used to perform the contact action of the user 1 relative to the actual object 3, the detection of the shooting area 6, the display control of the virtual image 4 on the display, etc. Therefore, it becomes easy to generate a virtual image 4 of the actual object 3 that the user 1 contacts and display the virtual image 4 in the virtual space, whereby the actual space and the virtual space can be seamlessly connected.

[0280] Fig. 20 is a perspective view schematically showing an appearance of a mobile terminal 300 according to another embodiment. Fig. 20 The left and right sides of the mobile terminal 300 schematically show the front side and the rear side opposite to the front side of the mobile terminal 300, respectively, which provide a display surface 310. On the front side of the mobile terminal 300, an inward camera 320 is installed. On the back side, an outward camera 330 is installed.

[0281] For example, on the display surface 310 of the mobile terminal 300, an image of the real space captured by the outward camera 330 is displayed. In addition, various virtual images 4 and the like are superimposed and displayed relative to the image in the real space on the display surface 310. This allows the user 1 to visually recognize the AR space that expands the real space.

[0282] For example, using the reference Figure 3 The controller 20 described above, etc., can capture the actual object 3 corresponding to the contact action of the user 1 from the image captured by the outward camera 330. This makes it easy to bring the actual object 3 into the AR space. As described above, the present technology is also applicable to the case of using a mobile terminal 300, etc. Alternatively, a tablet terminal, a notebook PC, etc. can be used.

[0283] In addition, the present technology can also be applied to virtual reality (VR) space. For example, in the real space where the user 1 actually moves in the VR space, the real object 3 that the user 1 touches is photographed. This makes it easy to bring objects in the real space into the VR space. As a result, it becomes possible to exchange clones (virtual images 4) of the real object 3 between users experiencing the VR space, thereby activating communication.

[0284] In the above description, a case where the information processing method according to the present technology is executed by a controller installed on an HMD, etc. is described. However, the information processing method and program according to the present technology may be executed by other computers that can communicate by a controller installed on an HMD, etc. via a network, etc. In addition, the controllers installed on the HMD, etc. and other computers may be interlocked to construct a virtual space display system according to the present technology.

[0285] In other words, the information processing method and program according to the present technology can be executed not only in a computer system configured by a single computer, but also in a computer system in which a plurality of computers operate in conjunction with each other. Note that in the present disclosure, a system refers to a group of components (devices, modules (components), etc.), and it is not important whether all components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected to each other via a network and a single device having a plurality of modules housed in a single housing are both the system.

[0286] Execution of the information processing method and program according to the present technology by a computer system includes, for example, two cases in which detection of a user's contact action, detection of a target area including an actual object, generation of a virtual image, display control of a virtual image, etc. are performed by a single computer, and in which each process is performed by a different computer. In addition, execution of each process by a predetermined computer includes causing other computers to execute some or all of these processes and obtain their results.

[0287] That is, the information processing method and program according to the present technology can be applied to a configuration of cloud computing in which one function is shared and processed together among a plurality of devices via a network.

[0288] In the present disclosure, “same”, “equal”, “perpendicular”, etc. are concepts including “substantially the same”, “substantially equal”, “substantially perpendicular”, etc. For example, reference is also included to a state where “exactly the same”, “completely equal”, “completely perpendicular”, etc. are included within a predetermined range (e.g., within a range of ±10%).

[0289] At least two features of the present technology described above may also be combined. In other words, the various features described in the various embodiments may be combined arbitrarily, regardless of the embodiments. In addition, the various effects described above are not restrictive, but merely illustrative, and other effects may be provided.

[0290] The present technology may also have the following structures.

[0291] (1) An information processing device comprising:

[0292] an acquisition unit that acquires one or more captured images obtained by capturing the actual space;

[0293] a motion detection unit that detects a contact motion, the contact motion being a series of motions when a user contacts a real object in the real space;

[0294] an area detection unit, which detects a target area including the actual object according to the detected contact action; and

[0295] A display control unit generates a virtual image of the actual object by extracting a partial image corresponding to the target area from the one or more captured images, and controls display of the virtual image according to the contact action.

[0296] (2) The information processing device according to (1), wherein:

[0297] The display control unit generates a virtual image representing the actual object that is not obscured by the obstructing object.

[0298] (3) The information processing device according to (2), wherein:

[0299] The display control unit generates the partial image from a photographed image that does not include the shielding object in the target area among the one or more photographed images.

[0300] (4) The information processing device according to any one of (1) to (3), wherein:

[0301] The display control unit superimposes and displays the virtual image on the real object and displays it.

[0302] (5) The information processing device according to any one of (1) to (4), wherein:

[0303] The acquisition unit acquires the one or more photographed images from at least one of a photographing device that photographs the real space and a database that stores an output of the photographing device.

[0304] (6) The information processing device according to (5), wherein:

[0305] The contact action includes an action of bringing the user's hand closer to the actual object,

[0306] The action detection unit determines whether the state of the contact action is a pre-contact state in which contact of the user's hand with the actual object is predicted, and

[0307] If the state of the contact action is determined to be the pre-contact state, the acquisition unit acquires the one or more photographed images by controlling the photographing device.

[0308] (7) The information processing device according to (6), wherein:

[0309] If the state of the contact action is determined to be the pre-contact state, the acquisition unit increases a photographing resolution of the photographing device.

[0310] (8) The information processing device according to any one of (1) to (7), wherein:

[0311] The motion detection unit detects a contact position between the real object and the user's hand, and

[0312] The area detection unit detects the target area based on the detected contact position.

[0313] (9) The information processing device according to (8), wherein:

[0314] The region detection unit detects a boundary of the real object including the contact position as the target region.

[0315] (10) The information processing device according to (9), further comprising:

[0316] A sight line detection unit detects the sight line direction of the user, wherein:

[0317] The area detection unit detects a boundary of the real object based on the line of sight direction of the user.

[0318] (11) The information processing device according to (10), wherein:

[0319] The sight line detection unit detects a gaze position based on the sight line direction of the user, and

[0320] The region detection unit detects a boundary of the real object including the contact position and the gaze position as the target region.

[0321] (12) The information processing device according to any one of (1) to (11), wherein:

[0322] The area detection unit detects a boundary of the real object based on at least one of a shadow, a size, and a shape of the real object.

[0323] (13) The information processing device according to any one of (1) to (12), wherein:

[0324] The motion detection unit detects the fingertip position of the user's hand, and

[0325] The region detection unit detects the target region based on a trajectory of the fingertip position accompanying movement of the fingertip position.

[0326] (14) The information processing device according to any one of (1) to (13), wherein:

[0327] The display control unit superimposes and displays an area image representing the target area on the actual object.

[0328] (15) The information processing device according to (14), wherein:

[0329] displaying the region image so that at least one of shape, size, and position can be edited, and

[0330] The area detection unit changes the target area based on the edited area image.

[0331] (16) The information processing device according to any one of (1) to (15), wherein:

[0332] The motion detection unit detects a contact position between the real object and the user's hand, and

[0333] The display control unit controls display of the virtual image according to the detected contact position.

[0334] (17) The information processing device according to any one of (1) to (16), wherein:

[0335] The motion detection unit detects a gesture of a user's hand contacting the real object, and

[0336] The display control unit controls display of the virtual image according to the detected hand gesture of the user.

[0337] (18) The information processing device according to any one of (1) to (17), wherein:

[0338] The virtual image is at least one of a two-dimensional image and a three-dimensional image of the actual object.

[0339] (19) An information processing method, comprising executing, by a computer system:

[0340] Acquiring one or more captured images obtained by capturing an actual space;

[0341] detecting a contact action, the contact action being a series of actions when a user contacts a real object in the real space;

[0342] detecting a target area of ​​the actual object corresponding to the detected contact action; and

[0343] A virtual image of the real object is generated by extracting a partial image corresponding to the target area from the one or more captured images, and display of the virtual image is controlled according to the contact action.

[0344] (20) A program for causing a computer system to execute the following steps:

[0345] A step of acquiring one or more photographed images obtained by photographing the actual space;

[0346] a step of detecting a contact action, wherein the contact action is a series of actions when a user contacts a real object in the real space;

[0347] A step of detecting a target area including the actual object according to the detected contact action; and

[0348] A step of generating a virtual image of the actual object by extracting a partial image corresponding to the target area from the one or more captured images, and controlling display of the virtual image according to the contact action.

[0349] Reference Symbols List

[0350] 1 User

[0351] 3 Actual objects

[0352] 4 Virtual Image

[0353] 5 fingers

[0354] 6 Shooting Area

[0355] 7 Boundaries

[0356] 8 Trajectory

[0357] 12 Transmissive Display

[0358] 14 Outward-facing camera

[0359] 21 Shot Image Database

[0360] 30 Controller

[0361] 31 Image acquisition unit

[0362] 32 Contact detection unit

[0363] 33. Eye-gaze detection unit

[0364] 34 Area Detection Unit

[0365] 35 AR display unit

[0366] 40 Capture images

[0367] 42 Area Image

[0368] 43, 43a, 43b Partial images

[0369] 100, 200 HMD

Claims

1. An information processing device, include: an acquisition unit that acquires one or more captured images obtained by capturing the actual space; a motion detection unit that detects a contact motion, the contact motion being a series of motions when a user contacts a real object in the real space; an area detection unit, which detects a target area including the actual object according to the detected contact action; as well as a display control unit that generates a virtual image of the actual object by extracting a partial image corresponding to the target area from the one or more captured images, and controls display of the virtual image according to the contact action, The acquisition unit acquires the one or more captured images from at least one of a shooting device that shoots the actual space and a database that stores outputs of the shooting device. The contact action includes an action of bringing the user's hand closer to the actual object, The action detection unit determines whether the state of the contact action is a pre-contact state in which the contact of the user's hand with the actual object is predicted, If the state of the contact action is determined to be the pre-contact state, the acquisition unit acquires the one or more captured images by controlling the capturing device, The motion detection unit detects a contact position between the actual object and a user's hand, The region detection unit detects a boundary of the actual object including the contact position as the target region, and The display control unit superimposes and displays an area image representing the target area on the actual object.

2. The information processing device according to claim 1, in, The display control unit generates a virtual image representing the actual object that is not obscured by the obstructing object.

3. The information processing device according to claim 2, in, The display control unit generates the partial image from a captured image that does not include the shielding object in the target area, among the one or more captured images.

4. The information processing device according to claim 1, in, The display control unit superimposes and displays the virtual image on the real object.

5. The information processing device according to claim 1, in, If the state of the contact action is determined to be the pre-contact state, the acquisition unit increases a photographing resolution of the photographing device.

6. The information processing device according to claim 1, further comprising: include: A sight line detection unit detects the sight line direction of the user, wherein: The area detection unit detects a boundary of the real object based on a sight line direction of a user.

7. The information processing device according to claim 6, in, The sight line detection unit detects a gaze position based on a sight line direction of the user, and The region detection unit detects a boundary of the real object including the contact position and the gaze position as the target region.

8. The information processing device according to claim 1, in, The area detection unit detects a boundary of the real object based on at least one of a shadow, a size, and a shape of the real object.

9. The information processing device according to claim 1, in, The motion detection unit detects the fingertip position of the user's hand, and The region detection unit detects the target region based on a trajectory of the fingertip position accompanying movement of the fingertip position.

10. The information processing device according to claim 1, in, displaying the region image so that at least one of shape, size, and position can be edited, and The area detection unit changes the target area based on the edited area image.

11. The information processing device according to claim 1, in, The motion detection unit detects a contact position between the real object and a user's hand, and The display control unit controls display of the virtual image according to the detected contact position.

12. The information processing device according to claim 1, in, The motion detection unit detects a gesture of a user's hand contacting the real object, and The display control unit controls display of the virtual image according to the detected hand gesture of the user.

13. The information processing device according to claim 1, in, The virtual image is at least one of a two-dimensional image and a three-dimensional image of the actual object.

14. An information processing method, comprising executing, by a computer system: Acquiring one or more captured images obtained by capturing an actual space; detecting a contact action, the contact action being a series of actions when a user contacts a real object in the real space; detecting a target area including the actual object according to the detected contact action; and generating a virtual image of the actual object by extracting a partial image corresponding to the target area from the one or more captured images, and controlling display of the virtual image according to the contact action, in, Acquiring the one or more captured images includes: acquiring the one or more captured images from at least one of a capturing device capturing the actual space and a database storing output of the capturing device, The contact action includes an action of bringing the user's hand closer to the actual object, determining whether the state of the contact action is a pre-contact state in which contact of the user's hand with the actual object is predicted, If the state of the contact action is determined to be the pre-contact state, acquiring the one or more captured images by controlling the capturing device, detecting a contact position between the actual object and a user's hand, A boundary of the actual object including the contact position is detected as the target area, and An area image representing the target area is superimposed and displayed on the actual object.

15. A computer readable medium having stored thereon a program, the program causing a computer system to perform the following steps: A step of acquiring one or more photographed images obtained by photographing the actual space; a step of detecting a contact action, wherein the contact action is a series of actions when a user contacts a real object in the real space; A step of detecting a target area including the actual object according to the detected contact action; and generating a virtual image of the actual object by extracting a partial image corresponding to the target area from the one or more captured images and controlling display of the virtual image according to the contact action, in, Acquiring the one or more captured images includes: acquiring the one or more captured images from at least one of a capturing device capturing the actual space and a database storing output of the capturing device, The contact action includes an action of bringing the user's hand closer to the actual object, determining whether the state of the contact action is a pre-contact state in which contact of the user's hand with the actual object is predicted, If the state of the contact action is determined to be the pre-contact state, acquiring the one or more captured images by controlling the capturing device, detecting a contact position between the actual object and a user's hand, A boundary of the actual object including the contact position is detected as the target area, and An area image representing the target area is superimposed and displayed on the actual object.