Information processing device, information processing method, computer program, and augmented reality system

The AR system addresses the challenge of storing and restoring AR applications by using sensor-based finger detection, allowing seamless interaction and maintaining application state through gestures, improving user experience and accuracy in AR environments.

JP7589696B2Active Publication Date: 2024-11-26SONY GROUP CORP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021570663
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-01-17
Filing Date
2020-11-20
Publication Date
2024-11-26
Estimated Expiration
2040-11-20

AI Technical Summary

Technical Problem

Existing augmented reality (AR) systems lack efficient methods for users to store and restore the execution state of AR applications without closing them, leading to inconvenience and loss of information, and struggle with accurately detecting hand and finger operations due to occlusion and image recognition limitations.

Method used

An AR system that uses sensors to detect the position and orientation of a user's fingers, allowing users to store and restore AR applications by performing specific gestures on their body, and controls virtual objects based on these detections, enabling seamless interaction and operation of AR applications without direct hand restriction.

Benefits of technology

Enables users to store and restore AR applications effortlessly, maintaining their execution state, and accurately detect hand and finger operations, enhancing user interaction and convenience in AR environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589696000001
    Figure 0007589696000001
  • Figure 0007589696000002
    Figure 0007589696000002
  • Figure 0007589696000003
    Figure 0007589696000003
Patent Text Reader

Abstract

Provided is an information processing apparatus for processing information about augmented reality. The information processing apparatus is provided with: an acquisition unit that acquires a shape or a positional orientation of a first portion on the body of a user; and a control unit that controls a virtual object displayed in a display device so as to be in real space, on the basis of the shape or the positional orientation of the first portion. When recognizing that the first portion has performed a first operation on a second portion on the body of the user on the basis of the shape or the positional orientation of the first portion acquired by the acquisition unit, the control unit stores the execution state of the virtual object in association with the second portion and stops display of the virtual object by the display device.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing device and an information processing method for processing information relating to augmented reality, a computer program, and an augmented reality system. [Background technology]

[0002] As technologies for realizing a realistic experience, virtual reality (VR), augmented reality (AR), and mixed reality (MR) are becoming widespread. VR is a technology that allows a user to perceive a virtual space as reality. AR is a technology that extends the real space seen by the user by adding, emphasizing, attenuating, or deleting information in the real environment surrounding the user. MR is a technology that displays virtual information corresponding to the real space to mix reality and virtuality. AR and MR are realized, for example, by using a see-through head-mounted display (hereinafter also referred to as "AR glasses") (see, for example, Patent Document 1). According to the AR technology, it is possible to superimpose digital information on the scenery of the real space observed by the user through the AR glasses, to emphasize or attenuate a specific real object, or to delete a specific real object so that it appears as if it does not exist. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-93050 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an information processing device, an information processing method, a computer program, and an augmented reality system that process information related to augmented reality. [Means for solving the problem]

[0005] A first aspect of the present disclosure is an acquisition unit that acquires a position, posture, or shape of a first part on a user's body; a control unit that controls a virtual object to be displayed in a real space on a display device based on the position, orientation, or shape of the first part; The information processing device includes:

[0006] When the control unit recognizes that the first part has performed a first operation on a second part on the user's body based on the position, posture or shape of the first part acquired by the acquisition unit, the control unit associates the execution state of the virtual object with the second part and stores it, and stops displaying the virtual object on the display device.

[0007] In addition, when the control unit recognizes that the first part has performed a second operation on a second part on the user's body based on the position, posture or shape of the first part acquired by the acquisition unit, the control unit restores the display of the virtual object based on the execution state associated with the second part and stored.

[0008] The second aspect of the present disclosure is An acquisition step of acquiring a position, posture, or shape of a first part on a body of a user; a control step of controlling a virtual object to be displayed in a real space on a display device based on the position, orientation or shape of the first part; The information processing method includes the steps of:

[0009] Moreover, a third aspect of the present disclosure is An acquisition unit that acquires a position, posture, or shape of a first part on a body of a user; a control unit that controls a virtual object to be displayed in a real space on a display device based on the position, posture, or shape of the first part; It is a computer program written in a computer-readable form so as to cause a computer to function as a

[0010] The computer program according to the third aspect of the present disclosure defines a computer program written in a computer-readable format to realize a predetermined process on a computer. In other words, by installing the computer program according to the third aspect of the present disclosure on a computer, a cooperative action is exerted on the computer, and the same action and effect as the information processing device according to the first aspect of the present disclosure can be obtained.

[0011] Moreover, a fourth aspect of the present disclosure is A display device that displays a virtual object in a real space; an acquisition unit that acquires a position, posture, or shape of a first part on a user's body; a control unit that controls display of a virtual object based on a position, orientation, or shape of the first part; An augmented reality system comprising:

[0012] However, the term "system" used here refers to a logical collection of multiple devices (or functional modules that realize a specific function), and it does not matter whether each device or functional module is contained within a single housing. Effect of the Invention

[0013] According to the present disclosure, it is possible to provide an information processing device, an information processing method, a computer program, and an augmented reality system that control the operation of a virtual object displayed to a user based on the position and orientation of the user's fingers.

[0014] Note that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited thereto. Furthermore, the present disclosure may have additional effects in addition to the effects described above.

[0015] Further objects, features and advantages of the present disclosure will become apparent from the following detailed description based on the embodiments and the accompanying drawings. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing an example of the functional configuration of an AR system 100. [Diagram 2] FIG. 2 is a diagram showing a state in which the AR glasses are worn on a user's head. [Diagram 3] FIG. 3 is a diagram showing an example of the configuration of the AR system 300. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of an AR system 400. [Diagram 5] FIG. 5 is a diagram showing an example of the controller 400 being worn on the user's hand. [Figure 6] FIG. 6 is a diagram showing an example of the functional configuration of the control unit 140. As shown in FIG. [Figure 7] FIG. 7 is a diagram showing how virtual objects are arranged around a user wearing the AR glasses on his or her head. [Figure 8] FIG. 8 is a diagram for explaining a mechanism by which the AR glasses display a virtual object so as to follow the movement of the user's head. [Figure 9] FIG. 9 is a diagram for explaining a mechanism for detecting a UI settable area by calibration. [Figure 10] FIG. 10 is a diagram for explaining a mechanism for detecting a UI settable area by calibration. [Figure 11] FIG. 11 is a diagram showing the storing and restoring operations of the UI of the AR application. [Figure 12] FIG. 12 is a diagram showing the storing and restoring operations of the UI of the AR application. [Figure 13] FIG. 13 is a diagram showing the storing and restoring operations of the UI of the AR application. [Figure 14] FIG. 14 is a diagram showing the storing and restoring operations of the UI of the AR application. [Figure 15] FIG. 15 is a flowchart showing an operation procedure for storing and restoring an AR application. [Figure 16]FIG. 16 is a diagram showing an example in which an aura of an area in which a UI can be installed is displayed on the user's body. [Figure 17] FIG. 17 is a diagram showing an example in which virtual objects of an operation UI for operating an AR application placed far away from the user's hand are arranged. [Figure 18] FIG. 18 is a diagram showing an example in which a virtual object of an operation UI is displayed on the palm of a user's hand. [Figure 19] FIG. 19 is a diagram showing an example in which a virtual object of an operation UI is displayed on the palm of a user's hand. [Figure 20] FIG. 20 is a diagram showing an example of assigning the functions of an AR application to the back of a user's hand. [Figure 21] FIG. 21 is a diagram showing an example in which an operation UI for inputting coordinates is placed on the palm of the hand. [Figure 22] FIG. 22 is a diagram showing an example in which an operation UI for inputting coordinates is placed on the palm of the hand. [Diagram 23] FIG. 23 is a diagram showing an example in which an operation UI for inputting coordinates is placed on the palm of the hand. [Figure 24] FIG. 24 is a diagram showing an example in which an operation UI for inputting coordinates is placed on the palm of the hand. [Diagram 25] FIG. 25 is a diagram showing an example in which an operation UI for inputting coordinates is placed on the palm of the hand. [Figure 26] FIG. 26 is a diagram showing an example in which an operation UI for inputting coordinates is placed on the palm of the hand. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0018] In operating systems (OS) such as Windows and Linux (registered trademark), a user interface (UI) is used that minimizes the windows of applications that are in the way when placed on the screen and stores them, for example, in a task bar at the bottom edge of the screen. When multiple applications are running at the same time, the screen can be made neat by minimizing the windows of the intrusive applications and retracting them to the periphery of the screen.

[0019] On the other hand, when an application such as a video player or a 3D object viewer is launched on the AR glasses (hereinafter, applications launched on the AR glasses are also referred to as "AR apps"), the running AR app is superimposed on the user's field of view, but there is no UI for minimizing the AR app, and the user has no choice but to close the AR app in order to hide it from view. When an AR app is closed, the execution state of the AR app is discarded and the initial state is restored, which is inconvenient. For example, when an AR app for video playback is closed and then launched again, the playback position of the content and information on the content being played are discarded, making it difficult to restore the execution state to the state immediately before the app was closed.

[0020] In addition, in an OS such as Windows or Linux (registered trademark), an operation is performed on a running application via an input device such as a keyboard, a mouse, or a touch panel. A controller for a user to operate an AR app may be used with the AR glasses. However, it is preferable that a user can look around the real space through the AR glasses and use the AR glasses while performing daily activities in the real space, such as walking and grasping objects (including real objects and virtual objects). For this reason, it is preferable that the user's fingers are not restricted by grasping a controller, and can be used freely. In addition, it is preferable that the user can store and restore the AR app, and furthermore, operate the AR app after it has been restored, with his or her fingers. Furthermore, it is preferable that the user can store and restore the AR app with his or her fingers without looking away, while directing his or her gaze toward the display position of the AR app in front of him or her, rather than looking at his or her hands.

[0021] Some AR glasses are equipped with an outward-facing camera to observe the real space. Based on image recognition of the image captured by the outward-facing camera, the user's hand and finger operations and other gestures can be input. However, it is not possible to input the user's hand and finger operations that are performed at a position outside the field of view of the outward-facing camera located at the user's head position. In addition, if occlusion occurs, such as the user's hand being hidden behind another object, the outward-facing camera cannot capture the hand and finger operations. In addition, there are cases where it is not possible to accurately obtain the presence or absence of contact between the hand or fingertip and an object, or the shape of the hand or fingertip that is in contact with the object, from the image recognition of the image captured by the outward-facing camera.

[0022] Therefore, in the present disclosure, information regarding the shape of the user's fingers is obtained using a sensor unit that detects the position and orientation of the user's fingers, and the AR application is stored or restored based on the obtained shape of the fingers.

[0023] A. System Configuration 1 shows an example of a functional configuration of an AR system 100 to which the present disclosure is applied. The illustrated AR system 100 includes a first sensor unit 110 that detects the position of the hand of a user wearing AR glasses and the shape of the user's fingers, a second sensor unit 120 mounted on the AR glasses, a display unit 131 that displays virtual objects such as an AR application on the AR glasses, and a control unit 140 that comprehensively controls the operation of the entire AR system 100. The first sensor unit 110 includes a gyro sensor 111, an acceleration sensor 112, and an orientation sensor 113. The second sensor unit 120 is mounted on the AR glasses and includes an outward camera 121, an inward camera 122, a microphone 123, a gyro sensor 124, an acceleration sensor 125, and an orientation sensor 126.

[0024] The AR system 100 may further include a speaker 132 that outputs audio signals such as sounds related to the virtual object, a vibration presentation unit 133 that provides feedback by vibration presentation to the back of the user's hand or other body parts, and a communication unit 134 that enables the AR system 100 to communicate with the outside. The control unit 140 may also be equipped with a large-scale storage unit 150 formed of a solid state drive (SSD) or the like.

[0025] The AR glasses are generally glasses-type or goggle-type devices that are worn by the user on the head, and can superimpose digital information on the field of view of both or one of the user's eyes, emphasize or attenuate a specific real object, or delete a specific real object to make it appear as if it does not exist. FIG. 2 shows the state in which the AR glasses are worn on the user's head. The illustrated AR glasses are provided with a left-eye display unit 131 and a right-eye display unit 131 in front of the user's left and right eyes, respectively. The display unit 131 is transparent or semi-transparent, and can superimpose a virtual object on a landscape in real space, emphasize or attenuate a specific real object, or delete a specific real object to make it appear as if it does not exist. The left and right display units 131 may be driven to display parallax images, i.e., virtual objects, in 3D. An outward-facing camera facing the user's line of sight is disposed approximately at the center of the AR glasses.

[0026] The AR system 100 can be configured with two devices, for example, AR glasses worn by a user on the head and a controller worn by the user on the hand. Fig. 3 shows an example of the configuration of the AR system 300 including the AR glasses 301 and the controller 302. The AR glasses 301 include a control unit 140, a storage unit 150, a second sensor unit 120, a display unit 131, a speaker 132, and a communication unit 134. The controller 302 includes a first sensor unit 110 and a vibration presentation unit 133.

[0027] As another configuration example, the AR system 100 is composed of three devices: AR glasses worn by the user on the head, a controller worn by the user on the hand, and an information terminal such as a smartphone or a tablet. FIG. 4 shows a configuration example of an AR system 400 composed of AR glasses 401, a controller 402, and an information terminal 403. The AR glasses 401 include a display unit 131, a speaker 132, and a second sensor unit 120. The controller 402 includes a first sensor unit 110 and a vibration presentation unit 133. The information terminal 403 includes a control unit 140, a storage unit 150, and a communication unit 134.

[0028] Note that the specific device configuration of the AR system 100 is not limited to that shown in Fig. 3 and Fig. 4. Moreover, the AR system 100 may further include components other than those shown in Fig. 1.

[0029] Each component of the AR system 100 will now be described.

[0030] As shown in Figs. 3 and 4, the first sensor unit 110 and the vibration presentation unit 133 are configured as a controller worn on the hand of a user. The first sensor unit 110 includes a gyro sensor 111, an acceleration sensor 112, and an orientation sensor 113. The first sensor unit 110 may be an IMU (Inertial Measurement Unit) including a gyro sensor, an acceleration sensor, and an orientation sensor. The vibration presentation unit 133 is configured by arranging electromagnetic or piezoelectric vibrators in an array. A sensor signal from the first sensor unit 110 is transferred to the control unit 140.

[0031] 5 shows an example of a controller 500 including a first sensor unit 110 and a vibration presentation unit 133 worn on a user's hand. In the example shown in FIG. 5, IMUs 501, 502, and 503 are attached to three locations, the thumb, and the proximal and middle phalanges of the index finger, by bands 511, 512, and 513, respectively. The vibration presentation unit 133 is attached to the back of the hand. This makes it possible to measure the posture of the thumb and the postures of the proximal and middle phalanges of the index finger (or the angle of the second joint of the index finger). The vibration presentation unit 133 may be fixed to the back of the hand by a band (not shown) or an adhesive pad.

[0032] However, Fig. 5 shows an example of the first sensor unit 110, and other IMUs may be attached to other locations on the thumb and index finger, or IMUs may be attached to fingers other than the thumb and index finger. Also, the method of fixing the IMU to each finger is not limited to a band. Also, Fig. 5 shows an example in which the first sensor unit 110 and the vibration presentation unit 133 are attached to the right hand, but they may be attached to the left hand instead of the right hand, or may be attached to both hands.

[0033] Also, it is assumed that there is a wired or wireless transmission path for transmitting a sensor signal by the first sensor unit 110 (IMUs 501, 502, and 503 in the example shown in FIG. 5) to the control unit 140 and receiving a drive signal for the vibration presentation unit 133 from the control unit 140. The control unit 140 can detect the position and posture of the fingers based on the sensor signal of the first sensor unit 110. As shown in FIG. 5, when IMUs 501, 502, and 503 are attached to three locations, namely the proximal and middle joints of the thumb and index finger, the control unit 140 can recognize the position and posture of the fingers (or the shape of the fingers) and the gesture of the fingers, such as the opening angle between the thumb and index finger, the angle of the second joint of the index finger, and the presence or absence of contact between the fingertips of the thumb and index finger, based on the detection signals of the IMUs 501, 502, and 503.

[0034] Referring again to FIG. 1, the description of each component of the AR system 100 will continue.

[0035] Second sensor unit 120 is mounted on the AR glasses, and includes outward facing camera 121, inward facing camera 122, microphone 123, gyro sensor 124, acceleration sensor 125, and orientation sensor 126.

[0036] The outward camera 121 is, for example, an RGB camera, and is installed so as to capture the outside of the AR glasses, that is, the front direction of the user wearing the AR glasses. The outward camera 121 can capture the operation of the user's fingers, but cannot capture the operation of the user's fingers when the user's fingers are hidden behind an obstacle, when the fingertips are hidden by the back of the hand, when the user puts the hand behind the body, etc. In addition, the outward camera 121 may further include either an IR camera consisting of an IR light emitter and an IR light receiver, or a TOF (Time Of Flight) camera. When an IR camera is used for the outward camera 121, a retroreflective material is attached to an object to be captured, such as the back of the hand, and the IR camera emits infrared light and receives infrared light reflected from the retroreflective material. The image signal captured by the outward camera 121 is transferred to the control unit 140.

[0037] Inward-facing camera 122 is, for example, an RGB camera, and is installed so as to capture the inside of the AR glasses, specifically the eyes of the user wearing the AR glasses. The user's line of sight direction can be detected based on the image captured by inward-facing camera 122. The image signal captured by inward-facing camera 122 is transferred to control unit 140.

[0038] The microphone 123 may be a microphone array consisting of a single sound pickup element or multiple sound pickup elements. The microphone 123 picks up the voice of a user wearing the AR glasses and the sounds around the user. The audio signal picked up by the microphone 123 is transferred to the control unit 140.

[0039] The gyro sensor 124, the acceleration sensor 125, and the orientation sensor 126 may be configured as an IMU. Sensor signals from the gyro sensor 124, the acceleration sensor 125, and the orientation sensor 126 are transferred to the control unit 140. The control unit 140 can detect the position and orientation of the head of the user wearing the AR glasses based on these sensor signals.

[0040] The display unit 131 is composed of a transmissive display (such as glasses lenses) that is placed in front of both eyes or one eye of a user wearing the AR glasses, and is used to display the virtual world. Specifically, the display unit 131 displays information (virtual objects) and enhances, attenuates, or deletes real objects to expand the real space seen by the user. The display unit 131 performs a display operation based on a control signal from the control unit 140. In addition, the mechanism for displaying virtual objects in a see-through manner on the display unit 131 is not particularly limited.

[0041] The speaker 132 is configured with a single sound-generating element or an array of multiple sound-generating elements, and is installed in, for example, AR glasses. For example, the speaker 132 outputs a sound related to a virtual object displayed on the display unit 131, but may also output other audio signals.

[0042] The communication unit 134 has a wireless communication function such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), etc. The communication unit 134 mainly performs a communication operation for realizing data exchange between the control unit 140 and an external system (not shown).

[0043] The control unit 140 is installed in the AR glasses, or is arranged in a device (such as a smartphone) separate from the AR glasses together with a storage unit 150 and a driving power source such as a battery. The control unit 140 executes various programs read from the storage unit 150 to perform various processes.

[0044] Fig. 6 shows a schematic diagram of an example of a functional configuration of the control unit 140. In the example shown, the control unit 140 includes an application execution unit 601, a head position / posture detection unit 602, an output control unit 603, a hand / finger position / posture detection unit 604, a hand / finger gesture detection unit 605, and a UI installation possible area detection unit 606. These functional modules are realized by the control unit 140 executing various programs read from the storage unit 150. However, Fig. 6 shows only the minimum functional modules necessary to realize the present disclosure, and the control unit 140 may further include other functional modules.

[0045] The application execution unit 601 executes application programs including an AR application under an execution environment provided by the OS. The application execution unit 601 may execute multiple application programs simultaneously in parallel. The AR application is, for example, an application such as a video playback or a 3D object viewer, and displays a virtual object superimposed in the field of view of a user wearing AR glasses (see FIG. 2), emphasizes or attenuates a specific real object, or deletes a specific real object to make it appear as if it does not exist. The application execution unit 601 also controls the display operation of the AR application (virtual object). The virtual object generated by the AR application is arranged all around the user. FIG. 7 shows a schematic diagram of a state in which multiple virtual objects 701, 702, 703, ... are arranged around the user wearing the AR glasses on his / her head 700. The application execution unit 601 arranges each of the virtual objects 701, 702, 703, ... around the user based on the position of the user's head or the position of the center of gravity of the body estimated based on the sensor information from the second sensor unit 120.

[0046] The head position and orientation detection unit 602 detects the position and orientation of the user's head based on the sensor signals of the gyro sensor 124, acceleration sensor 125, and orientation sensor 126 included in the second sensor unit 120 mounted on the AR glasses, and further recognizes the user's line of sight or field of view.

[0047] The output control unit 603 controls the output of the display unit 131, the speaker 132, and the vibration presentation unit 133 based on the execution result of an application program such as an AR app. For example, the output control unit 603 specifies the user's field of view based on the detection result of the head position and orientation detection unit 602, and controls the display operation of the virtual object by the display unit 131 so that the user can observe the virtual object arranged in the field of view through the AR glasses, that is, so that the virtual object follows the movement of the user's head.

[0048] A mechanism for displaying virtual objects so that the AR glasses follow the movement of the user's head will be described with reference to FIG. 8. In FIG. 8, the depth direction of the user's line of sight is z w axis, horizontal is y w axis, vertical direction is x w axis, the user's reference axis x w y w z w The origin of the roll θ is the user's viewpoint. z is the z of the user's head w Movement around the axis, tilt θ y is the y w Movement around axis, pan θ z is the x in the user's head w Based on the sensor signals of the gyro sensor 124, the acceleration sensor 125, and the orientation sensor 126, the head position and orientation detection unit 602 detects the movements of the user's head in each direction (roll, tilt, and pan) (θ z ,θ y ,θ z) and detects posture information consisting of translational movement of the head. Then, the output control unit 603 moves the display angle of view of the display unit 131 in the real space in which the virtual object is arranged (for example, see FIG. 7) so as to follow the posture of the user's head, and displays an image of the virtual object existing in the display angle of view on the display unit 131. Specifically, the display angle of view is moved so as to cancel the movement of the user's head by rotating the area 802-1 according to the roll component of the user's head movement, moving the area 802-2 according to the tilt component of the user's head movement, and moving the area 802-3 according to the pan component of the user's head movement. Therefore, the display unit 131 displays the virtual object arranged in the display angle of view moved in accordance with the position and posture of the user's head, so that the user can observe the real space on which the virtual object is superimposed through the AR glasses.

[0049] Referring again to FIG. 6, the functional configuration of the control unit 140 will be described.

[0050] The hand and finger position and orientation detection unit 604 detects the position and orientation of the hand and fingers of the user wearing the AR glasses based on the recognition result of the image captured by the outward camera 121 or the detection signal of the first sensor unit 110. The hand and finger gesture detection unit 605 detects the gesture of the fingers of the user wearing the AR glasses based on the recognition result of the image captured by the outward camera 121 or the detection signal of the first sensor unit 110. The hand and finger gesture referred to here includes the shape of the fingers, specifically the angle of the third joint and the second joint of the index finger, and the presence or absence of contact between the fingertips of the thumb and the index finger.

[0051] In this embodiment, the hand position / posture detection unit 604 and the hand gesture detection unit 605 detect the hand position and the hand gesture with high accuracy mainly by using the information on the position / posture from the first sensor unit 110 (gyro sensor 111, acceleration sensor 112, and orientation sensor 113) attached to the user's hand and the constraint conditions of the position / posture that the fingers can take. For example, when the user turns his / her hand to the back side of the body such as the back or buttocks, the hand position / posture cannot be detected by image recognition from the head, but the hand position / posture can be detected with high accuracy by using the sensor signal of the first sensor unit 110 attached to the hand. On the other hand, in the case of a method of detecting the hand position / posture and the hand gesture using the outward camera 121, there are cases where high accuracy detection is not possible due to occlusion or the like.

[0052] The UI installation area detection unit 606 detects an area on the user's body where the UI can be installed. The UI here specifically refers to a UI that is a minimized version of the AR application executed by the application execution unit 601. The UI installation area is, for example, the user's back, waist, or arm. Installing the UI means storing the UI in association with the location of the UI installation area while maintaining the execution state of the AR application. The virtual object of the stored AR application disappears from the user's field of vision.

[0053] The UI installable area detection unit 606 may detect one UI installable area, or may simultaneously detect multiple UI installable areas. When multiple UI installable areas are detected, a UI in which a different AR application is minimized can be stored in each UI installable area. In addition, the UIs of multiple AR applications may be stored in a single UI installable area in a redundant manner by using different hand gestures to instruct storage and restoration.

[0054] B.UI installation area Conventionally, the only way for a user to hide an AR app that is in the way from view is to close the AR app, and when the AR app is closed, the execution state of the AR app is discarded and the initial state is restored. In contrast, in this embodiment, the user can specify a part on the body with his / her fingers and store the UI with the AR app minimized in the specified part by a specific gesture of the fingers. After being stored, the virtual object of the AR app disappears from the user's view, but the minimized UI continues to hold the execution state of the AR app. After that, the user can restore the original AR app from the minimized UI to the execution state before it was stored by performing a specific gesture of the fingers to restore the UI to the part on the body where the UI is stored, and the virtual object of the AR app can appear in the original place in the user's view.

[0055] In order to realize such an AR application storage and restoration operation, the UI installation possible area detection unit 606 detects an area on the user's body where the UI can be installed. Furthermore, the application execution unit 601 identifies an AR application currently held by the user in the hand or an AR application designated by the user with the hand or finger based on the detection result of the hand and finger position and orientation detection unit 604. Then, when the application execution unit 601 recognizes based on the detection result of the hand and finger position and orientation detection unit 604 and the hand and finger gesture detection unit 605 that the user has performed a hand gesture instructing storage of an AR application or restoration of an AR application within the UI installation possible area detected by the UI installation possible area detection unit 606, the application execution unit 601 stores the UI of the AR application in a part of the body designated by the hand and finger gesture, or restores the AR application from the part of the body where the UI is stored.

[0056] The UI installable area will be described in more detail. The head position and orientation detection unit 602 can detect the position and orientation of the user's head with high accuracy based on detection signals from the gyro sensor 124, acceleration sensor 125, and orientation sensor 126 included in the second sensor unit 120 mounted on the AR glasses. Therefore, the UI installable area detection unit 606 detects a location obtained by adding a specific offset from the position of the user's head as a reference, or a part of the user's back, waist, etc., as the UI installable area.

[0057] In addition, since the first sensor unit 110 is also attached to the user's hand, it is possible to acquire the hand position from the gyro sensor 111, acceleration sensor 112, and orientation sensor 113 included in the first sensor unit 110 and roughly estimate the arm position from the hand position, so that the UI installation possible area detection unit 606 further detects the arm as a UI installation possible area. Of course, a tracking sensor (such as an IMU) may be attached to a place where the position is to be acquired so that the arm can be specified as a UI installation possible area.

[0058] In addition, even if a tracking sensor is not directly attached to the location where the position is to be obtained, the UI installation area detection unit 606 may detect other body parts that can be estimated using inverse kinematics calculation technology as UI installation areas based on position and orientation information of detectable body parts, such as the position and orientation of the hand detected from the sensor information of the first sensor unit 110 and the position and orientation of the head detected from the sensor information of the second sensor unit 120.

[0059] Furthermore, the UI installable area detection unit 606 may detect other body parts whose positions can be estimated using a machine learning model as the UI installable area based on information on the detectable body parts. The UI installable area detection unit 606 may use a machine learning model that has been deep-learned to estimate an optimal UI installable area based on sensor information from the first sensor unit 110 and the second sensor unit 120. For this machine learning model, a loss function based on, for example, the number of times the user's finger operation fails during use is defined, and deep learning is performed by backpropagation so as to minimize the loss function.

[0060] The UI installable area detection unit 606 may not only detect a part of the user's body that is a UI installable area, but also determine the size and shape of the UI installable area. For example, the size of an area of ​​a part that is difficult for a user to operate with his / her fingers accurately based on somatosensation may be expanded, or the shape of the area may be set to an area that is easy for a user to operate with his / her fingers accurately. The UI installable area detection unit 606 may use a machine learning model that is deep-learned to estimate the optimal size and shape of the UI installable area based on sensor information from the first sensor unit 110 and the second sensor unit 120. For this machine learning model, a loss function based on the number of times the user's finger operation fails during use is defined, and deep learning is performed by error backpropagation so as to minimize the loss function.

[0061] In addition, the offset from the head position to the chest, abdomen, waist, buttocks, etc. varies greatly depending on individual differences of users such as physique. Therefore, the UI installation possible area detection unit 606 may estimate the size of the user's waist circumference, etc. based on depth information from a depth sensor such as a TOF camera mounted on the AR glasses, estimate the position of each part of the user's body such as the chest, abdomen, waist, buttocks, etc. (or estimate the offset from the head position to each part), and regard these parts as the UI installation possible area.

[0062] The UI settable area detection unit 606 may also regard a place or a part on the body that can be specified by calibration as a UI installable area. For example, guidance that encourages the user to touch the waist, back, or arm during calibration is output by the image of the display unit 131 or the sound of the speaker 132, and the position touched by the user according to the guidance is regarded as the UI installable area. For example, during calibration, a guidance of "Please touch your arm" is displayed using AR glasses as shown in FIG. 9. In response to this, as shown in FIG. 10, when the user touches the arm according to the guidance, if the UI installable area detection unit 606 can detect the position of the arm (or the offset of the position touched by the user from the position of the head) based on the sensor information of the first sensor unit 110, the UI installable area detection unit 606 detects the arm as a UI installable area. Also, the waist and back are on the back of the torso, and the user has to touch them by feeling, so they cannot be touched as accurately as the position of the arm. Therefore, the position of the user's waist or back may be considered as a rough relative position from the position of the user's head (or the position of the AR glasses) to be a region in which the UI can be installed.

[0063] Since arm length and joint range of motion vary from person to person, the optimal position on the waist or back that a user can touch also varies from person to person. Therefore, a different UI installation area may be set for each user based on the user's usual behavior when using the AR system 100 (or AR glasses) or the results of additional tasks assigned to the user. Also, the UI installation area for each body part may be set based on the results of performing the above-mentioned calibration multiple times for each body part.

[0064] In addition, the physical data used for detecting the UI installation area, etc. may be not only information obtained from a measuring device within the AR system 100, such as the first sensor unit 110 or the second sensor unit 120, but also information obtained from a measuring device external to the AR system 100.

[0065] In addition to the UI installable area detection unit 606 automatically detecting the UI installable area, the UI installable area may be customized for each user. For example, the UI installable area detection unit 606 detects the UI installable area based on an offset from the position of the user's head (or the position of the AR glasses), but if the user has an injury to any of the limbs, the user may not be able to perform finger gestures such as touching or tapping in the automatically detected UI installable area. In such a case, the UI installable area that cannot be used may be excluded by the user's hand (or by manual operation).

[0066] C. AR app storage and restoration In the AR system 100 according to the present embodiment, the AR application can be stored in association with the location of the UI installable area without closing the AR application and while maintaining the execution state. The virtual object of the stored AR application disappears from the user's field of view, but when the user performs a restoration operation, the virtual object of the AR application appears in the user's field of view in the execution state just before the storage. Specifically, the application execution unit 601 identifies the AR application currently held by the user or the AR application designated by the user's hand or finger based on the detection result of the hand and finger position and orientation detection unit 604. Then, when the application execution unit 601 recognizes based on the detection results of the hand and finger position and orientation detection unit 604 and the hand and finger gesture detection unit 605 that the user has performed a gesture of hand and finger instructing storage of the AR application or restoration of the AR application within the UI installable area detected by the UI installable area detection unit 606, the application execution unit 601 stores the UI of the AR application in the part of the body designated by the hand and finger gesture, or restores the AR application from the part of the body where the UI is stored.

[0067] The application execution unit 601 detects that the user's fingers have approached or entered the UI installable area based on the position and orientation of the fingers detected by the hand and finger position and orientation detection unit 604. When the granularity of estimating the body part that the user's fingers approach or touch cannot be finely determined due to insufficient accuracy of the sensors of the gyro sensor 111, the acceleration sensor 112, and the orientation sensor 113 used in the first sensor unit 110, the distance for determining that the fingers have approached or entered may be widened. For example, if a gesture of the fingers occurs near the UI installable area, it is determined that the gesture has occurred within the UI installable area. There is a possibility that the position where the gesture of the fingers has occurred is actually the original body part. Therefore, if a gesture of the fingers occurs near a body part set in the UI installable area, the position of the body part may be updated to the position where the gesture has occurred. Furthermore, in the case where the outward camera 121 can capture an image of the fingers, the hand and finger position and orientation detection unit 604 may detect the position and orientation of the fingers from the camera image of the outward camera 121 .

[0068] Furthermore, the application execution unit 601 determines whether an instruction has been given to the AR application or its virtual object based on a gesture of the hand detected by the hand gesture detection unit 605 in the UI installation area. The instruction here includes storing or restoring the UI of the AR application. Furthermore, the hand gesture detection unit 605 basically detects the shape of the hand and fingers, specifically the angle of the third joint and the second joint of the index finger, and the presence or absence of contact between the fingertips of the thumb and the index finger, but may further detect the passage of the hand in the UI installation area, the action of hitting or pinching a body part that is the UI installation area, and the action of finger snapping in the UI installation area.

[0069] In this embodiment, the application execution unit 601 uses the detection of a hand gesture by the hand gesture detection unit 605 in the UI installable area as a trigger for storing and restoring the UI of the AR application. Different hand gestures may be assigned to storing the UI of the AR application and restoring the stored AR application, respectively, or the same hand gesture may be assigned to both storing and restoring the UI of the AR application. In the latter case, for example, the UI of the AR application is stored in a body part detected as a UI installable area by tapping the body part with a hand, and when the body part where the UI of the AR application is already stored is tapped again, the UI of the AR application is restored from that body part.

[0070] For example, as shown in FIG. 11, a virtual object 1100 of an AR application being executed by the application execution unit 601 appears on the palm of a user's hand. When the virtual object 1100 gets in the way, the user can store the UI of the AR application in the UI installable area while maintaining the execution state of the AR application by performing a predetermined gesture with the hand on a UI installable area on the user's body. The predetermined gesture of the hand and fingers may be, for example, a motion of tapping the UI installable area. The UI installable area may be the back. FIG. 12 shows an example of an operation in which the user taps a part of the body (for example, the back) with the palm of the hand to store the UI of the AR application. The application execution unit 601 holds the execution state of the AR application by linking it to the part of the body where the predetermined gesture of the hand and fingers instructing the storage of the UI of the AR application was performed, and erases the display of the virtual object 1100.

[0071] 11 and 12, it is assumed that the user wears the controller 500 as shown in Fig. 5 on his / her hand. Furthermore, when storage of the UI of the AR application is completed, the application execution unit 601 may provide feedback to the user by operating the vibration presentation unit 133, displaying a message notifying completion on the display unit 131, or emitting a warning sound from the speaker 132.

[0072] After that, when the user wants to use the AR application again, the user performs a predetermined finger gesture on the body part where the UI of the AR application is stored in order to instruct the restoration of the AR application. The gesture for instructing the restoration of the AR application may be the same as or different from the gesture for instructing the storage of the AR application. FIG. 13 shows an example of an operation in which the user restores the UI of the AR application by a gesture of pinching out the UI of the AR application with the thumb and index finger from the body part where the UI of the AR application is stored (e.g., the back). Alternatively, as in FIG. 12, the UI of the AR application may be restored by a gesture of hitting the body part where the UI of the AR application is stored (e.g., the back) with the palm of the hand. The application execution unit 601 restores the AR application, which is linked to the body part where the predetermined finger gesture for instructing the restoration of the AR application is performed, to the original execution state. Then, the application execution unit 601 reproduces the virtual object of the restored AR application on the palm of the user's hand as shown in FIG. 14.

[0073] 13 and 14, it is assumed that the user wears the controller 500 as shown in Fig. 5 on his / her hand. Furthermore, when restoration of the UI of the AR application is completed, the application execution unit 601 may provide feedback to the user by operating the vibration presentation unit 133, displaying a message notifying the user of completion on the display unit 131, or outputting an announcement notifying the user of completion from the speaker 132.

[0074] 11 and 12, the user holds the UI (virtual object) of the AR application to be stored in his / her hand (or places it on the palm of his / her hand), but the target virtual object does not need to be held directly in the hand. For example, the user may point to the target virtual object with his / her finger and then perform a predetermined hand gesture to store it in a body part.

[0075] In this embodiment, the operation of storing the UI of the AR app in or restoring it from a body part is similar to the operation of minimizing an application on a screen and storing it in a task bar or opening an application minimized in the task bar in a general OS from the viewpoint of minimizing digital information. In a general OS screen, an application that is temporarily in the way is minimized and stored in the task bar. In contrast, in the present disclosure, the UI of the AR app that is temporarily in the way is stored in a part detected as a UI installation possible area on the user's body, for example.

[0076] From the viewpoint of storing an object in a body part, the action of storing the UI of the AR app in or restoring it from the body part is similar to the action of a worker taking a tool in or out of a tool holder attached to his waist, or a barber taking scissors in or out of a scissor case attached to his waist. The barber returns the scissors that he has used to the scissor case at his waist and takes the next pair of scissors to be used out of the scissor case. In contrast, in the present disclosure, the UI of the AR app that is temporarily in the way can be stored in a body part detected as a UI installation possible area, or the stored UI of the AR app can be restored from a body part. Because the barber has somatosensation, he can reach out his hand to the scissor case and take the scissors in or out almost accurately without moving his line of sight. In the present disclosure, too, the user can perform the action of storing the UI of the AR app in a body part that is a UI installation possible area, and taking out and restoring the UI of the AR app from that body part, without moving his line of sight, by somatosensation.

[0077] In addition, it may be difficult for the user to understand which part of the body the UI installable area detection unit 606 has detected as the UI installable area, that is, which part of the body the UI of the AR application can be stored in. A user who has experience using the AR system 100 may remember which part of his / her body is the UI installable area, but a beginner may not know at all. In this regard, the application execution unit 601 may display virtual objects such as a miniature avatar or aura in the UI installable area to make it easier for the user to visually understand the UI installable area. FIG. 16 shows an example in which the aura of the UI installable area is displayed on the user's body. However, the UI installable area shown in FIG. 16 is an example and is not limited thereto. In addition, a method for displaying the miniature avatar or aura is not particularly limited.

[0078] The application execution unit 601 may store the UIs of multiple AR applications in separate UI installable areas according to a predetermined gesture of the user's fingers. In this case, it is assumed that the UI installable area detection unit 606 has detected multiple UI installable areas from the user's body. In other words, it is permitted to store the UI of one AR application in each UI installable area. When the user performs a predetermined gesture of the fingers in an already stored UI installable area to store the UI of the AR application by overlapping it, the application execution unit 601 may operate the vibration presentation unit 133, display a warning message on the display unit 131, or emit a warning sound from the speaker 132 to provide feedback to the user that the UI cannot be stored in that location.

[0079] Also, the application execution unit 601 may store the UIs of multiple AR applications in a single UI installation area in a duplicated manner. In this case, a different hand gesture may be used for each AR application to identify the AR application. However, it is difficult for a user to use multiple hand gestures appropriately. Therefore, when a user makes a predetermined hand gesture on a location where the UIs of multiple AR applications are stored, the application execution unit 601 may display a list of AR applications stored in that location so that the user can select an AR application by pointing or using a voice command. Alternatively, when a user makes a predetermined hand gesture on a location where the UIs of multiple AR applications are stored, the application execution unit 601 may restore the UI of the AR application by a first-in-first-out method or a first-in-last-out method.

[0080] FIG. 15 shows, in the form of a flowchart, an operational procedure for storing and restoring an AR application in the AR system 100.

[0081] First, the UI installable area detection unit 606 detects a UI installable area from the user's body parts and the like (step S1501). For example, the UI installable area detection unit 606 detects a location obtained by adding a specific offset from the position of the user's head as the reference, or a part of the user's back or waist, as the UI installable area. The method of detecting the UI installable area has already been described, so details will be omitted here.

[0082] Next, the hand position / posture detection unit 604 acquires the position / posture of the user's fingers (step S1502). The hand position / posture detection unit 604 basically acquires the position / posture of the user's fingers wearing the controller 500 (or the first sensor unit 110) as shown in Fig. 5. However, the hand position / posture detection unit 604 may detect the position / posture of the fingers from the camera image of the outward camera 121.

[0083] The application execution unit 601 monitors whether the user's fingers have approached or entered a UI installable area (step S1503) based on the position and orientation of the fingers detected by the hand position and orientation detection unit 604. If multiple UI installable areas are detected in step S1501, the application execution unit 601 monitors whether the user's fingers have approached or entered any of the UI installable areas.

[0084] When the application execution unit 601 detects that the user's fingers have approached or entered the UI installable area (Yes in step S1503), it then monitors the finger gestures detected by the finger gesture detection unit 605 in the UI installable area (step S1504). The finger gesture detection unit 605 basically detects the shape of the fingers, specifically the angle of the third joint and the second joint of the index finger, and the presence or absence of contact between the fingertips of the thumb and the index finger, but may further detect the passage of the hand in the UI installable area, the action of hitting or pinching a body part that is the UI installable area, and the action of finger snapping within the UI installable area.

[0085] Then, when the application execution unit 601 detects that a gesture of fingers has been made in the UI installation area to instruct the storage or restoration of the UI of the AR application (Yes in step S1504), the application execution unit 601 stores or restores the UI of the AR application in accordance with the instruction of the gesture of fingers (step S1505).

[0086] Here, when storage of the UI of an AR application is instructed, the user holds the UI (virtual object) of the AR application to be stored in his / her hand (or places it on the palm of his / her hand) (see, for example, FIGS. 11 and 12). However, the user does not necessarily hold the UI of the AR application to be stored directly in his / her hand. For example, the user may point with his / her finger at the UI of the AR application to be stored, and the application execution unit 601 may identify the AR application pointed by the user based on the detection result of the hand position / orientation detection unit 604, and store the UI of the AR application in the UI installation possible area detected in step S1503.

[0087] D. Separate display of AR app The application execution unit 601 executes application programs including an AR application under an execution environment provided by the OS, and also controls the display operation of the AR application (virtual object). The virtual object is placed at any location in the real space. For convenience of explanation, in Fig. 11 and Fig. 14, an example is shown in which the AR application is placed within the reach of the user's hand, that is, on the palm of the user's hand, but it is also assumed that the virtual object is placed far away and out of the reach of the user's hand.

[0088] Users can operate AR apps placed within reach with their hands. 。 The application execution unit 601 can estimate the operation performed by the user on the AR application with his / her hand, based on the position where the AR application is placed and the detection results of the hand and finger position and orientation detection unit 604 and the hand and finger gesture detection unit 605. On the other hand, an AR application placed far away and out of reach cannot be directly operated with the hand.

[0089] Therefore, in this embodiment, an operation app for operating an AR app, such as a remote control for remotely operating a device, is placed at the user's hand, and the user can use the operation app to operate an AR app placed far away and out of reach. The application execution unit 601 can estimate an operation performed by the user on the operation app based on the position where the operation app is placed and the detection results of the hand position and orientation detection unit 604 and the hand gesture detection unit 605. Then, the application execution unit 601 controls the operation of the AR app placed far away based on the content of the operation performed by the user on the operation app. The operation app has an operation UI for operating the target AR app.

[0090] The operation application is one of AR applications. The application execution unit 601 executes the operation application to display a virtual object including an operation UI at the user's hand. The user's hand may be, for example, a location obtained by adding a specific offset from the position of the user's head, or the palm or back of the user's hand.

[0091] The operation app displays different operation UIs depending on the AR app to be operated. For example, in the case of an AR app for video playback, the operation UI needs UI parts for instructing operations related to content playback, such as play, pause, fast forward, rewind, and repeat playback. On the other hand, in the case of an AR app for a racing game, UI parts for instructing operations such as the accelerator, brake, and steering wheel are needed. FIG. 17 illustrates an example of an operation UI 1702 consisting of UI parts for content playback operations displayed at the user's hand while watching an AR app 1701 for video playback. The AR app 1701 for video playback and the operation UI 1702 are both virtual objects displayed in the real space, and the user can observe them through the AR glasses. When the user is watching video content of the AR app 1701 displayed in a place that is out of reach, the user can use buttons such as play, pause, fast forward, rewind, and repeat playback included in the operation UI 1702 at hand to perform content playback operations.

[0092] As shown in FIG. 17, by arranging the operation app close to the user, the user can operate the AR app that is located far away and out of reach. However, even if the operation app is arranged close to the user, there is a problem that the user cannot know where each button is located without looking at his / her hand, and cannot operate it accurately. Therefore, all or part of the function (or UI part) for operating the AR app to be operated may be arranged directly on the user's hand. Since the user knows where each button is located by somatosensation, the operation UI can be accurately operated without taking his / her eyes off the AR app (e.g., video playback). Of course, the operation UI may be arranged on a body part other than the hand where the user can accurately grasp the position of the button by somatosensation. In the above section C, an example in which the UI of the AR app is stored in a part of the user's body (UI configurable area) was described, but in this section D, some or all of the functions of the AR app are arranged on a part of the user's body.

[0093] For example, all or a part of the operation UI is arranged on the left palm of the user, and the user operates this with the right hand wearing the controller 500 (see FIG. 5). The application execution unit 601 arranges the operation UI on the left palm of the user, whose position and orientation are recognized from an image captured by the outward camera 121, for example. The user can operate a desired operation UI with the right hand by looking at the operation UI displayed on his / her palm through the AR glasses. Then, the application execution unit 601 recognizes that the user's right hand is in the vicinity of the left palm based on the detection result of the hand and finger position and orientation detection unit 604, and further recognizes that any of the buttons arranged on the left palm has been operated based on the detection result of the hand and finger gesture detection unit 605, and executes the operation of the AR application instructed by the button. For example, when the application execution unit 601 recognizes an operation of the user pressing a play button while an AR application for playing videos is being executed, it starts playing video content, and when it recognizes an operation of pressing fast forward, it fast forwards the content.

[0094] Fig. 18 shows an example in which virtual objects 1801-1803 of operation UIs for content playback, such as play, fast forward, and rewind, are displayed on the left palm of the user. Fig. 19 shows an example in which a virtual object 1901 of a jog controller that instructs frame forward or backward by rotation is displayed on the left palm of the user as an operation UI for content playback. In both of the examples shown in Figs. 18 and 19, the application execution unit 601 places virtual objects of the operation UIs 1801-1803 or 1901 on the left palm of the user, whose position and orientation are recognized from an image captured by the outward camera 121, for example. The user can see the operation UIs 1801-1803 or 1901 displayed on his or her palm through the AR glasses. Then, based on the detection result of the finger position and posture detection unit 604, the application execution unit 601 recognizes that the user's right hand is in the vicinity of the left palm, and further recognizes the amount of operation on the operation UI placed on the left palm based on the detection result of the finger gesture detection unit 605.In response to the amount of operation, the application execution unit 601 executes the operation of the AR application, such as playing, fast-forwarding, rewinding, or frame-forwarding the played content.

[0095] In addition, functions for operating an AR application may be arranged on the back of the hand instead of the palm of the user's hand. While the palm is almost flat, the back of the hand has a characteristic shape in which the index finger, middle finger, ring finger, and little finger each have a convex part due to the metacarpal head. Therefore, as shown in FIG. 20, functions for operating an AR application such as play, pause, fast forward, and rewind may be individually assigned to the four convex parts of the index finger, middle finger, ring finger, and little finger of the left hand. The user can touch the convex parts of each finger with his / her right hand by somatosensation without directly looking at the back of the left hand. For example, the application execution unit 601 assigns the functions of play, pause, fast forward, and rewind to the four convex parts of the back of the user's left hand, the position and posture of which are recognized from the image captured by the outward camera 121, but it is not necessary to arrange virtual objects. The user touches any of the convex parts on the back of the left hand to operate the AR application without directly looking at the palm of the left hand. Then, based on the detection results of the finger position and posture detection unit 604, the application execution unit 601 recognizes that the user's right hand is near the back of the left hand, and further recognizes based on the detection results of the finger gesture detection unit 605 that the user has touched any of the convex parts on the back of the left hand with the fingers of the right hand.Then, depending on the function assigned to that convex part, the application execution unit 601 executes the operation of the AR application such as playing, fast-forwarding, rewinding, or frame-forwarding the played content.

[0096] In the AR system 100, as shown in Fig. 17 to Fig. 20, in order to arrange the operation UI of the AR application separately from the virtual object of the AR application main body, the AR application holds necessary information for making the operation UI separable or holds it in association with the AR application. In addition, when arranging the operation UI on the user's hand, such as the palm or back of the user's left hand, the number of pieces that can be arranged may be limited, such as the metacarpal head. For this reason, information for determining the operation UI that should be preferentially arranged on the user's hand, such as determining the priority order among multiple operation UIs, is also held by the AR application or holds it in association with the AR application.

[0097] Furthermore, when the operation unit of the AR application is placed at the user's hand separately from the virtual object of the AR application main body, a rule is determined such that the area in which the operation unit can be placed is within a certain distance from the position of the AR glasses (or the second sensor unit 120). This ensures that the hand position and orientation detection unit 604 and the hand gesture detection unit 605 can detect the position and orientation of the user's fingers operating the operation unit, and the application execution unit 601 can operate the AR application based on the operation of the user's fingers on the operation unit.

[0098] It is also possible to recognize the operation of the user's fingers on an operation unit arranged on the user's hand, palm, or back of the hand through an image captured by outward camera 121. In contrast, in this embodiment, the first sensor unit 110 is attached to the user's hand (right hand) performing the operation, so that even if occlusion occurs in the field of view of outward camera 121, the operation of the user's fingers on the operation unit can be detected with high accuracy based on sensor information from first sensor unit 110.

[0099] Furthermore, if the detection accuracy of the hand position and orientation detection unit 604 is insufficient, the application execution unit 601 may expand the size of an area to which one operation is assigned (for example, enlarge the play button) or simplify the shape of the operation UI displayed as a virtual object, so as to make it easier for the user to operate the operation unit. For example, the application execution unit 601 may determine the size and shape of the operation UI by using a machine learning model that estimates the optimal size and shape of the operation UI for the AR application. For this machine learning model, a loss function based on, for example, the number of times the user's hand operation fails during use is defined, and deep learning is performed by error backpropagation so as to minimize the loss function.

[0100] E. Operation UI layout Section D above has described an embodiment in which the operation UI of an AR application is arranged separately from the virtual objects of the AR application itself, and further, has described optimizing the size and shape of the operation UI.

[0101] As shown in Figure 20, when the operation UI is arranged using the characteristics of a body part such as the metacarpal head, the user can operate the operation UI accurately through somatosensation without directly looking at their own body part. Also, when buttons and menus are arranged on body parts with no distinctive shapes such as the back or palm, the user can operate the operation UI accurately by optimizing the size and shape of each button to prevent operation errors.

[0102] On the other hand, when an operation UI for inputting coordinates is placed on a part of the body, there is a concern that erroneous input may occur depending on the relative positions of the operation UI and the user's fingers operating it.

[0103] For example, as shown in Fig. 21, when the touch panel operation UI 2101 is placed on the palm of the user's left hand and the controller 500 (or the first sensor unit 110) is operated with the fingers of the right hand, the user may intend to perform a horizontal flick operation with the index finger of the right hand, but the input may not be correct depending on the position and posture of the left hand. Specifically, as shown in Fig. 22, if the position and posture of the left hand are such that the coordinate system of the touch panel placed on the palm of the user's left hand coincides with (or is parallel to) a local coordinate system having the center of gravity of the user's body as the origin (or a local coordinate system imagined by the user in his / her head), when the user performs a horizontal flick operation on the touch panel operation UI with his / her right hand, the hand gesture detection unit 605 can detect the coordinate input of the horizontal flick operation. On the other hand, in the case of a left hand posture in which the coordinate system of the touch panel is inclined with respect to the user's local coordinate system as shown in Fig. 23, even if the user intends to perform a horizontal flick operation with the right hand based on the user's local coordinate system, the hand gesture detection unit 605 will detect a coordinate input of a diagonal flick operation. For example, if the touch panel 2101 is arranged based on the coordinate system of the left hand recognized from an image captured by the outward camera 121, etc., different coordinate inputs will be made depending on the posture of the left hand even if the same flick operation is performed with the right hand.

[0104] Therefore, in this embodiment, the orientation of the operation UI to be placed on a body part where the operation UI for coordinate input is placed is corrected based on the posture of the body part where the operation UI for coordinate input is placed. For example, as shown in Fig. 21, when the operation UI 2101 of the touch panel is placed on the palm of the left hand of the user, the orientation of the left hand is acquired and the orientation (coordinate system) of the operation UI for coordinate input is corrected based on any one of the orientation of the user's body, the direction in which the object to be operated by the operation UI is placed, or the line of sight of the user as a reference, rather than the coordinate system of the body part detected by the finger position and orientation detection unit 604 of the touch panel.

[0105] FIG. 24 shows an example of correcting the orientation (coordinate system) of the operation UI for inputting coordinates based on the orientation of the user's body. The application execution unit 601 can detect the orientation of the body based on, for example, sensor information from the second sensor unit 120. The application execution unit 601 can also detect the position and orientation of the user's left palm based on, for example, the recognition result of the image captured by the outward camera 121. When the first sensor unit 110 is also attached to the left hand, the position and orientation of the left hand can be detected based on the sensor information from the first sensor unit 110. Then, the application execution unit 601 places the operation UI on the palm of the user's left hand so that the orientation of the operation UI matches the orientation of the body (or so that the coordinate system of the operation UI and the user's local coordinate system are parallel) regardless of the orientation of the left hand. In this case, when the user performs a horizontal flick operation with the right hand on the operation UI of the touch panel, the finger gesture detection unit 605 can detect the coordinate input of the horizontal flick operation.

[0106] FIG. 25 shows an example of correcting the direction (coordinate system) of the operation UI for inputting coordinates based on the direction in which the object to be operated by the operation UI is arranged. In FIG. 25, the operation object of the operation UI is a virtual object of an AR application to be operated on a touch panel. The application execution unit 601 knows the position where the virtual object 2501 of the AR application to be operated is arranged. In addition, the application execution unit 601 can detect the position and orientation of the palm of the left hand of the user based on the recognition result of the image captured by the outward camera 121, for example. In the case where the first sensor unit 110 is also attached to the left hand, the position and orientation of the left hand can be detected based on the sensor information from the first sensor unit 110. Then, the application execution unit 601 arranges the operation UI on the palm of the left hand of the user so that the orientation of the operation UI coincides with the front direction of the virtual object 2501 of the AR application to be operated. In this case, when the user looks at virtual object 2501 of the AR application and performs a horizontal flick operation with his / her right hand along the plane on which virtual object 2501 is placed, hand gesture detection unit 605 can detect the coordinate input of the horizontal flick operation on virtual object 2501.

[0107] FIG. 26 shows an example of correcting the direction (coordinate system) of the operation UI for inputting coordinates based on the line of sight of the user. The application execution unit 601 detects the line of sight of the user by, for example, performing image recognition on the position and orientation of the head obtained from the sensor information of the second sensor unit 120, or the image of the user's eyes captured by the inward-facing camera 122. In the example shown in FIG. 26, the user's line of sight moves on a virtual object 2601 of the AR application. In addition, the application execution unit 601 can detect the position and orientation of the left palm of the user based on, for example, the recognition result of the image captured by the outward-facing camera 121. In the case where the first sensor unit 110 is also attached to the left hand, the position and orientation of the left hand can be detected based on the sensor information from the first sensor unit 110. Then, the application execution unit 601 places the operation UI on the palm of the user's left hand so that the orientation of the operation UI coincides with the line of sight of the user to be operated. In this case, when the user performs a horizontal flick operation with his / her right hand based on the line of sight while looking in any direction, the hand gesture detection unit 605 can detect the coordinate input of the horizontal flick operation on the virtual object 2601.

[0108] The example shown in Fig. 26 assumes a case where a user operates an AR application while lying down on a sofa, for example. The orientation of the palm when lying down is significantly different from that when standing, but by correcting the orientation of the operation UI for the palm according to the rule that the orientation of the operation UI is corrected based on the user's line of sight, the user can maintain comfortable operation.

[0109] By correcting the orientation of the operation UI using the methods exemplified in FIGS. 24 to 26, operations unintended by the user are prevented from being performed on the AR application, and efficient operations can be realized by the user. [Industrial Applicability]

[0110] Although the present disclosure has been described in detail above with reference to specific embodiments, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure.

[0111] In this specification, the present disclosure has been mainly described with reference to an embodiment in which the present disclosure is applied to an AR system, but the gist of the present disclosure is not limited thereto. For example, the present disclosure can be similarly applied to a VR system that allows a user to perceive a virtual space as reality, or an MR system that mixes reality and virtuality.

[0112] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as being limited. In order to judge the gist of the present disclosure, the claims should be taken into consideration.

[0113] The present disclosure can also be configured as follows.

[0114] (1) an acquisition unit for acquiring a position, posture, or shape of a first part on a user's body; a control unit that controls a virtual object to be displayed in a real space on a display device based on the position, orientation, or shape of the first part; An information processing device comprising:

[0115] (2) the acquisition unit acquires a position and orientation of the first part based on a sensor attached to the first part; The information processing device according to (1) above.

[0116] (3) the first portion includes a user's hand or finger; The acquisition unit acquires a position and orientation of a user's hand or finger or a shape of a finger based on sensor information from a position and orientation sensor attached to the user's hand or finger. An information processing device according to any one of (1) or (2) above.

[0117] (4) The control unit recognizes that the first part has performed a predetermined operation on a second part on the user's body based on a position, posture, or shape of the first part. An information processing device according to any one of (1) to (3) above.

[0118] (5) When the control unit recognizes that the first part has performed a first operation on a second part on the user's body based on the position, posture or shape of the first part acquired by the acquisition unit, the control unit associates an execution state of the virtual object with the second part and stores the execution state, and stops displaying the virtual object on the display device. An information processing device according to any one of (1) to (4) above.

[0119] (6) When the control unit recognizes that the first part has performed a second operation on a second part on the user's body based on the position, posture, or shape of the first part acquired by the acquisition unit, the control unit restores the display of the virtual object based on the execution state associated with the second part and stored. The information processing device according to (5) above.

[0120] (7) Further comprising a detection unit that detects an area on the user's body where the second part can be placed. An information processing device according to any one of (4) to (6) above.

[0121] (8) The detection unit detects, as the second part, a location on the user's body that is obtained by adding a predetermined offset to a third part whose position and orientation can be acquired. The information processing device according to (7) above.

[0122] (9) The detection unit detects, as a second part, a region on a body whose position can be estimated based on the position and orientation of the first part. An information processing device according to any one of (7) or (8) above.

[0123] (10) The detection unit detects a second part whose position can be estimated by inverse kinematics calculation based on position and orientation information of the first part and a third part whose positions and orientations can be acquired. An information processing device according to any one of (7) to (9) above.

[0124] (11) The detection unit detects a size or shape of the second part based on the operability of the first part. An information processing device according to any one of (7) to (10) above.

[0125] (12) The detection unit detects a second portion whose location can be identified by calibration touched with the first portion. An information processing device according to any one of (7) to (11) above.

[0126] (13) The control unit controls to place an operation unit, which enables the user to operate a virtual object placed far away from the user, at the user's hand. An information processing device according to any one of (1) to (12) above.

[0127] (14) The control unit controls the operation unit to be placed at a fourth part on the user's body. The information processing device according to (13) above.

[0128] (15) The control unit corrects an orientation of the operation unit disposed in the fourth portion. The information processing device according to (14) above.

[0129] (16) The control unit corrects the orientation of the operation unit based on any one of the orientation of the user's body, the direction in which a virtual object is placed, or the direction of the user's line of sight. The information processing device according to (15) above.

[0130] (17) acquiring a position, posture, or shape of a first part on a body of a user; a control step of controlling a virtual object to be displayed in a real space on a display device based on the position, orientation or shape of the first part; An information processing method comprising the steps of:

[0131] (18) an acquisition unit that acquires a position, posture, or shape of a first part on a body of a user; a control unit that controls a virtual object to be displayed in a real space on a display device based on the position, posture, or shape of the first part; A computer program written in a computer-readable form to cause a computer to function as a

[0132] (19) A display device for displaying a virtual object in a real space; an acquisition unit that acquires a position, posture, or shape of a first part on a user's body; a control unit that controls display of a virtual object based on a position, orientation, or shape of the first part; An augmented reality system comprising: [Explanation of symbols]

[0133] 100...AR system, 110...first sensor 111... gyro sensor, 112... acceleration sensor, 113... orientation sensor 120: second sensor unit; 121: outward camera 122...inward-facing camera, 123...microphone, 124...gyro sensor 125...acceleration sensor, 126...orientation sensor 131: display unit, 132: speaker, 133: vibration presentation unit 134: communication unit, 140: control unit, 150: storage unit 300: AR system, 301: AR glasses, 302: controller 400: AR system, 401: AR glasses, 402: controller 403...Information terminal 500...controller, 501, 502, 503...IMU 511, 512, 513...Band 601: application execution unit; 602: head position / orientation detection unit 603: Output control unit; 604: Hand position / posture detection unit 605: hand gesture detection unit; 606: UI setting area detection unit

Claims

1. an acquisition unit that acquires a position, posture, or shape of a first part including at least one of a user's hand and fingers; a control unit that recognizes that the first part has performed a predetermined operation on a second part of a user's body other than a hand or a finger, based on a position, posture, or shape of the first part, and controls a virtual object to be displayed in a real space on a display device in response to the predetermined operation; An information processing device comprising:

2. the acquisition unit acquires a position and orientation of the first body part based on a sensor attached to the first body part; The information processing device according to claim 1 .

3. the acquisition unit acquires a position and orientation of the first part or a shape of a finger based on sensor information from a position and orientation sensor attached to the first part. The information processing device according to claim 1 .

4. When the control unit recognizes that the first body part has performed a first operation on the second body part based on the position and posture or the shape of the first body part acquired by the acquisition unit, the control unit associates an execution state of the virtual object with the second body part and stores the execution state, and stops displaying the virtual object by the display device. The information processing device according to claim 1 .

5. When the control unit recognizes that the first body part has performed a second operation on the second body part based on the position and posture or the shape of the first body part acquired by the acquisition unit, the control unit restores the display of the virtual object based on the execution state associated with the second body part and stored. The information processing device according to claim 4.

6. Further comprising a detection unit that detects an area on the user's body where the second part can be placed. The information processing device according to claim 1 .

7. the detection unit detects, as the second part, a location on the user's body that is obtained by adding a predetermined offset to a third part whose position and orientation can be acquired; The information processing device according to claim 6.

8. the detection unit detects, as the second part, a region on the body whose position can be estimated based on the position and orientation of the first part; The information processing device according to claim 6.

9. the detection unit detects the second part, the position of which can be estimated by inverse kinematics calculation, based on position and orientation information of the first part and a third part, the positions and orientations of which can be acquired; The information processing device according to claim 6.

10. The detection unit detects a size or a shape of the second part based on the operability of the first part. The information processing device according to claim 6.

11. The detection unit detects the second portion, the location of which can be identified by calibration touched with the first portion. The information processing device according to claim 6.

12. The control unit controls to place an operation unit, which allows the user to perform an operation on a virtual object placed far away from the user, at the user's hand. The information processing device according to claim 1 .

13. The control unit controls the operation unit to be placed at a fourth part on a body of a user. The information processing device according to claim 12.

14. The control unit corrects an orientation of the operation unit disposed in the fourth portion. The information processing device according to claim 13.

15. The control unit corrects the orientation of the operation unit based on any one of the orientation of the user's body, the direction in which a virtual object is placed, or the direction of the user's line of sight. The information processing device according to claim 14.

16. acquiring a position, posture, or shape of a first part including at least one of a user's hand and fingers; a control step of recognizing that the first part has performed a predetermined operation on a second part on the user's body other than a hand or a finger, based on a position, posture, or shape of the first part, and controlling a virtual object to be displayed in a real space on a display device in response to the predetermined operation; An information processing method comprising the steps of:

17. an acquisition unit that acquires a position, posture, or shape of a first part consisting of at least one of a user's hand and a finger; a control unit that recognizes that the first part has performed a predetermined operation on a second part of the user's body other than a hand or a finger, based on a position, posture, or shape of the first part, and controls a virtual object to be displayed in a real space on a display device in response to the predetermined operation; A computer program written in a computer-readable form to cause a computer to function as a

18. A display device that displays a virtual object in a real space; an acquisition unit that acquires a position, posture, or shape of a first part including at least one of a user's hand and fingers; a control unit that recognizes that the first part has performed a predetermined operation on a second part on a user's body other than a hand or a finger, based on a position, posture, or shape of the first part, and controls display of a virtual object in response to the predetermined operation; An augmented reality system comprising:

Citation Information

Patent Citations

  • Image processing apparatus, and image processing method

    JP2009104249A

  • Image display device, image display method, and computer program

    JP2014093050A

  • Image recognition device

    JP2016162321A

  • Glove interface object

    JP2017530452A

  • Information processing device, information processing method, and program

    WO2018008226A1