Method and apparatus for supporting interaction between virtual environment and real world

Abstract

To provide a method and apparatus for supporting interaction between a virtual environment and the real world.SOLUTION: An electronic apparatus 100 for supporting interaction between a virtual environment and the real world is provided, comprising a memory 120, and a control unit 140 configured to determine an interaction state for a physical object in the real world and provide a virtual environment according to a blending technique selected based on a determination result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments disclosed herein relate to methods and apparatus for supporting interaction between a virtual environment and the real world, and more particularly to methods and apparatus for supporting interaction state-adaptive virtual-reality blending techniques that enable interaction of physical objects within virtual reality.

[0002] This research was conducted as a result of the "Artificial Intelligence Graduate School Support (Seoul National University)" project (IITP-2021-0-01343) under the Ministry of Science and ICT and the Institute for Information and Communications Technology Planning (IITP)'s ICT Broadcasting Innovation Talent Development Project.

[0003] This research was conducted as a result of the "Ultra-Realistic Continuous Hybrid Telepresence Platform" project (NRF-2022R1A2C3008495) of the Ministry of Science and ICT and the National Research Foundation of Korea (NRF)'s Individual Basic Research Project. [Background technology]

[0004] Virtual reality (VR) allows users to experience a virtual environment by creating the illusion that they are actually in a virtual space. Users can easily experience the virtual environment through electronic devices such as a head-mounted display (HMD) or smart glasses.

[0005] Furthermore, recently popular electronic devices may support the blending of virtual and reality. These virtual reality systems support the blending of virtual and reality by using techniques such as substituting real-world objects for virtual objects and rendering them in a virtual environment, or by floating a camera view showing the real world in the virtual environment. However, both of these methods have problems that seriously reduce usability and immersion. Specifically, the frequent transitions between virtual and reality can reduce a user's immersion, and the difficulty of recognizing physical objects in the real world can lead to frequent errors during interaction with physical objects (e.g., collisions while moving an object, or frequent fumbling to grasp an object).

[0006] Therefore, a new dimension of functionality is needed to solve the above-mentioned problems and improve the interaction between the virtual environment and the real world.

[0007] On the other hand, the above-mentioned background art is technical information that the inventor possessed in order to derive the present invention or that he acquired in the process of deriving the present invention, and it cannot necessarily be said to be publicly known art that was disclosed to the general public prior to the filing of the present invention. Summary of the Invention [Problem to be solved by the invention]

[0008] The embodiments disclosed herein aim to present a method and apparatus for supporting interaction between a virtual environment and the real world.

[0009] The embodiments disclosed herein aim to facilitate a smooth transition between a virtual environment and a real-world physical environment.

[0010] The embodiments disclosed herein aim to provide a desirable virtual-reality blending technique that takes into account gradual transitions and interaction states between the virtual and real worlds.

[0011] The embodiments disclosed herein aim to maintain a sense of presence and usability even as the interaction state between a user and a physical object changes continuously.

[0012] The embodiments disclosed herein aim to support complex interactions with physical objects in the real world while maintaining a balance between usability for interactions with physical objects located in the real world and a sense of immersion in the virtual environment.

[0013] The embodiments disclosed herein aim to provide the physical world information necessary for each interaction state while maintaining a balance between presence and usefulness. [Means for solving the problem]

[0014] As a technical means for achieving the above-mentioned technical objectives, according to one embodiment, an electronic device for supporting interaction between a virtual environment and the real world can include a memory and a control unit that determines an interaction state with a physical object in the real world and provides a virtual environment using a mixing technique selected based on the determination result.

[0015] According to another embodiment, a method for supporting interaction between a virtual environment and a real world, executed by an electronic device, may include determining an interaction state for a physical object in the real world, and providing the virtual environment with a blending technique selected based on a result of the determination.

[0016] According to yet another embodiment, a computer-readable recording medium may have recorded thereon a program for executing a method for supporting interaction between a virtual environment and the real world, the method including determining an interaction state with a physical object in the real world, and providing the virtual environment with a blending technique selected based on a result of the determination.

[0017] According to yet another embodiment, there may be provided a computer program stored on a medium for executing a method for supporting interaction between a virtual environment and the real world when executed by an electronic device, the method including determining an interaction state with a physical object in the real world, and providing the virtual environment with a blending technique selected based on a result of the determination. [Effects of the Invention]

[0018] According to any one of the above-mentioned solutions, a method and an apparatus for supporting interaction between a virtual environment and the real world can be presented.

[0019] In addition, any one of the above-mentioned solutions can support a smooth transition between a virtual environment and a physical environment in the real world, thereby providing a gradual virtual-real world (Gradual Reality), and enabling gradual interaction with real objects in the virtual environment.

[0020] Furthermore, according to any one of the above-mentioned solutions, a desired virtual-reality blending technique can be provided taking into consideration gradual transitions and interaction states between the virtual and real worlds, allowing a user to naturally interact with physical objects located in the real world in a virtual environment, thereby significantly improving the user's virtual reality experience.

[0021] Furthermore, by using any one of the above-mentioned solutions, the user's presence and usability can be maintained at a consistently high level even when the interaction state between the user and the physical object continuously changes.

[0022] In addition, by using any one of the above-mentioned problem-solving means, it is possible to support complex interactions with physical objects in the real world while maintaining a balance between usability for interactions with physical objects located in the real world and immersion in the virtual environment.

[0023] Furthermore, by using any one of the above-mentioned solutions, it is possible to provide the physical world information necessary for each interaction state while maintaining a balance between presence and usefulness, thereby providing improved cross reality experiences.

[0024] The effects obtained from the disclosed embodiments are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the embodiments disclosed below pertain. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a block diagram illustrating an apparatus for supporting interaction between a virtual environment and the real world, according to one embodiment. [Figure 2] 1 is an exemplary diagram illustrating an apparatus for supporting interaction between a virtual environment and the real world, according to one embodiment. [Figure 3] 1 is an exemplary diagram illustrating an apparatus for supporting interaction between a virtual environment and the real world, according to one embodiment. [Figure 4] 1 is an exemplary diagram illustrating an apparatus for supporting interaction between a virtual environment and the real world, according to one embodiment. [Figure 5] 1 is an exemplary diagram illustrating an apparatus for supporting interaction between a virtual environment and the real world, according to one embodiment. [Figure 6] 1 is an exemplary diagram illustrating an apparatus for supporting interaction between a virtual environment and the real world, according to one embodiment. [Figure 7]1 is an exemplary diagram illustrating an apparatus for supporting interaction between a virtual environment and the real world, according to one embodiment. [Figure 8] 1 is an exemplary diagram illustrating an apparatus for supporting interaction between a virtual environment and the real world, according to one embodiment. [Figure 9] 1 is a flowchart illustrating a method for supporting interaction between a virtual environment and the real world, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] Various embodiments will be described in detail below with reference to the accompanying drawings. The embodiments described below may be implemented in various modified forms. In order to more clearly describe the features of the embodiments, detailed descriptions of matters that are well known to those skilled in the art to which the following embodiments pertain will be omitted. In addition, parts of the drawings that are not relevant to the description of the embodiments will be omitted, and similar parts will be designated by similar reference numerals throughout the specification.

[0027] Throughout the specification, when a certain component is said to be "connected" to another component, this includes not only "directly connected" but also "connected via another component in between." Furthermore, when a certain component is said to "include" another component, this does not exclude the other component, but means that the other component may also be included, unless otherwise specified.

[0028] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0029] Before describing the embodiments, the meanings of the terms used below will be defined.

[0030] A physical object in the real world can be divided into (i) a "basic shape", (ii) a "grasp area", (iii) a "manipulation area", and (iv) other combinations. The relationship between each area of ​​a physical object and its affordance is defined as follows:

[0031] A "primitive shape" refers to the simplest geometric representation of a physical object. For example, primitive shapes can include cubes, cylinders, spheres, and cones (e.g., a bottle is a cylinder, a basketball is a sphere, and a box is a cube). The area corresponding to a primitive shape indicates the maximum volume of the object and facilitates affordances including contain, wrap-grasp, support, etc.

[0032] "Grasping area" refers to the outer region of an object's primitive shape that is designed for effective grasping, such as a handle, knob, pistol, and bar. The grasping area also supports other affordances (e.g., movement, lifting) that can be achieved through precise grasping.

[0033] "Manipulating area" includes areas that provide affordances for simple manipulations such as opening, pouring, pushing, pulling, and holding. Examples include the opening of a Coke can, the hole in a weight plate, the cover and lid of a water bottle, and a spray gun.

[0034] Areas other than (i) the basic shape, (ii) the grasping area, and (iii) the operation area indicate areas that are not related to affordance and interaction, such as decoration of the object (e.g., rabbit ears on a home camera cover).

[0035] All physical objects necessarily have a primitive shape, but may not have an area that can be grasped or manipulated. That is, they may not have a graspable or a manipulation area. For example, a simple shape like a paper cup or a basketball may have neither a graspable nor a manipulation area, a mug may have only a graspable area, and a can may have only a manipulation area, while a water bottle may have all the components of a primitive shape, a graspable area, and a manipulation area. According to one embodiment disclosed herein, affordances may include affordances handled by the 3D AffordanceNet (such as contain, wrap-grab, grasp, support, grasp, move, lift, open, pull, push, hold, and pour).

[0036] Terms that require explanation other than those defined above will be explained separately below.

[0037] FIG. 1 is a block diagram showing an apparatus for supporting interaction between a virtual environment and the real world according to one embodiment, and FIGS. 2 to 8 are illustrative diagrams for explaining an apparatus for supporting interaction between a virtual environment and the real world.

[0038] A device for supporting interaction between a virtual environment and the real world may be implemented by the electronic device 100. That is, the electronic device 100 may support interaction between a virtual environment and the real world. Such an electronic device 100 may include a user terminal or may include a user terminal and a server. For example, the electronic device 100 may implement a method according to the embodiments disclosed herein by a user terminal or by a server-client system.

[0039] Here, according to an embodiment, the user terminal may be embodied as a computer, a portable terminal, a television, a wearable device, etc. that can be connected to a remote server via a network or can be connected to other terminals and servers. Here, the computer may include, for example, a notebook PC, a desktop PC, or a laptop PC equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that ensures portability and mobility, such as a Personal Communication System (PCS), a Personal Digital Cellular (PDC), a Personal Handyphone System (PHS), a Personal Digital Assistant (PDA), a Global System for Mobile communications (GSM), an International Mobile Telecommunication (IMT)-2000, a Code Division Multiple Access (CDMA)-2000, a W-Code Division Multiple Access (W-CDMA), a Wireless Broadband Internet (Wibro), a smartphone, a Mobile Worldwide Interoperability for Microwave Access (Mobile WiMAX), etc. The television may include Internet Protocol Television (IPTV), Internet Television, terrestrial TV, cable TV, etc. A wearable device is a type of information processing device that can be worn directly on the human body, such as a watch, glasses, accessory, clothing, or footwear, and can be connected to a remote server or other terminal via a network, either directly or through another information processing device.

[0040] The electronic device 100 can communicate with an external device and obtain tracking data from the external device about movements such as changes in the user's position, changes in the movement of the user's hands and fingers, changes in the position of physical objects, and movements of physical objects.

[0041] That is, electronic device 100 may acquire tracking data that tracks the movement of at least one of a user, a user's hand, and an object, according to various embodiments. That is, "tracking data" refers to data that tracks the movement of an object or a user (or a user's hand or finger), such as the position of an object in real life, the movement of an object due to its movement, the position of the user, the position of the user's hand, the movement of the user's hand or finger, etc. As described above, electronic device 100 may acquire data on the movement of a user's hand and / or data on the movement of an object sensed by an external device as tracking data, and may also acquire tracking data via an input / output unit, such as a sensor or camera, mounted inside electronic device 100.

[0042] According to one embodiment, the electronic device 100 is implemented as a server-client system. The user terminal included in the electronic device 100 is a wearable device that may be implemented as an HMD (Head-Mounted Display) that displays a virtual environment image to a user and changes the image according to the user's movements. Such a user terminal may have a built-in or attached camera to acquire a camera view image showing the real world and communicate with a server. The electronic device 100 may also communicate with one or more sensor devices. The sensor device may be, for example, a sensor for tracking the position and movement of a user's hand, such as a depth camera, a depth and motion sensing camera, and / or a 3D motion control device. The depth and motion sensing camera is a device that senses the depth and movement of an object, such as a "Zed Mini camera," and the 3D motion control device is a device that senses the movement of a user's hand or fingers, such as a "Leap Motion Controller." In addition, the electronic device 100 may communicate with a tracker (6DoF tracker) that is attached to a user or an object and tracks the movements of the user or object in real time to acquire information about the movements, for example, the electronic device 100 may communicate with a "VIVE tracker 3.0." Examples of how the electronic device 100 acquires tracking data are not limited to those described above, and the electronic device 100 may acquire tracking data using known techniques.

[0043] As described above, the electronic device 100 can use the acquired tracking data to analyze the distance between the user and an object, the distance between the user's hand and an object, or the distance between multiple objects, and monitor virtual button activation. For example, the electronic device 100 can analyze the distance between the hand and an object using the difference between the hand position and the object position extracted from the tracking data, or can detect finger movement from the tracking data and monitor whether the finger clicked a virtual button.

[0044] The electronic device 100 can use the tracking data to track the state of an interaction between a user and an object, an interaction between a user's hand and an object, or an interaction between objects. Hereinafter, for convenience of explanation, the distance in "the distance between a user and an object" refers to the distance between the object and at least one of the user's body, the user's hand, a specific finger of the user, and the user's head.

[0045] Referring to FIG. 1, an electronic device 100 according to an embodiment may include an input / output unit 110 , a memory 120 , a communication unit 130 , and a control unit 140 .

[0046] The input / output unit 110 may include an input unit for receiving input from a user and an output unit for displaying information such as a result of execution of a task or a status of the electronic device 100. For example, the input / output unit 110 may include an operation panel for receiving user input and a display panel for displaying a screen.

[0047] Specifically, the input unit may include devices capable of receiving various types of user input, such as a keyboard, physical buttons, a touch screen, a camera, a microphone, a sensor, etc. The output unit may include a display panel, a speaker, etc. However, without being limited thereto, the input / output unit 110 may include configurations supporting various inputs and outputs.

[0048] According to the embodiment, when the electronic device 100 is embodied as an HMD and worn on the user's head, the input / output unit 110 may sense the changed position when the user's position (or the user's head position) in a three-dimensional space changes due to the user's head or body movement, and may receive information about the change in the user's position as an input value. Alternatively, according to the embodiment, when the electronic device 100 is embodied as an HMD and worn on the user's head, the input / output unit 110 may be embodied as a camera and may receive a camera view of the real world as an input value.

[0049] Various types of data, such as files, applications, and programs, may be embedded or stored in the memory 120. The control unit 140 may access and use data stored in the memory 120 or store new data in the memory 120. The control unit 140 may also execute programs embedded in the memory 120. Referring to FIG. 1, the memory 120 may be embedded with a program for executing a method for supporting interaction between a virtual environment and the real world.

[0050] According to one embodiment, when an input requesting the start of a program is received from a user via the input / output unit 110, the control unit 140 can execute a program stored in the memory 120 to perform a method for supporting interaction between a virtual environment and the real world.

[0051] The communication unit 130 may perform wired or wireless communication with other devices or networks. To this end, the communication unit 130 may include a communication module that supports at least one of various wired or wireless communication methods. For example, the communication module may be implemented in the form of a chipset.

[0052] The wireless communication supported by the communication unit 130 may be, for example, Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Bluetooth (registered trademark), Ultra Wide Band (UWB), Near Field Communication (NFC), etc. The wired communication supported by the communication unit 130 may be, for example, USB or High Definition Multimedia Interface (HDMI), etc.

[0053] According to an embodiment, the communication unit 130 can communicate with an external device (not shown) that provides tracking data, and can obtain tracking data from the external device (not shown) and provide it to the control unit 140.

[0054] The control unit 140 controls the overall operation of the electronic device 100 and may include a processor such as a CPU, a GPU, etc. The control unit 140 may control other components included in the electronic device 100 to perform an operation corresponding to a user input received via the input / output unit 110.

[0055] For example, the control unit 140 may execute a program stored in the memory 120 , read a file stored in the memory 120 , or store a new file in the memory 120 .

[0056] The control unit 140 may provide a virtual environment image in which virtual objects corresponding to at least some of the physical objects located in the real world are rendered. For example, the control unit 140 may implement a virtual environment such that each physical object located in the real world corresponds to each virtual object located in the virtual environment. The control unit 140 may render and provide a virtual environment according to a "recognition state" among interaction states described below.

[0057] According to an embodiment, the control unit 140 can determine an interaction state with a physical object in the real world and provide a virtual environment rendered by a blending method selected based on the determination result.

[0058] The control unit 140 may track an interaction state for an object and apply a blending technique specific to each physical object. For example, the control unit 140 may track an interaction state between objects, an interaction state between a user and an object, or an interaction state between a user's hand and an object, and provide a rendered virtual environment by applying a blending technique corresponding to the interaction state to a virtual object corresponding to the object.

[0059] To this end, the control unit 140 may group each physical object included in the real-world image into "target objects" that interact with the user and "non-target objects" that do not interact with the user. The control unit 140 applies a blending technique to target objects to improve usability, and minimizes the blending technique for non-target objects to maintain their presence.

[0060] The control unit 140 may classify a physical interaction state between a user and an object. According to an embodiment, the control unit 140 may determine an interaction state for an object. Here, the interaction state may be any one of a perceive state, an approach state, a manipulate state, and an avoid state. Each interaction state will be described in more detail below.

[0061] The control unit 140 may determine the type of physical object, and if the object is a target object, select one of a recognition state, an approach state, and an operation state. If the object is a non-target object, select one of a recognition state and an avoidance state. Here, according to an embodiment, the control unit 140 may determine an object other than a physical object determined as a target object as a non-target object. For example, when a user extends his / her hand, the control unit 140 may determine an object located in the direction of the hand as a target object, or may track the movement of the user's pupils and determine an object the user is looking at based on the user's gaze direction as a target object.

[0062] FIG. 2 shows the interaction state flow for each of the target and non-target objects.

[0063] As shown in FIG. 2, a target object can be classified into one of a "recognition state," an "approach state," and an "operation state." Here, the user's need for detailed information about a physical object increases in the order of the recognition state, approach state, and operation state. Meanwhile, the control unit 140 can classify non-target objects, which do not interact with the user but whose presence must be recognized in the virtual environment, into one of a "recognition state" and an "avoidance state." Here, the "avoidance state" improves the user's awareness of surrounding objects, enabling safe and efficient exploration in the virtual environment, particularly while operating the target object. Incidentally, as shown in FIG. 2, if the control unit 140 determines a non-target state as a recognition state and then determines a target object as an operation state, it can determine that the non-target object is in an avoidance state.

[0064] Meanwhile, according to an embodiment, the control unit 140 may determine an interaction state for an object. Here, the interaction state may be any one of a recognition state, an approach state, an operation state, and an avoidance state.

[0065] Among the interaction states, the "recognition state" refers to the user identifying a virtual object of a physical object in a virtual environment. Unlike the physical world (real world), where physical objects are immediately recognized, a virtual environment is asynchronous with the physical world, so the control unit 140 can match and provide a physical object with a virtual object in the virtual environment. This allows the user to deeply immerse themselves in the virtual experience, easily identify the corresponding physical object through the virtual object, and have a natural and intuitive interaction with the physical object.

[0066] According to the embodiment, the control unit 140 may determine that the state is in a "recognition state" if no movement of the user's hand (or finger) is detected. Here, the control unit 140 may identify physical objects from a camera view image capturing the real world using a known image extraction technique, select an image corresponding to the identified object, and render it as a virtual object. A technique for arranging and rendering a virtual object onto which a physical object is projected in a virtual environment may be known. For example, if a campsite is implemented as a virtual environment, a desk located in front of a user wearing the electronic device 100 and items on the desk may be implemented as respective virtual objects and rendered in the virtual environment.

[0067] Among the interaction states, the "approach state" is a state in which a user's movement to contact a physical object is detected, such as when the user reaches out to grasp the physical object. To ensure high usability, the control unit 140 may provide a gripping area in any area on the virtual object corresponding to the physical object. In this manner, when the approach state is identified, the control unit 140 may provide a gripping area to balance the need for usability support to maintain presence and the provision of related physical world information (e.g., a gripping area).

[0068] The control unit 140 may determine that the state is approaching based on the movement of the user or the movement of the user's hand, and may determine that the state is approaching when the distance between the user and a physical object decreases due to the movement of the user (or the user's hand), or when the distance between the user's hand and a physical object decreases. The control unit 140 may also determine that the state is approaching if the distance between the physical object and the user or the user's hand is less than a predetermined value but greater than or equal to another threshold. Here, if the distance is determined to be less than the threshold, the control unit 140 may determine that the state is an operation state, as described below.

[0069] According to an embodiment, the control unit 140 can identify a gripping area on a physical object and generate a gripping area on a virtual object corresponding to the identified area to assist the user in safely gripping the physical object.

[0070] Among the interaction states, an "operation state" refers to a state in which a user operates a target object held by the user, and the nature of the operation may change depending on the user's purpose, the characteristics of the target object, and the surrounding circumstances of the target object in the virtual environment. Such operation states may be classified into a "simple operation state" and a "complex operation state." The control unit 140 may determine whether an interaction state with a physical object is a simple operation state or a complex operation state. The "simple operation state" refers to an interaction state related to a physical object that is less complex than a complex operation state. For example, a state that can be processed by displaying only the basic shape of the object and the object's gripping area may be determined to be a simple operation state. On the other hand, a "complex operation state" refers to a state related to a change in the state of the physical object itself, and may include a state related to a change in the state of the object, such as changing the liquid level in a bottle.

[0071] The control unit 140 may determine that the state is an operation state if it determines that the user has gripped the target object, or may determine that the state is an operation state if the distance between the user's hand and the physical object is less than a predetermined threshold. According to an embodiment, if the control unit 140 determines that the state is an operation state, it may further determine whether the state is a simple operation state or a complex operation state, and if a change in the state of the physical object is detected, it may determine that the state is a complex operation state, and if not, it may determine that the state is a simple operation state. According to yet another embodiment, the control unit 140 may immediately determine whether the state is a simple operation state or a complex operation state without determining whether the state is an operation state, and if a change in the state of the physical object is detected, it may determine that the state is a complex operation state, and if not, it may determine that the state is a simple operation state.

[0072] Among the interaction states, the "avoidance state" is a state for non-target objects and is activated when a user must navigate around a non-target object near a target object while manipulating a target object held by the user. Non-target objects require relatively simpler information (e.g., location and boundaries) than the target object with which the user interacts, and are therefore processed in a different interaction flow from the target object. That is, the control unit 140 distinguishes between target objects and non-target objects, and, as shown in FIG. 2, if an "operation state" is triggered for a target object, the control unit 140 may trigger an avoidance state for non-target objects located within a predetermined radius of the target object. A boundary box blending technique may be applied to non-target objects determined to be in the avoidance state. This allows the user to avoid collisions with non-target objects while maintaining a high sense of presence, thereby enabling the user to explore the environment more effectively.

[0073] According to an embodiment, the control unit 140 can determine an interaction state with a physical object, select a blending technique based on the determination result, and provide a virtual environment rendered based on the selected blending technique.

[0074] According to an embodiment, the control unit 140 may select various blending techniques, where the blending technique may be at least one of a virtual proxy, an affordance contour, a pass-through, and a boundary box.

[0075] Incidentally, Figure 3 is an example diagram for explaining the blending technique, and each of the physical objects, a storage box 310, a water bottle 320, and a glass 330, can be rendered in a virtual environment as virtual objects 311, 312, 313, 321, 322, 323, 331, 332, and 333 by applying the blending technique.

[0076] As a hybrid technique, the "virtual proxy" technique generates a virtual representation of a physical object while satisfying two key requirements: i) thematic consistency with the virtual environment, and ii) similarity in geometry and affordance within a primitive shape domain. For example, a physical hand cream bottle, having a cylindrical shape and a sealing function, can be rendered as a cylindrical motor oil container in a virtual environment embodied by a virtual house. Thematic consistency plays an important role in enhancing the presence of an object in a virtual environment. For example, in a VR space house scenario, physical objects on a desk must be represented by virtual proxies that fit the virtual space theme, such as robots or electrical devices. Therefore, if the virtual space theme is a VR space house, physical objects must be represented by virtual proxies that are consistent with the virtual space theme, such as robots or electronic components, rather than objects that conflict with the theme, such as a virtual curio box or camping lantern. To this end, the control unit 140 can select an object corresponding to the physical object from among the virtual objects already defined for the virtual space theme. To implement the virtual proxy technique, the control unit 140 may mirror the geometry and affordances of a physical object corresponding to the virtual object. For example, rendering a real-world electric fan as a large virtual toy car may confuse the user. Conversely, using a virtual proxy that accurately represents the shape of a physical object (e.g., a 3D reconstructed object) may facilitate user interaction but may not allow for the design of a diverse and immersive virtual environment. Therefore, to achieve a balance between high presence, usability, and creative design, an appropriate level of similarity between a physical object and its corresponding virtual proxy must be considered.Therefore, the control unit 140 can select and map a physical object to a virtual proxy having the same basic shape (e.g., cylindrical, cubic), size, and affordance (e.g., serving, drinking) as the physical object. When rendering a virtual object embodied by a virtual proxy in a virtual environment, the control unit 140 can apply a virtual proxy having the same size as the basic shape of the physical object. The control unit 140 can also provide a virtual object having the same affordance type as the physical object. The virtual proxy technique is a basic blending technique that integrates physical objects into a virtual environment and facilitates state recognition. The virtual proxy technique can provide a user with a high level of realism and convey basic information, such as the type and position of a physical object represented by a virtual object in a scene. According to an embodiment, the virtual proxy blending technique disclosed herein can be implemented in the same manner as a known virtual proxy technique that supports blending between virtual and reality.

[0077] For example, the control unit 140 may analyze the basic shape, size, and affordance of a physical object. These may be classified into six basic shapes: sphere, cylinder, cone, cube, pyramid, and torus. The control unit 140 may measure the size and basic shape area of ​​the physical object and select a virtual proxy that best matches the shape and affordance. That is, the control unit 140 may process a physical object with a virtual proxy. Therefore, as shown in FIG. 3, physical objects such as a storage box 310, a water bottle 320, and a glass 330 may be rendered in the virtual environment as virtual objects such as a wooden box 311, a vase 321, and a barrel glass 331 that may exist in a VR campsite, which is the theme of the virtual environment. Here, the objects may be rendered without areas for grasping or manipulating them. In this way, the control unit 140 can apply a virtual proxy that matches the basic shape, size, and affordance of a physical object using the virtual proxy technique, and as described below, when approaching or manipulating the object is required, the control unit 140 can provide a grasping or manipulation area for grasping or manipulating the object using the affordance contour technique.

[0078] Meanwhile, the "affordance contour" technique, a hybrid technique, supports interaction in both the approach state and the simple manipulation state. That is, when the interaction state is either the approach state or the simple manipulation state, the control unit 140 can overlay an affordance contour on an object processed as a virtual proxy of a physical object. That is, overlaying an affordance contour means overlaying a grasping area and a manipulating area on an object processed as a virtual proxy. The "grasping area" refers to a location where people intuitively grasp an object (e.g., a handle, knob, or bar), and the "manipulation area" refers to the orientation of a physical object and an interactive portion (e.g., an opening, lid, or button). The control unit 140 can visually process the grasping area or the manipulation area on the object processed as a virtual proxy by overlaying an affordance contour on the object processed as a virtual proxy of a physical object. If the approach state (or the manipulation state) is determined, the control unit 140 can apply the affordance contour technique. The control unit 140 can guide the user as to "where to reach out and grasp" by displaying 3D contours of the gripping and manipulation areas for a virtual object displayed using the virtual proxy technique through the affordance contour technique. For example, the control unit 140 can generate a mesh model using Blender and generate any one of a handle, a knob, a cover, a straw, a hole, a button, a pistol, and a lid for a physical object in a scenario. According to an embodiment, the control unit 140 can measure the relative positions of the gripping and manipulation areas on the virtual object based on information about the physical object acquired by a tracker attached to the physical object, map the measured positions, and map the gripping or manipulation areas to the virtual object embodied by the virtual proxy.For example, as shown in Fig. 3, by applying affordance contours to virtual objects 311, 321, and 321 displayed using the virtual proxy technique, it is possible to provide virtual objects with visually displayed gripping areas 312, 322, and 332. Simple interactions with everyday objects (e.g., opening, pulling, pushing, drinking, pouring, transferring, lifting, and holding) do not require detailed viewing of the shape of the object, so implementing affordance contours in a simple operation state reduces dependency on pass-through, thereby providing a better optimal sense of realism than when the pass-through technique is applied entirely in the operation state.

[0079] To simplify user interaction and reduce cognitive load, the control unit 140 may automate the triggering of affordance contours during frequently occurring interactions in approach states and simple manipulation states. That is, the control unit 140 may overlay an affordance contour on a virtual object corresponding to a physical object if the distance between the physical object and the user is less than a predetermined value, and may deactivate the applied affordance contour overlay if the distance is equal to or greater than a predetermined value. For example, the control unit 140 may trigger the affordance contour to apply the affordance contour to the virtual object if the physical object is located within a predetermined radius from the user's hand, and deactivate the affordance contour if the hand is located outside the radius or moves away from the physical object, thereby providing a smooth and immersive user experience.

[0080] In addition, the control unit 140 may display a virtual button for activating a pass-through (to be described later) by applying an affordance contour blending technique when in an approach state or an operation state.

[0081] As a hybrid technique, "pass-through" can be activated in a "complex manipulate state" that requires an accurate representation of the target object. In other words, pass-through is used only in complex manipulate states that require observation or detailed information about the deformation of a physical object, and affordance contours are used whenever possible. Pass-through allows the actual shape of a real-world object to be directly seen in a virtual environment. For example, the pass-through technique can be applied to a scenario in which a user pours water into a glass at a VR campsite or opens a can of Coke in a VR movie theater. Here, the control unit 140 can display a physical camera view of a physical object using the smallest ellipsoid shape that encompasses the virtual object to reduce presence loss. For example, the control unit 140 can search RGB frames that constitute a real-world image in real time, crop the smallest ellipsoid that encompasses the target physical object in the RGB frame, and overlay the cropped image on the corresponding virtual proxy.

[0082] The control unit 140 may trigger the pass-through blending technique automatically or in response to a user request. For example, a user may activate pass-through by pressing a virtual button displayed by the affordance contour blending technique. For example, when a user's hand approaches an object, a virtual button may be displayed on the virtual object, and when the user performs a gesture of pressing the virtual button, pass-through may be applied. Also, for example, when the pass-through technique is applied, if the user's hand is positioned outside a predetermined radius based on the area where the pass-through technique is applied for a set period of time, pass-through may be automatically deactivated, ending the interaction.

[0083] The boundary box blending technique is applied during an avoidance state among interaction states. The control unit 140 may apply the boundary box blending technique by overlaying the smallest cubic shape surrounding a virtual proxy for a non-target object to which the boundary box blending technique is applied. That is, the control unit 140 may surround the non-target object using the smallest possible translucent box, thereby maintaining the presence of the non-target object and allowing the user to easily avoid the non-target object as an obstacle. That is, the control unit 140 may apply the boundary box technique by generating and overlaying the smallest cubic shape surrounding the non-target object on a virtual proxy for the non-target object located near the target object. Here, the control unit 140 may generate boundary box lines according to known embodiments and apply the cubic shape to a translucent material in the same color as the lines. As shown in FIG. 3 , boundary boxes 313, 323, and 333 may be applied to surround virtual proxies 311, 321, and 331 corresponding to physical objects such as a storage box 310, a water bottle 320, and a glass 330. To display boundary boxes more naturally, in some embodiments, boundary boxes may be generated to surround virtual objects 312, 322, and 332 based on the affordance contours of the virtual objects. Non-target objects represented by boundary boxes may provide information about the boundaries of the non-target objects, thereby helping the user reach out to the target object without colliding with the protruding decorations of the non-target objects. This indicates that other physical objects located around the target object are nearby, improving the user's sense of immersion.If the control unit 140 determines that the user is moving a target object to the vicinity of a non-target object, for example, if it determines that there is a non-target object located within a predetermined radius of the target object, the control unit 140 can trigger a boundary box blending technique for the non-target object and automatically render a boundary box for the non-target object, and if the target object moves away from the non-target object to which the boundary box blending technique is applied, the application of the boundary box blending technique can be terminated.

[0084] As described above, the control unit 140 may determine an interaction state based on at least one of (i) the user's position (or the user's hand position), (ii) the physical object's position, and (iii) the virtual button's press state, and may select and trigger a hybrid technique based on the determination result. For example, if the control unit 140 determines that the distance between the user's hand and the target physical object is less than a predetermined value (e.g., 12 cm), the control unit 140 may determine that the interaction state is an approach state and a simple manipulation state, and may trigger an affordance contour. Furthermore, the control unit 140 may display a virtual button on the virtual proxy (or virtual object). Each time the user presses the virtual button, the control unit 140 may determine that the interaction state is a complex manipulation state and may trigger a pass-through. The control unit 140 may terminate the pass-through if the user's hand is outside the pass-through area for a specified time (e.g., 3 seconds). In addition, if the target object moves during the operation state, the control unit 140 measures the distance between the target object and the non-target object, and if the measured distance is less than a certain threshold value (e.g., 15 cm), the control unit 140 determines that the nearby non-target object is in an avoidance state and renders a boundary box for the non-target object.

[0085] 4 to 8 are diagrams illustrating a state in which a blending technique is applied based on each interaction state when a VR campsite is provided in a virtual environment.

[0086] For your information, Figure 4 shows an example of interacting with a single target object. Figure 4 illustrates the user's current actions over time. Figure 4 also shows the blending techniques 410 applied to the target object and 420 applied to non-target objects according to each user action.

[0087] 4, while the user is resting at a campsite, the control unit 140 may determine that there is no interaction 411, 421 with the target object and the non-target object, respectively, since it only needs to render and display the virtual environment. Since there is no interaction, the control unit 140 does not apply any blending technique.

[0088] Thereafter, if it is determined that the user recognizes the physical object and identifies the water bottle, the state may be determined as a recognition state. For example, the control unit 140 may determine the state as a recognition state if it acquires a camera view image. The control unit 140 may determine the state as a recognition state 412 for the target object and apply the virtual proxy technique, and may determine the state as a recognition state 422 for the non-target object and similarly apply the virtual proxy technique. For example, the control unit 140 may process a water bottle in the real world as a virtual proxy (e.g., a virtual vase) that matches the shape and affordance. Here, if it is determined that the user is reaching out to grab the water bottle, the control unit 140 may determine the state as an approach state 413 for the target object and apply the affordance contour. However, since the non-target object remains in the recognition state, the virtual proxy technique may be applied. If the user opens the lid of the water bottle they are holding to drink water, i.e., if the user opens the lid of the water bottle, the control unit 140 determines that the user is grasping the water bottle, determines the interaction state with the target object as a simple operation state 414, and may apply an affordance contour that displays a handle or lid (a gripping area or an operation area) that is useful for grasping and operating the water bottle. Here, since recognition of non-target objects around the target object needs to be improved during the process of lifting the water bottle, the control unit 140 may determine the non-target objects as an avoidance state 424 and apply a boundary box to the non-target objects. If the user then determines that they want to bring the water bottle closer to them and check the amount of liquid in the water bottle, the control unit 140 determines the interaction state as a complex operation state 415 and applies a pass-through technique to enable the user to check the amount of liquid in the physical water bottle. Here, because the user is now farther away from the non-target objects, the interaction state of the non-target objects changes to a recognition state 425, and therefore, a virtual proxy technique may be applied to the non-target objects.If the user does not move his / her body or hands after that, it is determined that the user is resting at the campsite again, and the control unit 140 may determine that no interaction has occurred and determine that there is no corresponding interaction state 416, 426.

[0089] 5 to 8 show virtual environments 520, 620, 720, and 820 provided based on real-world images 510, 610, 710, and 810 acquired through a camera mounted on an HMD when a user wears the HMD in which an electronic device according to the device disclosed herein is embodied. The virtual environments 520, 620, 720, and 820 are examples of virtual environments that appear on the screen of the HMD in which the device disclosed herein is embodied. Fig. 5 is a diagram for explaining a recognition state during an interaction state, Fig. 6 is a diagram for explaining an approach state during the interaction state, Fig. 7 is a diagram for explaining an operation state during the interaction state, and Fig. 8 is a diagram for explaining an avoidance state during the interaction state. According to the embodiment, the recognition state may be provided as a default, so that physical objects are mixed into the virtual environment as virtual proxies, allowing the user to recognize their presence. For example, if it is determined that the shapes of physical objects placed in front of the user are to be identified through a camera, i.e., if a real-world image 510 capturing the shapes of a water bottle 511, a drawer 512, a cup 513, and a drawer 514 placed on a desk 515 as shown in FIG. 5 is acquired, the control unit 140 may render and display each of the physical objects 511, 512, 513, 514, and 515 in a rendered virtual environment 520. To this end, a virtual proxy mixing technique may be applied to generate virtual objects 521, 522, 523, 524, and 525 corresponding to each physical object, which may then be rendered and displayed in the virtual environment. This allows the user to identify virtual objects corresponding to physical objects in the real world. That is, when the interaction state is the recognition state, the control unit 140 can integrate the physical object into the virtual environment using a virtual proxy (a virtual representation that reflects the shape, size, and function of the physical object) to help the user recognize the presence of the object. Here, the control unit 140 can overlay the virtual proxy according to the theme of the virtual environment, so that the water bottle 511, the drawer 512, the cup 513, the drawer 514, and the desk 515 can be rendered as a vase 521, a wooden box 522, a barrel cup 523, a camping storage box 524, and a camping tabletop 525, respectively.

[0090] When the virtual environment image 520 is provided and the control unit 140 determines that the user is reaching out to grasp the water bottle, that is, if the control unit 140 determines that the distance from the user's hand to the water bottle, which is a physical object, is within a predetermined range, the control unit 140 may determine that the interaction state with the water bottle, which is a target object, is an approach state. Of course, the control unit 140 may still determine that the interaction state with a non-target object is an awareness state. The real world image 610 of FIG. 6 is a real world image displayed when the user, having been provided with the real world image 510 of FIG. 5, approaches the water bottle 511. If the control unit 140 determines that the distance between the user's hand U and the water bottle 511 is within a predetermined range, the control unit 140 may provide a virtual environment 620 in which a handle 621 is rendered on the vase 521 by applying an affordance contour blending technique as shown in FIG. 6, thereby visually guiding the user to interactable parts. Here, the control unit 140 may analyze the real world image 610 and determine the position and shape of the handle 621 so that the handle 621 of the vase 521 corresponds to the shape of the handle of the water bottle 511. As mentioned above, visual assistance provided by the affordance contour blending technique allows users to perform simple operations such as precisely grasping and moving physical objects.

[0091] While the virtual environment image 620 is being provided, when the user turns his / her head, looks at the cup 513, reaches out his / her hand to grasp the cup 513, and lifts the cup upward (i.e., perpendicular to the desk), i.e., when the real world image 710 is acquired as shown in FIG. 7 , the control unit 140 can apply a mixed technique of affordance contours and boundary boxes to the virtual object. That is, when the user reaches out his / her hand toward the cup 513, the control unit 140 can provide the virtual environment 720 by determining that the user is in an approach state with respect to the cup 513 and applying the affordance contour to the barrel cup 523 corresponding to the cup 513 to render a handle 723 on the barrel cup 523. Furthermore, when the user grasps the cup 513, the control unit 140 can provide the virtual environment 720 by determining that the user is in a simple manipulation state with respect to the cup 513 and applying the affordance contour to the barrel cup 523 corresponding to the cup 513 to render a handle 723 on the barrel cup 523. Here, since the cup 513 is the target object, the storage box 512 and the drawer box 514 located on either side of the cup 513 are non-target objects, and if the distance between the target object and the non-target object becomes equal to or less than a predetermined distance, the control unit 140 determines that the non-target object is in an avoidance state and can apply a boundary box blending technique to the non-target object. That is, as shown in Fig. 7, the control unit 140 can provide a virtual environment 720 by rendering a boundary box 722 for the wooden box 522 corresponding to the storage box 512 and also rendering a boundary box 724 for the camping storage box 524 corresponding to the drawer box 514.

[0092] Thereafter, the user moves the glass 513, which was located at the rear between the storage box 512 and the drawer box 514, to the front between the storage box 512 and the drawer box 514. Then, as shown in the real-world image 810 of FIG. 8 , the user's movement is detected, and it is determined that the water bottle 511 held in the user's left hand is tilted, causing the water in the water bottle 511 to be poured into the glass. The control unit 140 determines that a complex operation state exists for the glass 513, which is the target object into which water is to be poured, and applies the pass-through technique to the barrel cup corresponding to the glass 513. Therefore, the control unit 140 can provide the virtual environment 820 by rendering a glass 823 in which the real-world glass 513 is projected at a position corresponding to the barrel cup, or by cropping the real-world glass 513 and inserting it at a position corresponding to the barrel cup. Thus, the user can confirm the liquid level when water is poured into the glass 513 located in the physical world. This allows for precise recognition of the target object. According to an embodiment, the user can activate the pass-through mode by, for example, clicking a button. For example, as in the above example, for objects with complex or dynamically changing shapes, such as a wine glass or plate that pours liquid or is fragile, the pass-through mode can be activated when the user clicks on a virtual button generated using the affordance contour technique. As described above, the device according to the embodiments disclosed herein can improve the quality of the user's virtual reality experience by helping the user have a high level of immersion in the virtual environment and assisting the user in performing necessary interactions with real-world objects.

[0093] FIG. 9 is a flow chart illustrating a method for supporting interaction between a virtual environment and the real world according to one embodiment.

[0094] The method according to the embodiment shown in Fig. 9 includes steps that are processed in time series by the device 100 shown in Figs. 1 to 8. Therefore, although the contents will be omitted below, the contents described above regarding the device 100 shown in Figs. 1 to 8 can also be applied to the method according to the embodiment shown in Fig. 9.

[0095] As shown in FIG. 9, the device 100 may determine an interaction state for a physical object in the real world (S910).

[0096] For example, the device 100 may determine the type of physical object, and if the object is a target object, select one of a recognition state, an approach state, and an operation state, and if the object is a non-target object, select one of a recognition state and an avoidance state.

[0097] Also, for example, the device 100 can determine whether the interaction state for the physical object is a simple operation state or a complex operation state among the operation states.

[0098] Also, for example, if the device 100 is determined to be in an operation state with respect to a physical object, it may determine that the device 100 is in an avoidance state with respect to non-target objects located within a predetermined radius based on the physical object.

[0099] The device 100 may then render and provide the virtual environment with the selected blending technique based on the interaction state (S920).

[0100] For example, the device 100 may apply any one of the blending techniques of virtual proxy, affordance contour, pass-through, and bounding box to render the virtual environment.

[0101] For example, the device 100 may overlay an affordance contour on an object that is a virtual proxy of a physical object when the interaction state is one of the approach state and the easy manipulation state.

[0102] Also, for example, the device 100 may overlay affordance contours on a virtual proxy representation of a physical object to visually represent a grasping or manipulation area on the virtual proxy representation of the object.

[0103] For example, the device 100 may overlay an affordance contour on a virtual object corresponding to a physical object if the distance between the physical object and the user is less than a predetermined value, and may deactivate the affordance contour overlay if the distance is greater than or equal to the predetermined value.

[0104] Also, for example, the apparatus 100 may apply a boundary box to non-target objects that are determined to be in an avoidance state.

[0105] In this way, the method disclosed herein can categorize physical object interactions in a virtual environment into four states (i.e., recognition, approach, manipulation, and avoidance) and dynamically apply various blending techniques. Furthermore, new blending techniques (i.e., affordance contours and boundary boxes) can be designed to accurately capture objects and maintain a high sense of presence while avoiding obstacles. Physical objects can also be classified as targets and non-targets, and different blending techniques can be applied depending on the classification.

[0106] Unlike conventional methods that focus only on simple interactions (e.g., keyboard typing, raising a cup) or use simple virtual-reality blending techniques (e.g., always applying a pass-through technique to all interactions), the embodiments disclosed herein can propose an "interaction state adaptive virtual-reality blending technique" that gradually integrates information about the real world into the virtual environment in accordance with various interaction states.

[0107] Typically, there is a trade-off between presence and usefulness, but the embodiments disclosed herein can improve both presence and usefulness.

[0108] The term "module" used in the above embodiments refers to software or hardware components such as FPGAs (field programmable gate arrays) or ASICs, and the "module" performs a certain function. However, the term "module" is not limited to software or hardware. A "module" may be configured to reside on an addressable storage medium or to execute one or more processors. Thus, by way of example, "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0109] The functionality provided within components and units may be combined into fewer components and units or separated into additional components and units.

[0110] Furthermore, the components and "units" may be implemented to implement one or more CPUs within a device or a secure multimedia card.

[0111] The method according to the embodiment described with reference to FIG. 9 may also be embodied in the form of a computer-readable medium storing computer-executable instructions and data. Here, the instructions and data may be stored in the form of program code, which, when executed by a processor, may generate a predetermined program module and perform a predetermined operation. Furthermore, the computer-readable medium may be any available medium accessible by a computer, including both volatile and nonvolatile media, and both separable and non-separable media. The computer-readable medium may also be a computer recording medium. The computer recording medium may include both volatile and non-volatile, separable and non-separable media embodied by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer recording medium may be a magnetic storage medium such as a hard disk drive (HDD) or solid-state drive (SSD), an optical storage medium such as a CD, DVD, or Blu-ray disc, or a memory included in a server accessible via a network.

[0112] 4 may be implemented as a computer program (or a computer program product) including computer-executable instructions. The computer program includes programmable machine instructions to be processed by a processor, and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, a machine language, or the like. The computer program may be recorded on any type of computer-readable recording medium (e.g., a memory, a hard disk, a magnetic / optical medium, or a solid-state drive (SSD)).

[0113] Therefore, the method according to the embodiment described with reference to Figure 4 can be implemented by executing the computer program as described above on a computing device. The computing device can include a processor, memory, storage device, a high-speed interface connecting to the memory and a high-speed expansion port, and at least a portion of a low-speed interface connecting to a low-speed bus and the storage device. Each of these components is connected to each other using various buses and can be mounted on a common motherboard or in other suitable manners.

[0114] Here, the processor may process instructions within a computing device. Such instructions may include instructions stored in a memory or storage device for displaying graphic information to provide a GUI (Graphical User Interface) on an external input and output device, such as a display connected to a high-speed interface. In other embodiments, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as appropriate. Also, the processor may be implemented as a chipset consisting of chips containing multiple independent analog and / or digital processors.

[0115] Also, memory stores information within a computing device. As an example, memory may be comprised of a volatile memory unit or collection thereof. As another example, memory may be comprised of a non-volatile memory unit or collection thereof. Memory may also be in other forms of computer-readable media, such as, for example, a magnetic or optical disk. The storage device can provide a large amount of storage space to a computing device. The storage device may be a computer-readable medium or a configuration that includes such a medium, such as a device in a Storage Area Network (SAN) or other configuration, and may be a floppy disk drive, hard disk drive, optical disk drive, tape drive, flash memory, or other similar semiconductor memory device or device array.

[0116] The above-described embodiments are merely illustrative, and those skilled in the art will understand that the above-described embodiments may be easily modified into other specific forms without changing the technical ideas or essential features of the above-described embodiments. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described as a single component may be implemented in a distributed form, and similarly, each component described as a distributed component may be implemented in a combined form.

[0117] The scope of protection sought by this specification is determined by the claims below rather than the above detailed description, and should be construed to include all modifications or variations derived from the meaning and scope of the claims and their equivalents. [Explanation of symbols]

[0118] 100 Electronic equipment 110 Input / output section 120 memory 130 Communications Department 140 Control Unit

Claims

1. 1. An electronic device for supporting interaction between a virtual environment and the real world, comprising: Memory and a control unit that determines an interaction state with the real-world physical object and provides a virtual environment with a selected blending technique based on a result of the determination.

2. 2. The device of claim 1, wherein the control unit determines a type of the physical object, and if the object is a target object, selects one of a recognition state, an approach state, and an operation state, and if the object is a non-target object, selects one of a recognition state and an avoidance state.

3. The device according to claim 1 , wherein the control unit determines whether an interaction state for the physical object is a simple operation state or a complex operation state among operation states.

4. The apparatus of claim 1 , wherein the blending technique is any one of a Virtual Proxy, an Affordance Contour, a Pass-Through, and a Boundary Box.

5. The device of claim 1 , wherein the control unit overlays an affordance contour on an object that is a virtual proxy of the physical object if the interaction state is one of a close-up state and an easy-operation state.

6. The device of claim 1 , wherein the control unit overlays affordance contours on a virtual proxy representation of the physical object to visually represent a grasping or manipulation area on the virtual proxy representation of the object.

7. 2. The device of claim 1, wherein the control unit overlays an affordance contour on a virtual object corresponding to the physical object if a distance between the physical object and the user is less than a predetermined value, and deactivates the affordance contour overlay if the distance is greater than or equal to a predetermined value.

8. The device of claim 1 , wherein the control unit determines a state of avoidance for non-target objects located within a predetermined radius of the physical object when the control unit determines a state of operation for the physical object.

9. The apparatus of claim 8 , wherein the control unit applies a boundary box to the non-target object determined to be in the avoidance state.

10. 1. A method for supporting interaction between a virtual environment and a real world, performed by an electronic device, comprising: determining an interaction state for the real-world physical object; and providing the virtual environment with a blended technique selected based on the determination.

11. The step of determining the interaction state comprises: determining a type of the physical object; selecting one of a recognition state, an approach state, and an operation state if the object is a target object, and selecting one of a recognition state and an avoidance state if the object is a non-target object.

12. The step of determining the interaction state comprises: The method of claim 10 , further comprising determining whether the interaction state for the physical object is a simple manipulation state or a complex manipulation state.

13. The step of providing a virtual environment includes: The method of claim 10 , further comprising overlaying an affordance contour on an object that represents the physical object as a virtual proxy if the interaction state is one of a close-up state and an easy-to-operate state.

14. The step of providing a virtual environment includes:

11. The method of claim 10, comprising overlaying affordance contours on a virtual proxy representation of the physical object to visually represent grasp or manipulation areas on the virtual proxy representation of the object.

15. The step of providing a virtual environment includes:

11. The method of claim 10, comprising overlaying an affordance contour on a virtual object corresponding to the physical object if a distance between the physical object and a user is less than a predetermined value, and deactivating the affordance contour overlay if the distance is greater than or equal to a predetermined value.

16. The step of determining the interaction state comprises: The method of claim 10 , further comprising determining, if the operation state is determined for the physical object, that the state is an avoidance state for non-target objects located within a predetermined radius based on the physical object.

17. The step of providing a virtual environment includes: The method of claim 16 , including applying a boundary box to the non-target object determined to be in an avoidance state.

18. A computer-readable recording medium having a program recorded thereon for executing the method according to claim 10.

19. A computer program stored on a medium for executing by an electronic device and for performing the method of claim 10.

Images