Interaction system for tracking user motion

Abstract

This interaction system comprises: a remote controller configured to point at a specific point of a display device, and to control an object displayed on the display device; and a host device disposed to be spaced apart from the display device or disposed to be included in the display device to recognize an operation performed by the remote controller pointing at the specific point of the display device. The host device obtains direction information of the remote controller pointing at the specific point, and selects or displays a specific object on a screen of the display device according to two-dimensional coordinates of the specific point and the direction information.

Description

Interaction system that tracks user motion

[0001] The present invention relates to an interaction system for tracking user motion. More specifically, the present invention relates to an interaction system for estimating the motion of a user holding a remote controller.

[0002] In an interaction system that tracks the position and direction of a user's motion, the position and direction of an object in a 3D environment can be acquired based on optical markers and photo detectors. The position and direction of the object acquired in the 3D environment can also be used to control the 3D object within the display.

[0003] However, even if the position and orientation of an object in 3D space is detected, there is no specific method for determining where and how to display the detected object in 3D space based on the position and orientation of the detected object. Therefore, only 2D or 3D objects can be controlled on the screen, and 3D objects cannot be controlled by connecting the virtual space within the display with the external real space.

[0004] Additionally, even when selecting a virtual 3D object in a 3D scene, there is no clear method presented on how to move and rotate the selected object within the 3D space.

[0005] The purpose of this specification is to provide an interaction system that estimates the motion of a user holding a remote controller.

[0006] The purpose of this specification is to provide a specific method for detecting the position and orientation of an object in a three-dimensional space and for determining at what point and in what manner the detected object in the three-dimensional space is to be displayed in a display area according to the position and orientation of the detected object.

[0007] The purpose of this specification is to provide a clear method for selecting a virtual 3D object in a 3D scene and moving and rotating the selected object within 3D space.

[0008] An interaction system for estimating user motion according to the present disclosure comprises a remote controller configured to point at a specific point of a display device and to control an object displayed on the display device; and a host device disposed separately from the display device or included in the display device and configured to recognize an action performed by the remote controller pointing at the specific point of the display device.

[0009] According to an embodiment, the host device obtains direction information of the remote controller pointing to the specific point, and selects or displays a specific object on the screen of the display device according to the two-dimensional coordinates of the specific point and the direction information. The host device detects pointing, selection, movement and rotation operations by the remote controller in the real space between the remote controller and the display device, and controls the operation of a three-dimensional object (3D object) placed in a virtual space inside the display device based on the direction information and detection of the operations.

[0010] According to an embodiment, the remote controller may include an IR camera configured to acquire images of IR markers displayed on one side of the host device; a wireless communication unit configured to exchange information by transmitting and receiving wireless signals with the host device; an input device configured to receive a user input for controlling a movement of the three-dimensional object; and a sensor module configured to perform 6-DoF tracking associated with the direction information.

[0011] According to an embodiment, the host device may obtain an actual physical size and resolution of the display device to calculate coordinates directed to the display device, and may obtain a difference value between the pose of the IR markers in the remote controller and the pose of the display device in the remote controller. The remote controller may obtain images of the IR markers through the IR camera, and then obtain synthesized 6-DoF pose information through the sensor module.

[0012] According to an embodiment, the host device compensates for a difference in attitude between the IR markers and the display device, and, based on the compensated difference in attitude, converts attitude information of the remote controller acquired based on the IR markers into attitude information based on the display device, and transforms 2D coordinate values ​​of the X and Y axes of the attitude information of the remote controller to suit the resolution of the display device and converts them into 2D coordinate values ​​within a screen of the display device.

[0013] According to an embodiment, the host device controls the remote controller to point to an arbitrary point of the display device at a specific time according to instructions of a guide displayed on the display device, and obtains size information of the display device based on control information of the remote controller and resolution information of the display device. The control information of the remote controller may include a distance from the remote controller to the display device, a position of a point pointed by the remote controller, and an angle from the remote controller to the display device.

[0014] According to an embodiment, the host device converts the remote controller, the display device, and the host device into a three-dimensional coordinate system of the same three-dimensional space based on the two-dimensional coordinate and the direction information, and calculates a point of contact where a light ray emitted from the remote controller passes through the three-dimensional space and the light ray meets the display device. The angle between the light ray and the plane of the display device at the point of contact is calculated to obtain an angle of incidence of the light ray, and the two-dimensional coordinate can be obtained from the position of the point of contact.

[0015] According to an embodiment, the host device may control an object to move out of the pointing area in a first background situation based on the pointing action from the input device, and may control the pointing area to be enlarged and displayed on the display device based on the brightness and width of the light when there is no object within the pointing area.

[0016] According to an embodiment, the host device detects an input action of spraying water or blowing wind following the pointing action in a second background situation different from the first background situation, and if there are second objects within the pointing area according to the input action, a specific object among the second objects that is the target of the pointing action can be controlled to move. The movement distance, movement speed, and direction of the specific object can be controlled based on the strength and amount of water or wind adjusted according to the input action.

[0017] According to an embodiment, in a third background situation where an obstacle is placed between the object and the remote controller, the host device may control an input operation to the object to be deactivated when the object is placed on a straight line connecting the remote controller and the obstacle, control the input operation to be activated when a part of the object moves out of the area where the obstacle is placed as the object moves, and control the input operation to be activated when a part of the object moves out of the area where the obstacle is placed on a straight line changed according to the movement of the remote controller.

[0018] According to an embodiment, the display device may include a first display and a second display disposed adjacent to the first display. The host device may control the three-dimensional object to be displayed on the second display based on a movement or rotation operation that causes the three-dimensional object to leave a first display area of ​​the first display after selecting the three-dimensional object displayed on the first display, and may control the three-dimensional object to be displayed as a hologram in real space by detecting an input that is pulled toward the front of the display device by the remote controller.

[0019] According to an embodiment, the interaction system may further include a glass device worn by a user holding the remote controller to display an AR object. The host device may control the three-dimensional object to be displayed as a hologram toward a front area in a lateral direction of the display corresponding to the real space based on a second rotation operation that displays the three-dimensional object at a different angle in the real space, and may control the three-dimensional object to be displayed as a hologram toward a front area of ​​the glass device corresponding to the real space based on a second movement operation that displays the three-dimensional object in the front area of ​​the real space.

[0020] According to an embodiment, the display device may include a first screen area on which another user's face is displayed and a second screen area on which a game screen is displayed at a lower portion of the first screen area. The screen displayed on the display device may be displayed in a first screen configuration in which the entirety of the other user's face is displayed and the game screen is displayed at a first inclination angle, or in a second screen configuration in which a part of the other user's face is displayed and the game screen is displayed at a second inclination angle greater than the first inclination angle. The host device may control the display device to display the first and second screen areas in the first screen configuration or the second screen configuration based on the user's gaze direction, and may control the specific object to move to a specific position on the game screen based on an action of selecting a specific object displayed on the game screen in the second screen configuration and a second action of moving the specific object.

[0021] According to an embodiment, the remote controller may select a first object placed in the real space of the front area of ​​the display device displayed through the hologram, the glass device, or the projector device. The host device may determine a first distance in a specific direction within the virtual space from a contact point on the display device based on a specific key input that causes the first object to move to the virtual space of the internal area of ​​the display device, display the first object within the virtual space based on the first distance in the specific direction, and display the first object within the virtual space so as to be rotated in the specific direction and moved by the second distance based on a user's motion to rotate in a specific direction and move by a second distance based on a straight line in the specific direction, and when the input for selecting the first object is released, control the first object to move in the direction of gravity within the virtual space.

[0022] According to an embodiment, when a specific object is selected from among a plurality of objects arranged in the virtual space by the remote controller in a screen layout for purchasing a plurality of items, the host device may control the specific object to be rotated on an enlarged screen based on a rotation operation that causes the specific object to be rotated in a horizontal axis direction and a vertical axis direction, based on a rotation operation that causes the specific object to be rotated about a center point of the specific object.

[0023] According to an embodiment, the host device may control light to be emitted from the reference device in the direction based on a selection operation in the direction toward the reference device placed in the virtual space, and may control the first object in the virtual space to be rotated and moved to coordinates where a second object associated with the first object in the virtual space is placed based on a rotation and movement operation in a predetermined direction while the first object in the virtual space is placed within a radiation area by the emitted light.

[0024] According to this specification, an interaction system for estimating the motion of a user holding a remote controller is provided.

[0025] According to the present specification, a specific method can be provided for detecting the position and direction of an object in a three-dimensional space, and for displaying the detected object in the three-dimensional space at a certain point in a display area and in what manner, depending on the position and direction of the detected object.

[0026] According to this specification, even when selecting a virtual 3D object in a 3D scene, a clear method can be provided on how to move and rotate the selected object within 3D space.

[0027] According to this specification, the coordinates of a 3D object can be extracted within a controller and then transmitted to a host device via a standard interface. Accordingly, a single controller can control multiple host devices.

[0028] According to this specification, a user experience can be provided that makes virtual space feel like real space by allowing users to control objects in 3D virtual space in a manner identical to reality. This enables the provision of diverse user experiences in metaverse or digital twin products.

[0029] Further scope of the applicability of the present invention will become apparent from the detailed description below. However, since various modifications and variations within the spirit and scope of the present invention will become apparent to those skilled in the art, it should be understood that the detailed description and specific examples, such as preferred embodiments of the present invention, are given by way of example only.

[0030] FIG. 1 illustrates a user interaction system for tracking user motion according to the present specification.

[0031] Figure 2 shows detailed configurations of a remote controller that interfaces with a host device.

[0032] Figure 3 illustrates a flowchart of an interaction method related to obtaining an incident angle associated with coordinates and direction vectors oriented within a screen of a display device.

[0033] FIG. 4 illustrates a flowchart of an interaction method associated with obtaining the actual size and resolution of a display device according to the present specification.

[0034] Figure 5 illustrates a flowchart of an interaction method associated with a method of obtaining actual size information of a display device according to guide information displayed on the display device.

[0035] Figure 6 illustrates an interaction system that obtains the size of a display device and performs coordinate transformation between multiple devices.

[0036] Figure 7 shows a flowchart of an interaction method for obtaining a point and direction of contact with a display according to the orientation of a remote controller according to the present specification.

[0037] Figure 8 illustrates a conceptual diagram for controlling the movement of a specific object in a virtual space by tracking the motion of a user in a real space through an interaction method according to the present specification.

[0038] FIG. 9 illustrates 3D UI operations of an interaction method associated with mixed reality according to the present specification implemented by user inputs.

[0039] Figure 10 illustrates control operations according to various backgrounds in a virtual space according to pointing input according to the present specification.

[0040] Figure 11 illustrates interaction methods applicable in an environment with objects and obstacles according to the present specification.

[0041] FIG. 12 illustrates a conceptual diagram of object movement within a virtual space in a structure having multiple display areas according to the present specification.

[0042] Figure 13 illustrates a screen configuration based on a user's gaze and pointing input in a user interaction method according to the present specification.

[0043] FIG. 14 illustrates a conceptual diagram of an interaction method for moving a 3D object through a hologram, glass device, or projector device according to the present specification.

[0044] Figure 15 illustrates a conceptual diagram of an interaction method applicable to a scenario of displaying and purchasing items in a virtual space according to the present specification.

[0045] FIG. 16 is a conceptual diagram of an interaction method applied to a scenario of moving an object by shining light within a virtual space according to the present specification.

[0046] The technology disclosed herein is applicable to a user interaction system that tracks a user's motion. However, the technology disclosed herein is not limited to this system and can be applied to any user interaction system, method, or device to which the technical principles of the technology can be applied.

[0047] It should be noted that the technical terms used in this specification are used merely to describe specific embodiments and are not intended to limit the present invention. Furthermore, unless specifically defined otherwise herein, the technical terms used herein should be interpreted as having a meaning generally understood by those skilled in the art to which the present invention pertains, and should not be interpreted in an excessively broad or narrow sense. Furthermore, if a technical term used herein is incorrect and does not accurately express the spirit of the present invention, it should be replaced with a technical term that can be correctly understood by those skilled in the art. Furthermore, general terms used herein should be interpreted according to their dictionary definitions or according to the context, and should not be interpreted in an excessively narrow sense.

[0048] Additionally, the singular expressions used herein include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "consist of" or "comprises" should not be construed to necessarily include all of the various components or various steps described in the specification, and should be construed to mean that some of the components or some of the steps may not be included, or that additional components or steps may be included.

[0049] In addition, the suffixes "module" and "part" used in this specification for components are given or used interchangeably only for the convenience of writing the specification, and do not have distinct meanings or roles in themselves.

[0050] Additionally, terms including ordinal numbers, such as "first," "second," etc., used herein may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."

[0051] Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components are given the same reference numbers and redundant descriptions thereof will be omitted.

[0052] Furthermore, when describing the present invention, detailed descriptions of related known technologies will be omitted if they are deemed to obscure the gist of the present invention. Furthermore, it should be noted that the attached drawings are intended solely to facilitate understanding of the spirit of the present invention and should not be construed as limiting the spirit of the present invention.

[0053] In this regard, FIG. 1 illustrates a user interaction system for tracking user motion according to the present specification. Referring to FIG. 1, the user interaction system may be configured to include a remote controller (100) and a host device (200). The user interaction system may further be configured to include a display device (300). The host device (200) may be configured to be included in the display device (300). As another example, the host device (200) may be configured to be placed outside the display device (300).

[0054] IR markers (210) may be displayed on one side of the host device (200), for example, the front surface. The remote controller (100) may include an IR camera (105) for recognizing the IR markers (210). The remote controller (100) is equipped with necessary additional input devices, for example, a key, a joystick, a trackball, etc. In addition, a wireless device such as Bluetooth (BT / BLE) for communicating with the host device (200) and a 6-axis motion sensor for sensor fusion during 6-DoF tracking are included. The 6-axis motion sensor may include an acceleration sensor and a gyro sensor.

[0055] The host device (200) can display a 3D virtual environment on the display device (300). In addition, the host device (200) can display 6-DoF information or coordinate information acquired from the remote controller (100) within the 3D virtual environment. The host device (200) may also be included within the display device (300).

[0056] IR markers (210) are information that serves as a reference for the remote controller (100) to calculate 6-DoF values ​​through the IR camera (105). Based on the distance / pattern / number between markers of the IR markers (210), the overall size and recognizable distance for estimating user motion can be determined. The IR markers (210) can be included in the host device (200) or can be included in the outer area of ​​the display device (300) or in a separate structure.

[0057] When the host device (200) is included in the display device (300), the IR markers (210) may be placed in the bezel area of ​​the display device (300) or on a separate stand. The shape of the IR markers (210) may be circular or polygonal, but is not limited thereto and may be any shape depending on the application. The arrangement structure of the IR markers (210) may be arranged in a predetermined cycle on the horizontal axis and the vertical axis or may be a structure shifted by a predetermined interval.

[0058] The display device (300) is a device that creates and displays a 3D virtual space and object. The IR camera (105) can be mounted on the remote controller (100). By ignoring external visible light through the optical filter and the IR camera (105), only the IR marker image can be extracted and acquired, thereby minimizing the amount of computation during 6-DoF tracking.

[0059] The point indicated by the remote controller (100) is the point where the straight line indicated by the remote controller (100) and the display device (300) meet when the remote controller (100) is directed toward the display device (300). At this time, a 3D virtual object within the display device (300) can be controlled through the 2D coordinates of the point indicated by the remote controller (100) and the direction vector pointed by the remote controller (100).

[0060] In this regard, the user interaction system according to the present specification may be configured to include a remote controller (100) and a host device (200). The remote controller (100) may be configured to point at a specific point of the display device (300) and control an object displayed on the display device (300). The host device (200) may be disposed spaced apart from the display device (300) or may be disposed as included in the display device (300). The host device (200) may recognize an operation performed by the remote controller (100) pointing at a specific point of the display device (300).

[0061] The host device (200) can obtain direction information of the remote controller (100) pointing to a specific point of the display device (300). The host device (200) can select or display a specific object on the screen of the display device (300) according to the two-dimensional coordinates and direction information of the specific point of the display device (300). The host device (200) can detect pointing, selection, movement, and rotation operations by the remote controller (100) in the real space between the remote controller (100) and the display device (300). The host device (200) can control the operation of a three-dimensional object (3D object) placed in a virtual space inside the display device (300) based on the detection of the direction information and operations.

[0062] A remote controller (100) that interfaces with a host device (200) may be equipped with a camera, an input device, and a plurality of sensors. In this regard, FIG. 2 illustrates detailed configurations of a remote controller that interfaces with a host device. Referring to FIG. 2, the remote controller (100) may be configured to include an IR camera (105), a wireless communication unit (110), an input device (120), a sensor module (130), and a control unit (140). The IR camera (105) may be configured to acquire images of IR markers displayed on one surface of the host device (200). The wireless communication unit (110) may be configured to exchange information by transmitting and receiving wireless signals with the host device (200). The wireless communication unit (110) may be configured to include an RFIC chip that transmits and receives wireless signals such as RF signals, and a processor chip (or modem chip) that processes wireless signals to acquire information.

[0063] The input device (120) may be configured to receive user input to control the operation of a three-dimensional object placed in a virtual space inside the display device (300). The sensor module (130) may be configured to perform 6-DoF tracking associated with direction information. The sensor module (130) may be configured to include an acceleration sensor and a gyro sensor. The control unit (140) may be operably coupled to the IR camera (105), the wireless communication unit (110), the input device (120), and the sensor module (130). The control unit (140) may be configured to control the operation of the IR camera (105), the wireless communication unit (110), the input device (120), and the sensor module (130).

[0064] Meanwhile, with regard to the motion control of a 3D object, a method for obtaining an incident angle associated with the coordinates and direction vectors oriented within the screen of the display device (300) will be described in detail with reference to the drawings. FIG. 3 shows a flowchart of an interaction method related to obtaining an incident angle associated with the coordinates and direction vectors oriented within the screen of the display device. Referring to FIGS. 1 to 3, each process of the interaction method can be performed by a remote controller (100) or a host device (200). The interaction method can include a size / resolution obtaining process (S210), a pose difference value obtaining process (S220), an IR-marker-based 6-DoF pose obtaining process (S230), a display-based 6-DoF pose obtaining process (S240), and a 2D coordinate / incident angle obtaining process (S250).

[0065] In order to calculate the coordinates directed to the display device (300) in the size / resolution acquisition process (S210), the actual physical size and resolution of the display device (300) can be acquired in advance. If the host device (200) is built into the display device (300), the manufacturer can store the corresponding values ​​in the host device (200) in advance when manufacturing the product, so that the corresponding values ​​can be used. If the host device (200) and the display device (300) are separated, the operation can be performed by selecting the model of the display device (300) or directly inputting a value. The operation of selecting the model of the display device (300) or directly inputting a value can be performed by the remote controller (100).

[0066] In the process of obtaining a difference in posture value (S220), a difference value between the posture of the IR markers (210) in the remote controller (100) and the posture of the display device (300) in the remote controller (100) can be obtained. The posture of the IR markers (210) in the remote controller (100) means the three-dimensional position of the IR markers (210) and the three-dimensional direction information from the remote controller (100) to the IR markers (210). The posture of the display device (300) in the remote controller (100) means the three-dimensional position of the display device (300) and the three-dimensional direction information from the remote controller (100) to the display device (300).

[0067] When the IR markers (210) are located outside the display device (300) in a modular form, it is possible to select whether the position of the IR markers (210) is at the top or bottom of the display device (300). Meanwhile, the difference in the 3D distance value can be inferred based on the information of the display device (300) acquired in the size / resolution acquisition process (S210). In this case, the 3D direction can be assumed to be the same between the IR markers (210) in the remote controller (100) and the display device (300) in the remote controller (100). When the IR markers (210) are built into the display device (300), the values ​​stored at the time of product manufacture can be read and used.

[0068] In the IR-marker-based 6-DoF pose acquisition process (S230), after acquiring images of IR-markers (210) through the IR-camera (105) of the remote controller (100), the 6-DoF pose value of the remote controller (100) can be acquired through the PnP Solver. At this time, the accuracy of the pose value of the remote controller (100) can be improved through sensor fusion using a 6-axis motion sensor including an acceleration sensor and a gyro sensor. The calculated 6-DoF pose information can be transmitted to the host device (200) through wireless communication such as Bluetooth or Wi-Fi.

[0069] A display-based 6-DoF posture acquisition process (S240) is performed through the posture difference value between the IR markers (210) and the display device (300) acquired in the posture difference value acquisition process (S220). The posture information of the remote controller (100) acquired based on the IR markers (210) can be converted into posture information based on the display device (300). In this regard, the conversion between posture information can be performed through a translate and rotate matrix operation.

[0070] A 2D coordinate / incident angle acquisition process (S250) can be performed using the attitude information of the remote controller (100) acquired in the display-based 6-DoF attitude acquisition process (S240). The 2D coordinate values ​​of the X and Y axes among the attitude information of the remote controller (100) can be transformed to 2D coordinate values ​​within the display device (300) to suit the resolution of the display device (300). The incident angle in the direction indicated by the remote controller (100) can be acquired through the orientation value among the attitude information of the remote controller (100).

[0071] The interaction method including the above-described size / resolution acquisition process (S210) to the 2D coordinate / incident angle acquisition process (S250) can be performed by the remote controller (100) and the host device (200). The host device (200) can acquire the actual physical size and resolution of the display device (300) in order to calculate the coordinates directed to the display device (300). The host device (200) can acquire the difference value between the pose of the IR markers (210) in the remote controller (100) and the pose of the display device (300) in the remote controller (100). The remote controller (100) can acquire the 6-DoF pose value after acquiring the image of the IR markers (210) through the IR camera (105). To this end, the remote controller (100) can acquire images of IR markers (210) and then acquire synthesized 6-DoF posture information through the sensor module (130).

[0072] The host device (200) obtains display-based 6-DoF attitude information through the attitude difference value between the IR markers (210) and the display device (300). The host device (200) compensates for the attitude difference value between the IR markers (210) and the display device (300). Based on the compensated attitude difference value, the host device (200) can convert the attitude information of the remote controller (100) obtained based on the IR markers (210) into attitude information based on the display device (300). The host device (200) can transform the 2D coordinate values ​​of the X and Y axes among the attitude information of the remote controller (100) to suit the resolution of the display device (300) and convert them into 2D coordinate values ​​within the screen of the display device (300).

[0073] Meanwhile, Fig. 4 is a flowchart of an interaction method related to obtaining an actual size and resolution of a display device according to the present specification. Referring to Fig. 4, the interaction method may include a process for determining whether a host device is built-in (S210a), a process for obtaining a display size / resolution (S211), a process for obtaining a resolution from EDID (S212), a process for determining whether a display size is included in EDID (S210b), and a process for obtaining display size information from EDID (S213). The interaction method may further include a process for determining whether a display model name is included in EDID (S210c), a process for obtaining an actual size based on a model name (S214), a process for obtaining a display size based on a guide (S215), and a process for transmitting resolution / size information (S215).

[0074] Referring to FIGS. 1 to 4, an interaction method related to obtaining the actual size and resolution of a display device will be described. In this regard, a host device (200) can obtain the size information and resolution of the display in the EDID (Extended Display Identification Data) format through a display interface. The display interface may be HDMI, DVI, or a display port. EDID essentially includes information on the resolution supported by the display device (300) and information on the currently selected resolution, but may optionally include actual size information and a model name.

[0075] In the process of determining whether the host device is built-in (S210a), it is determined whether the host device (200) is included in the display device (300) as a built-in structure. If the host device (200) is included in the display device (300) as a built-in structure, the process of obtaining the display size / resolution (S211) is performed. In the process of obtaining the display size / resolution (S211), the resolution information and the size information of the display device (300) stored in the host device (200) can be obtained.

[0076] If the host device (200) is not included in the display device (300) as a built-in structure, a resolution acquisition process (S212) is performed from EDID. The host device (200) can acquire currently selected resolution information from EDID. In the display size inclusion determination process (S210b) in EDID, it is determined whether the actual size information of the display is included in EDID. If the actual size information of the display is included in EDID, a display size information acquisition process (S213) is performed from EDID. The host device (200) can acquire the actual size information of the display device (300) from EDID.

[0077] If the EDID does not include the actual size information of the display, a process (S210c) for determining whether the EDID includes the display model name is performed. In the process (S210c) for determining whether the EDID includes the display model name, it is determined whether the EDID includes the model name of the display device (300). If the EDID includes the model name of the display device (300), a process (S214) for obtaining the actual size based on the model name is performed. If the EDID includes the display model name, a process (S214) for obtaining the actual size based on the model name is performed. The host device (200) obtains the model name of the display device (300). A database (DB) for the actual size by model name can be constructed as its own DB by collecting display information disclosed by each display manufacturer on the web. The DB can be installed during the manufacturing of the host device (200) or can be searched through a network.

[0078] If the display model name is not included in the EDID, a guide-based display size acquisition process (S215) is performed. The actual size information of the display device (300) can be acquired according to the guide information displayed on the display device (300) according to the guide from the host device (200). In the resolution / size information transmission process (S215), the resolution and size information of the display device (300) can be transmitted from the host device (200) to the remote controller (100).

[0079] Meanwhile, according to the interaction method according to the present specification, actual size information of the display device (300) can be obtained according to guide information displayed on the display device (300) according to a guide on the host device (200). In this regard, FIG. 5 shows a flowchart of an interaction method related to a method of obtaining actual size information of the display device according to guide information displayed on the display device. Meanwhile, FIG. 6 shows an interaction system for obtaining the size of the display device and performing coordinate conversion between multiple devices. FIG. 6(a) shows a UI screen for obtaining the size of the display device under the guide of the host device. FIG. 6(b) shows coordinate conversion between IR markers of the host device, a remote controller, and the display device.

[0080] Referring to FIG. 5, the interaction method may be configured to include a marker-based size measurement process (S251), a host device-based size information acquisition process (S252), a display size input process (S253), and a mobile terminal-based size information acquisition process (S254). Referring to FIGS. 1 to 6, an interaction method for acquiring display size information will be described.

[0081] In this regard, the interaction method presents an input guide on the host device (200) for inputting the actual size of the display, and the user can input the size of the display according to the guide. In this regard, the size of the display device (300) can be estimated through the absolute coordinate information of the remote controller (100) and the resolution information of the display device (300). In the marker-based size measurement process (S251), the host device (200) can control the remote controller (100) to point to an arbitrary point of the display device (300) at a specific point in time according to the instructions of the guide displayed on the display device (300).

[0082] The host device (200) can obtain size information of the display device (300) based on the control information of the remote controller (100) and the resolution information of the display device (300). The control information of the remote controller (100) includes the distance from the remote controller (100) to the display device (300), the position of the point pointed by the remote controller (100), and the angle from the remote controller (100) to the display device (300).

[0083] In this regard, after displaying a guide phrase for measuring the size of the display, the display device (300) is sequentially pointed to the upper left (A), upper right (B), lower left (C), and lower right (D) at points spaced a certain distance from the center of the screen. In this regard, the phrase "I will measure the size of the display. After standing about 1 m away from the center, point the remote controller to the point indicated on the screen" may be displayed on a specific area (310) of the display device (300).

[0084] Meanwhile, when pointing to the upper left and upper right, the horizontal length of the display device (300) can be estimated using trigonometric functions based on the change angle and the observed Z value. When pointing to the lower left and lower left, the vertical length of the display device (300) can be estimated using trigonometric functions based on the change angle and the observed Z value. When pointing to the lower left and lower right, the horizontal length of the display device (300) can be estimated using the change angle. When pointing to the upper right and lower right, the vertical length of the display device (300) can be estimated using the change angle.

[0085] The horizontal and vertical lengths of the display device (300) are calculated using the average of two horizontal lengths of the estimated display device (300) and the average of two vertical lengths of the estimated display device (300). Thereafter, the calculated horizontal and vertical lengths of the display device (300) can be compared with standard display sizes to select the most similar display size.

[0086] In the host device-based size information acquisition process (S252), the horizontal and vertical sizes of the display photographed using the camera of the host device (200) are acquired. If the host device (200) is equipped with a depth camera, the host device (200) is held at a certain distance in front of the display device (300) and the entire area of ​​the display is photographed. Thereafter, the horizontal and vertical sizes are acquired by detecting corners, curves, and planes of the display using a depth map. For example, an image filter that finds corners for the depth map is applied to acquire x, y, and z values ​​for four corners, and the horizontal and vertical sizes are acquired based on the distance information of each corner.

[0087] In the display size input process (S253), the user can input the display size directly to any device using a remote controller (100) or a separate user terminal (e.g., a mobile terminal) according to the guide of the host device (200). As an example of the input method, an operation input guide of the remote controller (100) is displayed through the host device (200). Thereafter, the display size can be input using the joystick or trackball of the remote controller (100).

[0088] In the mobile terminal-based size information acquisition process (S254), the horizontal and vertical dimensions of the display can be measured using a length measurement application program on the mobile terminal. Size measurements can be made by photographing the display itself using a length measurement application program (e.g., AR Ruler) on a mobile terminal equipped with an RGB camera, depth camera, or inertial measurement unit (IMU). Alternatively, the horizontal and vertical dimensions of the display can be measured based on coordinates using a length measurement application program on the terminal.

[0089] Meanwhile, an interaction method according to the present specification, such as controlling an object within a virtual space, can be implemented by acquiring the point and direction of contact with a display based on the orientation of the remote controller. In this regard, Fig. 7 illustrates a flowchart of an interaction method according to the present specification for acquiring the point and direction of contact with a display based on the orientation of the remote controller.

[0090] Referring to Fig. 7, the interaction method may include a coordinate conversion process (S260), a controller ray progression process (S270), a display and contact point calculation process (S280), and an incident angle acquisition process (S290). The interaction method may further include a 2D coordinate acquisition process (S290b). Hereinafter, an interaction method for acquiring a point and direction of contact with a display according to the orientation of a remote controller will be described with reference to Figs. 1 to 7.

[0091] In the coordinate conversion process (S260), the host device (200) can convert the remote controller (100), the display device (300), and the host device (200) into a three-dimensional coordinate system of the same three-dimensional space based on the direction information and the two-dimensional coordinates. Here, the direction information is the direction information of the remote controller (100) pointing to a specific point. In addition, the two-dimensional coordinates are the coordinates of the contact point (P) where the light direction of the remote controller (100) pointing to a specific point meets the display device (200).

[0092] In this regard, in the display-based 6-DoF posture acquisition process (S240) of FIG. 3, the 6-DOF posture of the display can be acquired using the posture of the display and the posture of the remote controller (100) based on the IR markers (210). In the coordinate transformation process (S260), each of the IR markers (210), the display device (300), and the remote controller (100) can be transformed into the same 3D coordinate system through a Translate & Rotate Matrix operation. For example, by defining the upper left corner of the display device (300) as the origin (O) of the coordinate system (X, Y, Z), coordinate transformation is possible among the IR markers (210), the display device (300), and the remote controller (100).

[0093] In the controller ray progression process (S270), the host device (200) can march a ray emitted from the remote controller (100) in a three-dimensional space. The three-dimensional ray can be marched using the position of the remote controller (100) and the vector value of the front direction within the three-dimensional coordinate system.

[0094] In the display and contact point calculation process (S280), the host device (200) can calculate the contact point where the light radiated from the remote controller (100) meets the display device (300). In this regard, the contact point where the light ray meets the plane of the display within the same three-dimensional coordinate system can be calculated.

[0095] In the incident angle acquisition process (S290), the host device (200) can obtain the incident angle (q) of the light by calculating the angle between the light ray and the display plane of the display device (300) at the contact point. In the 2D coordinate acquisition process (S290b), a 2D coordinate can be acquired from the position of the contact point within the display plane of the display device (300). Specifically, the 2D coordinate of a specific object to be displayed on the display device (300) can be acquired based on the position of the corresponding contact point within the display plane. If the upper left corner of the display is the origin of the coordinate system, the x and y coordinate values ​​excluding the z-axis value among the 3D coordinates at the corresponding contact point can be used as is.

[0096] Meanwhile, the interaction method according to the present specification can track the motion of a user in a real space and control the motion of a specific object in a virtual space. In this regard, Fig. 8 illustrates a conceptual diagram for controlling the motion of a specific object in a virtual space by tracking the motion of a user in a real space through the interaction method according to the present specification. Referring to Fig. 8, the implementation of a 3D UI of mixed reality of real space and virtual space is possible. Real space, virtual space, and target objects will be described with reference to Figs. 1 to 8.

[0097] Real space (RS) refers to a 3D space where a real person, not a virtual one, exists. Real space can be defined as the space between the remote controller (100) and the display device (300). Real space can be defined as the space where a user holding the remote controller (100) exists. Virtual space (VS) refers to a 3D space that is not a real space but is drawn through rendering within the display device (300). Virtual space can be defined as the space inside the display device (300).

[0098] A target object (320) refers to a 3D virtual object that exists virtually within a virtual space. The contact point exists on the screen of the display, which is the surface where the real space and the virtual space meet. The contact point refers to the point where a point in the real space that a user points to via a remote controller (100) and a straight line extending from that point into the virtual space meet. Through the contact point that meets the screen of the display device (300), a user in the real space (RS) can control the target object (320) in the virtual space.

[0099] Meanwhile, the 3D UI operations of the interaction method associated with mixed reality according to the present specification can be implemented by multiple user inputs. In this regard, FIG. 9 illustrates 3D UI operations of the interaction method associated with mixed reality according to the present specification implemented by user inputs. The user's motion of FIG. 9 is recognized by the host device (200) of FIGS. 1 and 6, so that the host device (200) can control the operation of an object within the virtual space of FIG. 8.

[0100] Referring to Fig. 9(a), when the remote controller (100) points at a 3D object (320) drawn on the screen of the display device (300), the 3D object (320) is drawn in the virtual space (VS). In this regard, the starting point and direction of a straight line toward the virtual space obtained through the angle of incidence of the straight line and the point of contact (P) pointed at in the real space (RS) can be drawn in the virtual space. The user can point at the desired object in the same manner as if shooting a laser pointer out of a window.

[0101] Referring to Fig. 9(b), the moment a desired 3D object (320) is selected (using a key, etc.), the distance and direction between the remote controller (100) and the 3D object (320) can be fixed as if the 3D object (320) is attached to the end of the bar. The user selects the 3D object (320) using a virtual control bar, and the distance of the virtual control bar can be fixed. Accordingly, the distance between the remote controller (100) and the contact point (P1) in real space is maintained constant. The distance of the virtual control bar can be determined as the sum of the distance in the virtual space at the time of selecting the 3D object (320) and the distance in the real space.

[0102] Referring to FIG. 9(c), when the 3D object (320) is moved while the 3D object (320) is selected, the position and direction of the 3D object (320) can be controlled as if the 3D object (320) is moved while attached to the end of the bar. When the position of the 3D object (320) is moved to the left, the distance between the remote controller (100) and the contact point (P2) in real space increases. Accordingly, the distance in virtual space decreases, and the direction of the 3D object (320) can be rotated in the virtual space as much as it is rotated in the real space. The object can be moved while the entire distance of the control bar is fixed according to the user's movement motion. The object can be moved in the virtual space with the same effect as picking up and moving an object with a pair of tongs in real space.

[0103] Referring to Fig. 9(d), when rotating an object (320), the selected coordinates are fixed, and only the rotation direction of the remote controller (100) can be changed to intuitively rotate the object (320). This is similar to the method used when tightening or loosening a bolt with a wrench. The angle at which the object (320) is positioned can be controlled by the rotation motion. With regard to the rotation motion, the Z-axis rotation, which has fewer restrictions on the movement of the joints, can be performed with the wrist, while the X / Y axes, which have many restrictions, can be replaced with a track ball.

[0104] Meanwhile, the 3D UI operations of the interaction method associated with mixed reality according to this specification can be used in conjunction with a specific scene for each of multiple user inputs. In this regard, FIG. 10 illustrates control operations according to various backgrounds within a virtual space based on pointing input according to this specification.

[0105] Referring to Fig. 10(a), when the remote controller (1000) points at a dark, deep-sea virtual space, the dark portion of the virtual space can be seen through a straight line of light obtained through an angle of incidence of a straight line with the point pointed by the remote controller (1000). The fish fleeing from the area illuminated by the light or the aquatic plants can be seen in detail. The brightness and width of the light in the virtual space can be adjusted using a trackball. The user can experience a sensation similar to illuminating a dark, deep-sea space with a flashlight.

[0106] Referring to Fig. 10(b), when the remote controller (1000) points at a virtual forest, water can be sprayed into the virtual space at an angle of incidence that is parallel to the point pointed by the controller. Plants in the sprayed area can be seen moving, and animals / insects hit by the water can be seen fleeing. The strength and amount of water can be controlled using a trackball. The user can experience the same sensation as if they were spraying water into the virtual forest through a lake.

[0107] Referring to Fig. 10(c), when the remote controller (1000) points at a virtual space, wind can be sent into the virtual space at an angle of incidence that is in a straight line with the point pointed by the remote controller (1000). If there are cherry blossoms or leaves in the direction of the wind, you can see them fluttering or shaking in the direction of the wind. The strength and amount of wind or the strength and amount of water can be controlled using the trackball.

[0108] Referring to FIGS. 1 to 10, 3D UI operations of an interaction method related to mixed reality according to the present specification implemented by user inputs are described. The host device (200) can control an object to leave the pointing area in a first background situation when there is an object within the pointing area in response to a pointing operation from an input device (120) of a remote controller (100). The first background situation may be a virtual space of a dark deep sea as described above, but is not limited thereto and may be changed according to the application. The host device (200) can control the pointing area to be enlarged and displayed on the display device (300) based on the brightness and width of the light when there is no object within the pointing area.

[0109] The host device (200) can detect an input action of spraying water or blowing wind following the pointing action in a second background situation that is different from the first background situation. As described above, the second background situation may be a virtual forest or a virtual space with flowers in bloom, but is not limited thereto and may be changed according to the application. The host device (200) can control a specific object that is a target of the pointing action among the second objects to move if there are second objects within the pointing area according to the input action. The host device (200) can control the movement distance, movement speed, and direction of the specific object based on the strength and amount of water or wind adjusted according to the input action.

[0110] Meanwhile, the interaction method according to this specification can be applied in an environment containing objects and obstacles. In this regard, Fig. 11 illustrates interaction methods applicable in an environment containing objects and obstacles according to this specification. The interaction method of Fig. 11 can be applied to a shooting game, but is not limited thereto, and can be applied to various application programs that have environments containing objects and obstacles.

[0111] Referring to FIGS. 11(a) to 11(c), a situation is shown where an obstacle (320b) is placed in front of an object (320) as shown in the upper region. Meanwhile, as shown in the lower region of FIGS. 11(a) to 11(c), the positions of the object (320) and the obstacle (320b) may be changed and displayed on the screen according to changes in the position and orientation angle of the remote controller (100).

[0112] Referring to Fig. 11(a), the object (320) is behind the obstacle (320b) and is located on a straight line with the remote controller (100), so the object (320) cannot be shot. Referring to Fig. 11(b), the remote controller (100) can be rotated or moved so as to be out of the straight line with the obstacle (320b). As the remote controller (100) rotates or moves, the aiming point and aiming angle of the remote controller (100) move to a position and an incident angle that can be on a straight line with the object (320). Referring to Fig. 11(c), the obstacle (320b) can be avoided as the remote controller (100) is pointed to a specific point (P). Therefore, the object (320) is placed on an extension line that the remote controller (100) aims at the specific point (P), and the object (320) can be shot by the remote controller (100).

[0113] In this regard, an interaction method applicable to an environment with objects and obstacles is described with reference to FIGS. 1 to 11. The host device (200) is capable of various motion controls in a third background situation in which an obstacle (320b) is placed between an object (320) and a remote controller (100).

[0114] The host device (200) can control the input operation to the object (320) to be deactivated when the object (320) is placed on a straight line connecting the remote controller (100) and the obstacle (320b) in the third background situation. The host device (200) can control the input operation to the object (320) to be activated when a part of the object (320) moves out of the placement area of ​​the obstacle (320b) as the object (320) moves. The host device (200) can control the input operation to the object (320) to be activated when a part of the object (320) moves out of the placement area of ​​the obstacle (320b) on a straight line changed according to the movement of the remote controller (100).

[0115] Meanwhile, the movement of objects within a virtual space according to the present specification can also be applied to a structure comprising multiple display areas. In this regard, FIG. 12 illustrates a conceptual diagram of object movement within a virtual space in a structure comprising multiple display areas according to the present specification.

[0116] Referring to FIG. 12(a), the remote controller (100) can select a three-dimensional object (320) drawn on the first display (301) and move the pointing direction of the remote controller (100) to the second display (302). With the distance and direction between the remote controller (100) and the three-dimensional object (320) fixed, the three-dimensional object (320) moves to the second display (302) through the contact point between the first display (301) and the second display (302). The three-dimensional object (320) can move from the internal area of ​​the first display (301) to the internal area of ​​the second display (302) by the rotational motion by the remote controller (100).

[0117] Referring to FIG. 12(b), when a three-dimensional object (320) selected on the second display (302) is pulled toward the user, the distance and direction between the remote controller (100) and the three-dimensional object (320) are fixed and pulled toward the user. When the distance between the remote controller (100) and the three-dimensional object (320) is located in the front area outside the first display (310), the three-dimensional object (320) is converted into a hologram and displayed on the first display (301). The three-dimensional object (320) can be moved from the inner area of ​​the first display (301) to the outer area of ​​the first display (302) by a movement operation toward the user by the remote controller (100).

[0118] Referring to FIG. 12(c), when a three-dimensional object (320) selected in a virtual space inside the display (300) is pulled toward the user, the distance and direction between the remote controller (100) and the three-dimensional object (320) are fixed and pulled toward the user. When the distance between the remote controller (100) and the three-dimensional object (320) is located in the front area outside the first display (310), the three-dimensional object (320) can be displayed on the display of the glass device (100b). The three-dimensional object (320) can be displayed on the display of the glass device (100b) by moving from the inside area of ​​the display (300) to the outside area of ​​the display (300) by the movement action toward the user by the remote controller (100).

[0119] Referring to FIGS. 1 to 12, an interaction method for moving an object within a virtual space in a structure having multiple display areas is described. A display device (300) may include a first display (301) and a second display (302) positioned adjacent to one side of the first display (301). The display device (300) may further include a third display (303) positioned adjacent to the other side of the first display (301).

[0120] The remote controller (100) can detect a movement or rotation motion that causes the three-dimensional object (320) displayed on the first display (301) to leave the first display area of ​​the first display (301) after selecting the object. The host device (200) that is linked with the remote controller (100) can control the three-dimensional object (320) to be displayed on the second display (302) based on the movement or rotation motion that causes the object to leave the first display area. The host device (200) can detect an input that is pulled in the front direction of the display device (300) by the remote controller (100) and control the three-dimensional object (320) to be displayed as a hologram in real space.

[0121] Meanwhile, Fig. 13 illustrates a screen configuration based on a user's gaze and pointing input in a user interaction method according to the present specification. Fig. 13(a) illustrates a first screen configuration based on a first gaze of the user. Fig. 13(b) illustrates a second screen configuration based on a second gaze of the user. The user's first gaze may correspond to a gaze looking up at the screen or a motion of rolling the trackball of the remote controller upward. The user's second gaze may correspond to a gaze looking down at the screen or a motion of rolling the trackball of the remote controller downward.

[0122] Referring to Figure 13(a), a first screen configuration is displayed on the display, in which the opponent's front face (330a) and the chessboard of the board game are displayed at a first inclined angle. It is as if the user is facing the opponent with the board game in the center. In the first screen configuration.

[0123] Referring to Fig. 13(b), when the user slightly lowers the screen or rolls the trackball of the remote controller (100) downward, the display switches to a second screen configuration that looks down on the chessboard and the opponent from above. In the second screen configuration, a portion of the other user's face (330b) is displayed, and the chessboard of the board game is displayed at a second inclined angle.

[0124] When the remote controller (100) points at a 3D chess piece placed on the chessboard of the display, a 3D chess piece in a straight line with the pointing point (P1) can be selected. The selected 3D chess piece can be moved to a new pointing point (P2) using the remote controller (100). When the user slightly raises the screen or rolls the trackball of the controller upward, the display switches to the first screen configuration facing the chessboard of the board game and the opponent, as shown in FIG. 13(a).

[0125] Referring to FIGS. 1 to 13, 3D UI operations of an interaction method related to mixed reality on a game screen with an opponent are described. A display device (300) includes a first screen area where another user's face is displayed and a second screen area where a game screen is displayed at a lower portion of the first screen area. The screen displayed on the display device (300) may be a first screen configuration in which a frontal face (330a), which is the entirety of the other user's face, is displayed and a game screen (340a) is displayed at a first incline angle. Alternatively, the screen may be displayed as a second screen configuration in which a part of the other user's face (330b) is displayed and a game screen (340b) is displayed at a second incline angle greater than the first incline angle.

[0126] The host device (200) can control the display device (300) to display the first and second screen areas in the first screen configuration or the second screen configuration based on the user's gaze direction. In the second screen configuration, based on an action of selecting a specific object displayed on the game screen (340b) and a second action of moving the object, the host device (200) can control the object to move to a specific location on the game screen (340b).

[0127] In this regard, the glass device (100b) may be configured to be worn by a user carrying a remote controller (100) to display an AR object. The host device (200) may control the three-dimensional object (320) to be displayed as a hologram based on a second rotation operation that causes the three-dimensional object (320) to be displayed at a different angle in real space. The host device (200) may control the three-dimensional object (320) to be displayed as a hologram toward a front area in the lateral direction of the display (300) corresponding to real space.

[0128] The host device (200) can control the three-dimensional object (320) to be displayed as a hologram based on a second movement operation that causes the three-dimensional object (320) to be displayed in a front area of ​​a real space. The host device (200) can control the three-dimensional object (320) to be displayed as a hologram toward the front area of ​​a glass device (100b) corresponding to the real space.

[0129] Meanwhile, the interaction method according to the present specification can perform an interaction of moving a 3D object through a hologram, a glass device, a projector device, etc. In this regard, FIG. 14 illustrates a conceptual diagram of an interaction method of moving a 3D object through a hologram, a glass device, a projector device, etc. according to the present specification.

[0130] Referring to Fig. 14(a), a first object (350) of a 3D block in a real space outside a display device (300) displayed through a hologram, a glass device (100b), or a projector device can be selected using a remote controller (100). Referring to Fig. 14(b), the first object (350) of the selected 3D block can be moved to a point (P) indicated by the remote controller (100) by moving the remote controller (100). The distance at which the first object (350) of the 3D block moves is controlled by a joystick or a button key. When the selection of the first object (350) of the 3D block is released, the 3D block (350) can be controlled to move in the direction of gravity.

[0131] Referring to FIGS. 1 to 14, an interaction method for moving a 3D object through a hologram, a glass device, or a projector device will be described. A remote controller (100) can select a first object (350) placed in a real space in a front area of ​​a display device (300) displayed through a hologram, a glass device (100b), or a projector device. Based on a specific key input that causes the first object (350) to move into a virtual space of an internal area of ​​the display device (300), a first distance in a specific direction within the virtual space from a contact point on the display device (300) can be determined. A host device (200) can display the first object (350) within the virtual space based on the first distance in the specific direction.

[0132] Based on a user's motion to rotate in a predetermined direction and move a second distance relative to a straight line in a specific direction, the host device (200) can control the first object (350) within the virtual space. Based on the user's motion, the host device (200) can display the first object (350) to be rotated in a predetermined direction and moved a second distance within the virtual space. When the input for selecting the first object is released, the host device (200) can also control the first object to move in the direction of gravity within the virtual space.

[0133] Meanwhile, the interaction method according to this specification can be applied to a scenario of displaying and purchasing items in a virtual space. In this regard, Figure 15 illustrates a conceptual diagram of an interaction method applicable to a scenario of displaying and purchasing items in a virtual space according to this specification.

[0134] Referring to Fig. 15(a), when an item is selected from a shopping list displayed in a virtual space (VS) using a key or the like, the item is focused, and the distance and direction between the item and the user are fixed by a virtual control bar. Referring to Fig. 15(b), when rotating the item, an intuitive and interactive user experience using a 3D image of the item can be provided by changing only the rotation direction of the remote controller (100) while keeping the selected coordinates fixed.

[0135] Referring to FIGS. 1 to 15, an interaction method applicable to a scenario of displaying and purchasing items in a virtual space will be described. When a specific object (360) is selected from among a plurality of objects arranged in a virtual space (VS) by a remote controller (100) in a screen layout where a plurality of items can be purchased, the host device (200) can display the specific object and the arrangement structure in which the specific object is arranged so as to be distinguished from other objects. The host device (200) can control the specific object (360) to be rotated around the center point of the specific object on an enlarged screen based on a rotation operation that causes the specific object (360) to be rotated in the horizontal and vertical axis directions. In this regard, the specific object (360) can be displayed on the screen so as to be rotated around the Z-axis, or the specific object (360) can be displayed on the screen so as to be rotated around the X-axis (or Y-axis), which is a horizontal axis.

[0136] Meanwhile, the interaction method according to this specification can be applied to a scenario in which an object is moved by shining light within a virtual space. In this regard, Fig. 16 illustrates a conceptual diagram of the interaction method applied to a scenario in which an object is moved by shining light within a virtual space according to this specification.

[0137] Referring to Fig. 16(a), a ship can be found and selected by using a key or the like by shining light on the lighthouse beacon in a dark virtual space. When a battle is selected, the distance and direction of the remote controller (100) and the ship can be fixed by a virtual control bar with the lighthouse as the axis.

[0138] Referring to Fig. 16(b), when moving the position of the boat, the rotation direction of the remote controller (100) around the lighthouse can be changed to move the boat to the position of a school of fish in the dark. The moment the boat is released, the fish can be caught using a key or the like.

[0139] Referring to FIGS. 1 to 16, an interaction method for moving an object by shining light in a virtual space is described. A host device (200) can control light to be emitted from a reference device (370) in a direction based on a selection operation in a direction toward a reference device (370) placed in a virtual space. The host device (200) can control a first object (380) in a virtual space to be rotated and moved while the first object (380) is placed within a radiation area by the emitted light. Based on a rotation and movement operation of the first object (380) in a predetermined direction, the first object (380) can be controlled to be rotated and moved to coordinates where a second object (380b) associated with the first object (380) is placed.

[0140] The above describes an interaction system that estimates user motion. The technical benefits of this system are as follows.

[0141] According to this specification, an interaction system for estimating the motion of a user holding a remote controller is provided.

[0142] According to the present specification, a specific method can be provided for detecting the position and direction of an object in a three-dimensional space, and for displaying the detected object in the three-dimensional space at a certain point in a display area and in what manner, depending on the position and direction of the detected object.

[0143] According to this specification, even when selecting a virtual 3D object in a 3D scene, a clear method can be provided on how to move and rotate the selected object within 3D space.

[0144] According to this specification, the coordinates of a 3D object can be extracted within a controller and then transmitted to a host device via a standard interface. Accordingly, a single controller can control multiple host devices.

[0145] According to this specification, a user experience can be provided that makes virtual space feel like real space by allowing users to control objects in 3D virtual space in a manner identical to reality. This enables the provision of diverse user experiences in metaverse or digital twin products.

[0146] The configuration that performs control on the interaction system that estimates the user's motion can be implemented as a computer-readable code on a program-recorded medium. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of the computer-readable medium include hard disk drives (HDDs), solid state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also includes media implemented in the form of carrier waves (e.g., transmission via the Internet). In addition, the computer may include a control unit of a terminal or vehicle, i.e., a processor. Therefore, the above detailed description should not be construed as limiting in all respects, but should be considered as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are intended to be included in the scope of the present invention.

Claims

1. In an interaction system that estimates user motion, A remote controller configured to point at a specific point on a display device and control an object displayed on said display device; A host device configured to recognize an action performed by a remote controller directed to a specific point of the display device, the remote controller being positioned separately from the display device or included in the display device; The above host device, Obtaining direction information of the remote controller aimed at the specific point, Selecting or displaying a specific object on the screen of the display device according to the two-dimensional coordinates and direction information of the specific point, Detecting pointing, selecting, moving and rotating actions by the remote controller in the real space between the remote controller and the display device, An interaction system that controls the movement of a three-dimensional object (3D object) placed in a virtual space inside the display device based on the detection of the above directional information and the above movements.

2. In paragraph 1, The above remote controller, An IR camera configured to acquire images of IR markers displayed on one side of the host device; A wireless communication unit configured to exchange information by transmitting and receiving wireless signals with the host device; An input device configured to receive user input for controlling movement of the three-dimensional object; and An interaction system comprising a sensor module configured to perform 6-DoF tracking associated with the above orientation information.

3. In paragraph 2, The above host device, To compute the coordinates directed to the display device, the actual physical size and resolution of the display device are obtained, Obtaining the difference value between the pose of the IR markers in the above remote controller and the pose of the display device in the above remote controller, The above remote controller, An interaction system that acquires images of the IR markers through the IR camera and then acquires synthesized 6-DoF pose information through the sensor module.

4. In paragraph 3, The above host device, Compensate for the difference in position between the above IR markers and the above display device, Based on the above compensated posture difference value, the posture information of the remote controller acquired based on the IR markers is converted into posture information based on the display device, An interaction system that converts 2D coordinate values ​​of the X and Y axes among the detailed information of the remote controller into 2D coordinate values ​​within the screen of the display device by transforming them to match the resolution of the display device.

5. In paragraph 3, The above host device, Controlling the remote controller to point to any point on the display device at a specific point in time according to the instructions of the guide displayed on the display device; Based on the control information of the remote controller and the resolution information of the display device, size information of the display device is obtained, An interaction system, wherein the control information of the remote controller includes a distance from the remote controller to the display device, a position of a point pointed at by the remote controller, and an angle from the remote controller to the display device.

6. In paragraph 1, The above host device, Based on the two-dimensional coordinates and the direction information, the remote controller, the display device and the host device are converted into a three-dimensional coordinate system of the same three-dimensional space, The light ray emitted from the remote controller is transmitted in the three-dimensional space and the point of contact where the light ray meets the display device is calculated. By calculating the angle between the light ray and the plane of the display device at the above contact point, the angle of incidence of the light ray is obtained, An interaction system that obtains the two-dimensional coordinates from the location of the above contact point.

7. In paragraph 2, The above host device, In response to the pointing action from the input device, if there is an object within the pointing area in the first background situation, the object is controlled to leave the pointing area, An interaction system that controls the pointing area to be enlarged and displayed on the display device based on the brightness and width of the light beam when there is no object within the pointing area.

8. In Article 7 The above host device, In a second background situation different from the first background situation, an input motion of spraying water or blowing wind following the pointing motion is detected, If there are second objects within the pointing area according to the above input operation, a specific object among the second objects that is the target of the pointing operation is controlled to move, An interaction system in which the movement distance, movement speed and direction of a specific object are controlled based on the strength and amount of water or wind adjusted according to the input action.

9. In paragraph 6, The above host device, In a third background situation where an obstacle is placed between the object and the remote controller, the input operation to the object is controlled to be disabled when the object is placed on a straight line connecting the remote controller and the obstacle. Controlling the input action to be activated when a part of the object moves out of the obstacle placement area as the object moves; An interaction system that controls the input operation to be activated when a part of the object moves out of the obstacle placement area on a changed straight line according to the movement of the remote controller.

10. In paragraph 2, The display device includes a first display and a second display positioned adjacent to the first display, The above host device, After selecting the three-dimensional object displayed on the first display, based on a movement or rotation operation that causes the three-dimensional object to leave the first display area of ​​the first display, the three-dimensional object is controlled to be displayed on the second display, An interaction system that detects an input pulled toward the front of the display device by the remote controller and controls the three-dimensional object to be displayed as a hologram in real space.

11. In paragraph 10, Further comprising a glass device worn by a user holding said remote controller to display an AR object; The above host device, Based on the second rotation operation that causes the three-dimensional object to be displayed at a different angle in the real space, the three-dimensional object is controlled to be displayed as a hologram toward the front area of ​​the side direction of the display corresponding to the real space, An interaction system that controls the three-dimensional object to be displayed as a hologram toward the front area of ​​the glass device corresponding to the real space based on a second movement operation that causes the three-dimensional object to be displayed on the front area of ​​the real space.

12. In paragraph 2, The display device includes a first screen area on which another user's face is displayed and a second screen area on which a game screen is displayed at a lower portion of the first screen area, The screen displayed on the display device is displayed in a first screen configuration in which the entire face of the other user is displayed and the game screen is displayed at a first inclined angle, or in a second screen configuration in which a part of the face of the other user is displayed and the game screen is displayed at a second inclined angle greater than the first inclined angle. The above host device, Controlling the display device to display the first and second screen areas in the first screen configuration or the second screen configuration based on the user's gaze direction; An interaction system that controls a specific object to move to a specific location on the game screen based on a second action of selecting and moving a specific object displayed on the game screen in the second screen configuration.

13. In paragraph 11, The above remote controller selects a first object placed in the real space in the front area of ​​the display device displayed through the hologram, the glass device or the projector device, The above host device, Based on a specific key input that causes the first object to move into the virtual space of the internal area of ​​the display device, a first distance in a specific direction within the virtual space is determined from a contact point on the display device, Displaying the first object within the virtual space based on the first distance in the specific direction; Based on a user's motion to rotate in a predetermined direction and move a second distance relative to a straight line in the specific direction, the first object is displayed to be rotated in the predetermined direction and moved a second distance within the virtual space. An interaction system that controls the first object to move in the direction of gravity within the virtual space when an input for selecting the first object is released.

14. In paragraph 2, The above host device, In a screen layout where multiple items can be purchased, when a specific object is selected from among multiple objects placed in the virtual space by the remote controller, the specific object and the layout structure in which the specific object is placed are displayed so as to be distinguished from other objects. An interaction system that controls a specific object to be rotated around a center point of the specific object on an enlarged screen based on a rotational motion that causes the specific object to be rotated in the horizontal axis direction and the vertical axis direction.

15. In paragraph 2, The above host device, Based on a selection operation in the direction toward a reference device placed within the above virtual space, light is controlled to be emitted from the reference device in the direction, An interaction system that controls a first object to rotate and move to coordinates at which a second object associated with the first object in the virtual space is placed based on a rotation and movement operation in a predetermined direction while the first object in the virtual space is placed within a radiation area by the light emitted.

Images