Device and method of displaying user interactive

Abstract

A user interactive display device and method are disclosed. The user interactive display device according to the same may include: a communication unit that receives a three-dimensional image including depth information from an external server; a display unit that displays the three-dimensional image; a sensing unit that recognizes a user position and a user gaze direction; and a control unit that sets a reference point in the three-dimensional image, and when at least one of the user position and / or the user gaze direction changes, reconstructs the three-dimensional image based on the reference point by reflecting the depth information corresponding to at least one of the user position and / or the user gaze direction, and then displays the result on the display unit.

Description

Technology Field

[0001] The present invention relates to a user interactive display device and method, and more specifically, to a user interactive display device and method that dynamically displays an image based on depth information. Background Technology

[0002] The positive effects of windows on patients with limited mobility have been proven through various studies (View through a window may influence recovery from surgery, May 1984, Science 224(4647):420-1, Roger S Ulrich). Accordingly, hospitals have windows of a certain size or larger. However, patients in hospital rooms have limited movement and always look outside through the same window, and as a result, can only see static images.

[0003] For patients with restricted mobility, conventional static images viewed through a window remain the same regardless of changes in their position or gaze. Consequently, patients are only exposed to static and monotonous images through the window. Currently, there are no display devices that interact with the user to dynamically display the screen in response to changes in position or gaze. Prior art literature

[0004] Paper: View through a window may influence recovery from surgery, May 1984, Science 224(4647):420-1, Roger S Ulrich The problem to be solved

[0005] The problem that the present invention aims to solve is to provide a user interactive display device and method that can create a virtual window that operates interactively with a real person by dynamically displaying a screen in interaction with the user's location or gaze, and thereby provide a new user experience identical to a real window. means of solving the problem

[0006] A user interactive display device according to one embodiment of the present invention may include: a communication unit that receives a three-dimensional image including depth information from an external server; a display unit that displays the three-dimensional image; a sensing unit that recognizes a user position and a user gaze direction; and a control unit that sets a reference point in the three-dimensional image, and when at least one of the user position and / or the user gaze direction changes, reconstructs the three-dimensional image based on the reference point by reflecting the depth information corresponding to at least one of the user position and / or the user gaze direction, and then displays the result on the display unit.

[0007] In the above user interactive display device, the reference point is set in correspondence with a predetermined object included in the three-dimensional image, and the control unit can reconstruct the three-dimensional image by changing the depth of the predetermined object in correspondence with at least one of the user position and / or the user gaze direction.

[0008] In the above user interactive display device, the reference point is set corresponding to a predetermined position of the three-dimensional image, and the control unit can reconstruct the three-dimensional image by moving the predetermined position in at least one of the left, right, up, and down directions within the three-dimensional image corresponding to at least one of the user position and / or the user gaze direction.

[0009] In the above user interactive display device, the depth information of the three-dimensional image is estimated by the external server based on a depth estimation algorithm, and the control unit can determine the depth order of each of a plurality of objects existing in the three-dimensional image by reflecting the depth information.

[0010] In the above user interactive display device, the control unit can arrange the plurality of objects in order from objects closer to the user to objects farther away, based on the depth order of each of the plurality of objects.

[0011] In the above user interactive display device, the control unit can convert the three-dimensional image into a three-dimensional mesh image by extracting a background hidden between the plurality of objects within the three-dimensional image and expanding the hidden background through prediction.

[0012] A user interactive display method according to another embodiment of the present invention may include: receiving a three-dimensional image including depth information from an external server; displaying the three-dimensional image; recognizing a user position and a user gaze direction; setting a reference point in the three-dimensional image; and, when at least one of the user position and / or the user gaze direction changes, reconstructing the three-dimensional image based on the reference point by reflecting the depth information corresponding to at least one of the user position and / or the user gaze direction, and then displaying it. Effects of the invention

[0013] According to an embodiment of the present invention, by showing different scenes depending on the user's eye position through an augmented reality display, the user can be given the feeling of a real window.

[0014] In addition, according to an embodiment of the present invention, various dynamic and interactive images can be provided to a user residing in a narrow space, and the expansion of psychological space can be provided through such virtual windows. Brief explanation of the drawing

[0015] FIG. 1 is a block diagram illustrating the configuration of a user interactive display device according to the present invention. FIG. 2 is a drawing for explaining an example of a virtual window displayed by a user interactive display device according to the present invention. FIG. 3 is a diagram illustrating an example of a three-dimensional image including depth information received by a user interactive display device according to the present invention. FIG. 4 is a diagram illustrating a method for a user interactive display device according to the present invention to display a dynamic image interacting with a user. FIG. 5 is an example of a user interactive display device according to the present invention displaying a dynamic screen to a user. FIG. 6 is a drawing showing a computing device according to an embodiment of the present invention. Specific details for implementing the invention

[0016] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0017] In this specification, redundant descriptions of identical components are omitted.

[0018] Furthermore, when a component is described in this specification as being 'connected' or 'connected' to another component, it should be understood that it may be directly connected to or connected to the other component, or that there may be other components in between. On the other hand, when a component is described in this specification as being 'directly connected' or 'directly connected' to another component, it should be understood that there are no other components in between.

[0019] Furthermore, the terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention.

[0020] Additionally, in this specification, singular expressions may include plural expressions unless the context clearly indicates otherwise.

[0021] Furthermore, in this specification, terms such as 'comprising' or 'having' are intended merely to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not excluding in advance the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0022] Additionally, in this specification, the term "and / or" includes a combination of the plurality of described items or any of the plurality of described items. In this specification, "A or B" may include "A," "B," or "both A and B."

[0023] In addition, detailed descriptions of known functions and configurations that may obscure the essence of the invention will be omitted in this specification.

[0024] FIG. 1 is a block diagram illustrating the configuration of a user interactive display device according to the present invention.

[0025] The user interactive display device (100) according to the present invention dynamically displays images based on the user's location or the user's gaze and can perform the role of a virtual window that interacts with the user.

[0026] A user interactive display device (100) according to the present invention may be configured to include a communication unit (110), a display unit (120), a detection unit (130), and a control unit (140).

[0027] The communication unit (110) can receive a three-dimensional image containing depth information from an external server. In this case, the depth information of the three-dimensional image can be estimated by the external server based on a depth estimation algorithm. However, according to the embodiment, the depth information may also be estimated by a user interactive display device (100).

[0028] The display unit (120) can display a three-dimensional image.

[0029] According to one embodiment, the three-dimensional image can be implemented as at least one of Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR).

[0030] The detection unit (130) can recognize the user's location and the user's line of sight.

[0031] To this end, the detection unit (130) may include a camera, a scanner, a light sensor, a distance measuring sensor, a proximity sensor, etc. For example, the detection unit (130) may include a camera and can recognize the user's location and the user's gaze direction by analyzing an image of the user acquired by the camera.

[0032] The control unit (140) can set a reference point in a three-dimensional image.

[0033] In this case, the reference point may be set corresponding to a specific object included in the 3D image or to a specific location in the 3D image. For example, the reference point may be set to a place, object, character, etc., such as a landmark that can generally attract the user's attention. In addition, the reference point may be set to a specific location, such as the center area or the right area of ​​the 3D image.

[0034] According to an embodiment, the reference point may be set by a preset or set in real time in response to changes in the surrounding environment.

[0035] When at least one of the user position and / or user gaze direction changes, the control unit (140) can reconstruct a three-dimensional image based on a reference point by reflecting depth information corresponding to at least one of the user position and / or user gaze direction, and then display it on the display unit (120).

[0036] According to one embodiment, the control unit (140) can set a reference point corresponding to a predetermined object included in the three-dimensional image. In this case, the control unit (140) can reconstruct the three-dimensional image by changing the depth of the predetermined object corresponding to at least one of the user position and / or the user's line of sight.

[0037] According to another embodiment, the control unit (140) can set a reference point corresponding to a predetermined position in a three-dimensional image. In this case, the control unit (140) can reconstruct the three-dimensional image by moving a predetermined position in at least one of the left, right, up, and down directions within the three-dimensional image, corresponding to at least one of the user position and / or the user's line of sight direction.

[0038] The control unit (140) can determine the depth order of each of the multiple objects existing in the three-dimensional image by reflecting depth information. In this case, the control unit (140) can arrange the multiple objects in order from objects closer to the user to objects farther away based on the depth order of each of the multiple objects.

[0039] The control unit (140) can convert a 3D image into a 3D mesh image by extracting a background hidden between multiple objects in a 3D image and expanding the hidden background through prediction.

[0040] FIG. 2 is a drawing for explaining an example of a virtual window displayed by a user interactive display device according to the present invention.

[0041] The user interactive display device (100) according to the present invention can display a virtual window that is dynamically displayed by interacting with a user. In this case, the virtual window can be implemented as at least one of a two-dimensional image, a three-dimensional image, augmented reality (AR), virtual reality (VR), and mixed reality (MR). For example, the user interactive display device (100) can display a virtual window implemented in augmented reality by overlaying a virtual object that interacts with the user onto a real image corresponding to an external environment.

[0042] According to one embodiment, a user interactive display device (100) may acquire an image corresponding to an external environment and configure it as a virtual window. To this end, the user interactive display device (100) may further include a camera (not shown) that captures the external environment. In this case, the camera (not shown) may capture the external environment at predetermined intervals or continuously capture the external environment while rotating by a predetermined angle.

[0043] Referring to FIG. 2, the user interactive display device (100) captures an external environment. The external environment image (200) is acquired in the form of a two-dimensional image. The user interactive display device (100) can transmit the acquired external environment image (200) to an external server for image processing. In this case, the processing load on the user interactive display device (100) is reduced, so it is acceptable for the user interactive display device (100) to have low performance and low specifications.

[0044] However, the present invention is not limited thereto, and according to an embodiment, the user interactive display device (100) may directly process the acquired external environment image (200). In this case, the user interactive display device (100) must be implemented with high performance and high specifications to provide processing performance for image processing.

[0045] FIG. 3 is a diagram illustrating an example of a three-dimensional image including depth information received by a user interactive display device according to the present invention.

[0046] A user interactive display device (100) can receive a three-dimensional image containing depth information from an external server. According to one example, the three-dimensional image may be composed of a three-dimensional mesh model.

[0047] When an external server receives an external environment image from a user interactive display device (100), it can estimate the depth of an object within the image based on a depth estimation algorithm and divide the boundary from the depth order information. According to one embodiment, the depth estimation algorithm may be a deep learning-based depth estimation algorithm.

[0048] An external server can convert an external environment image into a 3D mesh model by using the depth of objects to distinguish between near and far objects and by using a method that expands the occluded background through prediction.

[0049] In this case, the external server can transmit the converted 3D mesh model to the user interactive display device (100).

[0050] Meanwhile, according to an embodiment, image processing that converts an external environment image into a three-dimensional mesh model may be performed on a user interactive display device (100) rather than an external server.

[0051] Referring to FIG. 3, a converted three-dimensional mesh model (300) is shown. The three-dimensional mesh model (300) contains depth information, and boundaries are divided and partitioned based on depth order information between objects. Based on this three-dimensional mesh model (300), a three-dimensional space can be reconstructed and the generated space can be interacted with through a display viewed by a user.

[0052] FIG. 4 is a diagram illustrating a method for a user interactive display device according to the present invention to display a dynamic image interacting with a user.

[0053] Specifically, FIG. 4 illustrates a case where an image is dynamically displayed in response to the user's location.

[0054] According to one embodiment, a user interactive display device (100) can recognize the user's location through a detection unit (130). If the detection unit (130) is implemented as a camera, the user's location coordinates can be recognized by analyzing and comparing video frames that capture the user. A vector from the user's location coordinates to the user interactive display device (100) is calculated. Corresponding to the displacement of the calculated vector, a 3D mesh model received from an external server can be displayed at the same viewpoint as the user's viewpoint.

[0055] According to another embodiment, the user interactive display device (100) can recognize the direction of the user's gaze through the detection unit (130). If the detection unit (130) is implemented as a camera, a three-dimensional space is generated from video frames capturing the user to recognize the user's eye position coordinates, and a vector from the recognized eye position coordinates to the user interactive display device (100) is calculated. Corresponding to the displacement of the calculated vector, a three-dimensional mesh model received from an external server can be displayed at the same viewpoint as the user's viewpoint.

[0056] Referring to FIG. 4, the user moves from position A to position B. At position A, a direction vector (410a) is calculated from the user to the upper left (400_LU) of the user interactive display device (100), and a direction vector (420a) is calculated from the user to the lower right (400_RL) of the user interactive display device (100). Additionally, at position B, a direction vector (410b) is calculated from the user to the upper left (400_LU) of the user interactive display device (100), and a direction vector (420b) is calculated from the user to the lower right (400_RL) of the user interactive display device (100).

[0057] In this case, in response to a change in the displacement of the calculated direction vector, that is, when the direction vector from the user to the upper left (400_LU) of the user interactive display device (100) changes from 410a to 410b and the direction vector from the user to the lower right (400_RL) of the user interactive display device (100) changes from 420a to 420b, the image is displayed to match the view of the changed direction vector. For example, the image can be moved to match the view of the changed direction vector. In this case, the area newly displayed as the image is moved can be generated by prediction. Alternatively, the depth of the image can be adjusted in response to the view of the changed direction vector.

[0058] According to the present embodiment, the direction of movement of the user or the direction of movement of the user's gaze is recognized, and a virtual space can be represented as a view corresponding to the user's location or gaze direction.

[0059] FIG. 5 is an example of a user interactive display device according to the present invention displaying a dynamic screen to a user.

[0060] Specifically, the left screen (500_L) of FIG. 5 is when the user looks from the left, and the right screen (500_R) is when the user moves to the right and looks.

[0061] In Fig. 5, rather than the image being changed or composed, the user's position or gaze direction is recognized, and the image is moved based on the recognized position or gaze direction. That is, the image moves naturally according to the user's movement so that it feels like a connected space. In this case, even if the user gazes at the image while their position or gaze has shifted, they can get the same feeling as looking through an actual window.

[0062] Referring to FIG. 5, when the user looks at the user interactive display device (100) from the left, the user sees the left screen (500_L). In this state, when moving to the right, the user sees the right screen (500_R).

[0063] When the user moves from left to right, the image moves to the right in response to the user's movement, changing from the left screen (500_L) to the right screen (500_R). In this case, as the image moves to the right, an Expected Area that was previously invisible is newly displayed on the left side of the user interactive display device (100). The Expected Area can be predicted based on the image.

[0064] Meanwhile, in FIG. 5, the reference point is set in correspondence with a predetermined object within the image, namely the lake (510). Accordingly, the user interactive display device (100) can move the image in correspondence with the position or gaze direction of the user looking at the reference point, the lake (510).

[0065] As described above, according to embodiments of the present invention, the user interactive display device (100) can perform the function of an interactive virtual window by displaying a dynamic image that interacts with the user. This user interactive display device (100) can be applied to a single or double ward for severe patients, etc., and can be manufactured as a virtual window that displays dynamic augmented reality in the space of a patient who is unable to go out for a long period of time. In addition, it can be installed in a single-person household, etc., and can perform the role of an artwork in a home.

[0066] Meanwhile, a user interactive display method according to the present invention may include: receiving a three-dimensional image including depth information from an external server; displaying the three-dimensional image; recognizing a user position and a user gaze direction; setting a reference point in the three-dimensional image; and, when at least one of the user position and / or the user gaze direction changes, reconstructing the three-dimensional image based on the reference point by reflecting the depth information corresponding to at least one of the user position and / or the user gaze direction, and then displaying it.

[0067] FIG. 6 is a drawing showing a computing device according to an embodiment of the present invention.

[0068] The computing device (TN100) of FIG. 6 may be a user interactive display device (100) described in this specification.

[0069] In the embodiment of FIG. 6, the computing device (TN100) may include at least one processor (TN110), a transceiver (TN120), and a memory (TN130). Additionally, the computing device (TN100) may further include a storage device (TN140), an input interface device (TN150), an output interface device (TN160), etc. The components included in the computing device (TN100) may be connected by a bus (TN170) to communicate with each other.

[0070] The processor (TN110) can execute a program command stored in at least one of the memory (TN130) and the storage device (TN140). The processor (TN110) may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. The processor (TN110) may be configured to implement the procedures, functions, and methods described in relation to embodiments of the present invention. The processor (TN110) can control each component of the computing device (TN100).

[0071] Each of the memory (TN130) and the storage device (TN140) can store various information related to the operation of the processor (TN110). Each of the memory (TN130) and the storage device (TN140) may be composed of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (TN130) may be composed of at least one of read-only memory (ROM) and random access memory (RAM).

[0072] The transmitting and receiving device (TN120) can transmit or receive wired or wireless signals. The transmitting and receiving device (TN120) can be connected to a network to perform communication.

[0073] Meanwhile, embodiments of the present invention are not limited to being implemented only through the apparatus and / or methods described so far, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments of the present invention or a recording medium on which such program is recorded, and such implementation can be easily achieved by a person skilled in the art to which the present invention belongs based on the description of the embodiments described above.

[0074] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by a person skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention. Explanation of the symbols

[0075] 100: User interactive display device 110: Communication unit 120: Display unit 130: Detector 140: Controller

Claims

Claim 1 A user interactive display device comprises: a communication unit receiving a three-dimensional image including depth information from an external server; a display unit displaying the three-dimensional image; a sensing unit recognizing a user position and / or a user gaze direction; and a control unit setting a reference point in the three-dimensional image, and when at least one of the user position and / or the user gaze direction changes, changing the depth of at least one object included in the three-dimensional image based on the reference point using the depth information, determining the depth order among multiple objects based on the depth information, reconstructing the three-dimensional image, and then displaying it on the display unit. A user interactive display device comprising, wherein the control unit calculates a direction vector from the user to the upper left corner of the user interactive display device and a direction vector from the user to the lower right corner of the user interactive display device, and when the displacement of the direction vectors changes according to a change in at least one of the user position and / or the user gaze direction, moves the 3D image to match the view of the changed direction vectors, generates a newly expressed area by prediction as the 3D image is moved, and displays the 3D image including the newly expressed area on the display unit. Claim 2 A user interactive display device according to claim 1, wherein the reference point is set in correspondence with a predetermined object included in the three-dimensional image, and the control unit reconstructs the three-dimensional image by changing the depth of the predetermined object in correspondence with at least one of the user position and / or the user gaze direction. Claim 3 A user interactive display device according to claim 1, wherein the reference point is set corresponding to a predetermined position of the three-dimensional image, and the control unit reconstructs the three-dimensional image by moving the predetermined position in at least one of the left, right, up, and down directions within the three-dimensional image corresponding to at least one of the user position and / or the user gaze direction. Claim 4 A user interactive display device according to claim 1, wherein the depth information of the three-dimensional image is estimated by the external server based on a depth estimation algorithm, and the control unit determines the depth order of each of a plurality of objects existing within the three-dimensional image by reflecting the depth information. Claim 5 In paragraph 4, the control unit arranges the plurality of objects in order from objects closest to the user to objects farther away, based on the depth order of each of the plurality of objects, in a user interactive display device. Claim 6 A user interactive display device according to claim 4, wherein the control unit extracts a background hidden between the plurality of objects within the three-dimensional image and expands the hidden background through prediction, thereby converting the three-dimensional image into a three-dimensional mesh image. Claim 7 A user interactive display method comprises: receiving a three-dimensional image including depth information from an external server; displaying the three-dimensional image; recognizing a user position and / or a user gaze direction; setting a reference point in the three-dimensional image; and, when at least one of the user position and / or the user gaze direction changes, changing the depth of at least one object included in the three-dimensional image based on the reference point using the depth information, determining the depth order among multiple objects based on the depth information, and reconstructing and displaying the three-dimensional image; wherein the step of reconstructing and displaying the three-dimensional image comprises: calculating a direction vector from the user to the upper-left corner of the user interactive display device and a direction vector from the user to the lower-right corner of the user interactive display device; and, when the displacement of the direction vectors changes according to a change in at least one of the user position and / or the user gaze direction, moving the three-dimensional image to match the view of the changed direction vectors. A user interactive display method comprising: a step of generating a newly expressed region by prediction as the 3D image moves, and displaying the 3D image including the newly expressed region.

Images