Display device for providing virtual object through mirror display and control method thereof

Abstract

A display device is disclosed. The display device comprises a mirror display, a depth camera capable of capturing an image of an object reflected in the mirror display, and one or more processors configured to: obtain an image including an object through the depth camera to obtain depth information of the object; obtain skeleton information of the object on the basis of the depth information of the object; obtain shape information of the object on the basis of the obtained skeleton information; and match a virtual object with the object on the basis of the shape information of the object and shape information of the virtual object, and provide the matched virtual object through the mirror display.

Description

Display device providing virtual objects through mirror display and control method thereof

[0001] The present invention relates to a display device and a control method thereof, and more particularly, to a display device that provides a virtual object through a mirror display and a control method thereof.

[0002] Advances in electronic technology have led to the development and widespread adoption of various types of electronic devices. In particular, various types of electronic devices, including TVs, have recently become commonplace in everyday households. These devices are increasingly equipped with diverse functions to meet user needs.

[0003] For example, mirror displays, which offer both mirroring and display functions, can now offer a variety of services. For example, mirror displays can simultaneously reflect the user's image while displaying virtual objects with which the user can interact.

[0004] However, conventional mirror displays failed to realistically display virtual objects by considering the user's appearance. For example, there were limitations in providing users with the impression that the virtual object was located in the same space as the user and reflected on the mirror display, such as overlapping or penetrating the user's corresponding mirror image.

[0005] There has been a need for a method to 3D identify a mirror image corresponding to a user reflected on a mirror display and provide an effect whereby virtual objects feel like their mirrored image.

[0006] In order to achieve the above object, according to one embodiment of the present disclosure, a display device includes a mirror display, a depth camera capable of photographing an object reflected in the mirror display, and one or more processors that obtain an image including the object through the depth camera to obtain depth information of the object, obtain skeleton information of the object based on the depth information of the object, obtain shape information of the object based on the obtained skeleton information, and align the virtual object with the object based on the shape information of the object and the shape information of the virtual object and provide the result through the mirror display.

[0007] A method for controlling a display device including a mirror display according to one embodiment of the present disclosure includes the steps of: acquiring an image of an object reflected on the mirror display to acquire depth information of the object; acquiring skeleton information of the object based on the depth information of the object; acquiring shape information of the object based on the acquired skeleton information; and providing the virtual object to the object through the mirror display based on the shape information of the object and the shape information of the virtual object.

[0008] According to one embodiment for achieving the above-described object of the present disclosure, a computer-readable recording medium including a program for executing a method for controlling a display device including a mirror display, the method for controlling the display device includes the steps of: acquiring an image of an object reflected on the mirror display to acquire depth information of the object; acquiring skeleton information of the object based on the depth information of the object; acquiring shape information of the object based on the acquired skeleton information; and providing the virtual object to the object through the mirror display based on the shape information of the object and the shape information of the virtual object.

[0009] FIG. 1 is a drawing showing the characteristics of a mirror display according to one embodiment of the present disclosure.

[0010] FIG. 2 is a block diagram showing the configuration of a display device according to one embodiment of the present disclosure.

[0011] FIG. 3 and FIG. 4 are drawings for explaining the operation of a switchable mirror according to an example of the present disclosure.

[0012] FIG. 5 is a drawing for explaining a display device providing objects and virtual objects according to one embodiment of the present disclosure.

[0013] FIG. 6 is a drawing for explaining depth information according to one embodiment of the present disclosure.

[0014] FIG. 7 is a drawing for explaining a display device that obtains shape information based on skeleton information according to one embodiment of the present disclosure.

[0015] FIG. 8 is a diagram illustrating a neural network model for obtaining shape information according to an embodiment of the present disclosure.

[0016] FIG. 9 is a drawing for explaining a display device communicating with an external device according to one embodiment of the present disclosure.

[0017] FIG. 10 is a drawing for explaining a virtual object aligned with an object according to one embodiment of the present disclosure.

[0018] FIG. 11 is a drawing for explaining a display device that provides feedback according to one embodiment of the present disclosure.

[0019] FIG. 12 is a drawing for explaining a display device communicating with an external device according to one embodiment of the present disclosure.

[0020] FIG. 13 is a flowchart for explaining a method for controlling a display device according to an embodiment of the present disclosure.

[0021] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0022] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0023] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0024] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0025] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0026] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0027] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0028] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0029] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0030] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0031] FIG. 1 is a drawing showing the characteristics of a mirror display according to one embodiment of the present disclosure.

[0032] A display device (100) according to one embodiment of the present disclosure may be implemented as various types of mirror display devices that can be installed in various locations requiring a mirror and provide mirror functionality while transmitting information. Here, the term "mirror display" is a compound word of "mirror," meaning a mirror, and "display," meaning the work of visually expressing information.

[0033] As illustrated in FIG. 1, the display device (100) includes a mirror display (110), and the mirror display (110) may include a switchable mirror (111) and a display panel (112).

[0034] For example, it may include a display panel (112) that displays an image, and a switchable mirror (111) that is arranged on the front of the display panel (112) and has an adjustable reflectivity.

[0035] For example, a switchable mirror (111) may be implemented as a glass plate or a transparent plastic plate on which a metal thin film or a dielectric multilayer film is deposited to reflect a portion of the incident light and transmit the other portion, thereby providing a mirror function.

[0036] According to an example, the display panel (112) may include a display including a self-luminous element, or may include a non-luminous element and a backlight. For example, the display layer (112) may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc.

[0037] In the present disclosure, for convenience of explanation, the position of the user (1) is defined as the front of the display device (100) as shown in FIG. 1.

[0038] FIG. 2 is a block diagram showing the configuration of a display device according to one embodiment of the present disclosure.

[0039] According to FIG. 2, the display device (100) includes a mirror display (110), a depth camera (120), a memory (130), and one or more processors (140).

[0040] The mirror display (110) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc.

[0041] The mirror display (110) may also include a driving circuit, a backlight unit, etc., which may be implemented in the form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. In one example, a touch sensor that detects a touch operation in the form of a touch film, a touch sheet, a touch pad, etc. may be arranged on the front of the mirror display (110) so as to be implemented to detect various types of touch inputs. For example, the mirror display (110) may detect various types of touch inputs, such as a touch input by a user's hand, a touch input by an input device such as a stylus pen, and a touch input by a specific electrostatic material. Here, the input device may be implemented as a pen-type input device, which may be referred to by various terms such as an electronic pen, a stylus pen, an S-pen, etc. In one example, the mirror display (110) may be implemented as a flat display, a curved display, a flexible display that can be folded or / and rolled, etc.

[0042] Meanwhile, the mirror display (110) can be implemented as a display that provides a mirror function and a display function.

[0043] For example, a mirror display (110) can be implemented in a form in which a half mirror (or mirror film) is added to a conventional display panel. The display (210) illustrated in FIGS. 3A and 3B illustrates an LCD including a half mirror, which is an example of an implementation of the mirror display (110). The LCD, also called a liquid crystal display, operates on the principle that when light is generated from a backlight, the light is transmitted between molecules of the liquid crystal to obtain a desired image.

[0044] FIG. 3 and FIG. 4 are drawings for explaining the operation of a switchable mirror according to an example of the present disclosure.

[0045] A switchable mirror (111) according to an example of the present disclosure may be implemented in a form including a polarizer, upper glass, lower glass, and a reflective polarizer.

[0046] In one example, a liquid crystal (LC) layer may be formed between the upper and lower glass layers. The LC (liquid crystal) layer is in an intermediate state between a liquid and a crystal, and thus has a structure in which rod-shaped molecules (liquid crystal molecules) are aligned in one direction, similar to a solid crystal.

[0047] In one example, the polarizer may be implemented to transmit polarized light. In one example, the upper glass and the lower glass may be implemented using, but are not limited to, transparent conductive oxide (TCO) glass.

[0048] Fig. 3 illustrates a switchable mirror (111) when the voltage is turned off. When the voltage is turned off, the liquid crystal molecules maintain a vertical state, and the incident polarized light can be reflected along the reflection axis of the reflective polarizer (111-5) by passing through the LC (liquid crystal) layer (111-3). Accordingly, the switchable mirror (111) can be maintained in a mirror state.

[0049] For example, when electricity is not applied, the incident polarized light is reflected along the reflection axis of the reflective polarizer, so that the mirror function (hereinafter, Mirror state) of the switchable mirror (111) can be provided, and as shown in FIG. 4, when electricity is applied, the incident polarized light rotates 90 degrees and transmits along the reflection axis of the reflective polarizer (111-5), so that the switchable mirror (111) can be maintained in a transparent state (hereinafter, Clear state).

[0050] In addition, depending on the implementation example, a protective film that protects the polarizing plate, a film that classifies light from the polarizing plate, etc. may be further included.

[0051] Returning to FIG. 2, the depth camera (120) can be turned on and take pictures according to a preset event. The depth camera (120) can include a ToF (Time of Flight) camera sensor.

[0052] A ToF camera sensor may be a sensor that measures the distance (or depth) between the ToF camera sensor and the subject by irradiating a signal (e.g., near-infrared, ultrasound, laser, etc.) and measuring the time it takes for the irradiated signal to be reflected by the subject and received by the ToF camera sensor. In one example, one or more processors (140) may obtain depth information of the subject based on the distance between the ToF camera sensor and the subject.

[0053] The ToF camera sensor is an example, and is not limited thereto, and the depth camera (120) may of course include a Lidar sensor, a radar sensor, an ultrasonic sensor, an infrared sensor, etc.

[0054] The display device (100) may, of course, include various types of cameras in addition to the depth camera (120). For example, a camera may convert a captured image into an electrical signal and generate image data based on the converted signal. For example, the camera may convert a subject into an electrical image signal through a semiconductor optical element (CCD; Charge Coupled Device), amplify the converted image signal, convert it into a digital signal, and then perform signal processing. For example, the camera may include at least one of a general (or basic) camera, an RGB camera, and an ultra-wide-angle camera.

[0055] One or more processors (140) control the overall operation of the display device (100). Specifically, one or more processors (140) may be connected to each component of the display device (100) to control the overall operation of the display device (100).

[0056] For example, one or more processors (140) may be electrically connected to the mirror display (110) and the memory (130) to control the overall operation of the display device (100). One or more processors (140) may identify the distance between the ToF camera sensor and the subject through the depth camera (120), and may obtain depth information of the subject based on the identified distance. According to an example, the subject may include a user positioned in front of the display device (100).

[0057] For example, the display device (100) can reflect an image of an object (or a user) located in front of the display device (100) through a mirror function, and can acquire an image using the object as a subject through a depth camera (120). According to an example, the image includes the distance between the depth camera (120) and the subject, and one or more processors (130) can acquire depth information of the subject based on the image. Hereinafter, for the convenience of explanation, the subject will be collectively referred to as an object.

[0058] The memory (130) can store data required for various embodiments.

[0059] The memory (130) may be implemented in the form of memory embedded in the display device (100) or may be implemented in the form of memory that can be attached or detached from the display device (100) depending on the purpose of data storage. For example, data for driving the display device (100) may be stored in a memory embedded in the display device (100), and data for the expansion function of the display device (100) may be stored in a memory that can be attached or detached from the display device (100). Meanwhile, in the case of memory embedded in the display device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)). In addition, in the case of memory that can be detachably attached to the display device (100), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented.

[0060] As an example, the memory (130) may store a computer program including at least one instruction or instructions for controlling the display device (100).

[0061] For example, the memory (130) may store an image received from an external device (e.g., a source device), an external storage medium (e.g., USB), an external server (e.g., web hard), etc., i.e., an input image. Alternatively, the memory (130) may store an image (or depth information) acquired through a depth camera (120) provided in the display device (100). Here, the image may be a 2D video, but is not limited thereto.

[0062] For example, the memory (130) may store various information required for image quality processing, such as information, algorithms, image quality parameters, etc. for performing at least one of Noise Reduction, Detail Enhancement, Tone Mapping, Contrast Enhancement, Color Enhancement, or Frame Rate Conversion. In addition, the memory (130) may also store an intermediate image generated by image processing and an image generated based on depth information.

[0063] According to one embodiment, the memory (130) may be implemented as a single memory that stores data generated from various operations according to the present disclosure. However, according to another embodiment, the memory (130) may be implemented to include multiple memories that each store different types of data or each store data generated at different stages.

[0064] In addition, the memory (130) can store various data, programs, or applications for driving / controlling the display device (100). In addition, the memory (130) can include a user sensing module, a communication control module, a voice recognition module, a motion recognition module, a light receiving module, a display control module, an audio control module, an external input control module, a power control module, a voice database (DB), or a motion database (DB).

[0065] In the above-described embodiment, it has been described that various data are stored in the external memory (130) of the processor (140), but at least some of the above-described data may be stored in the internal memory of the processor (140) depending on the implementation example of at least one of the display device (100) or the processor (140).

[0066] One or more processors (140) may be composed of one or more processors.

[0067] One or more processors (140) can perform operations of the display device (100) according to various embodiments by executing at least one instruction stored in the memory (130).

[0068] The one or more processors (140) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (140) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (140) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0069] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0070] One or more processors (140) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (140) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0071] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0072] In embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0073] One or more processors (140) can obtain depth information from an input image. Here, the input image may be received from a depth camera (120) or may be received from an external device. The input image may include a still image, a plurality of consecutive still images (or frames), or a moving image. For example, the input image may be a 2D image. Here, the depth information may be in the form of a depth map. A depth map refers to a table containing depth information for each region of the image. The region may be divided into pixel units, or may be defined as a preset region larger than a pixel unit. For example, the depth map may be in the form of a grayscale value from 0 to 255, with 127 or 128 as a reference value, that is, 0 (or focal plane), and a value smaller than 127 or 128 is represented as a - value, and a larger value is represented as a + value. The focal plane's reference value can be arbitrarily selected between 0 and 255. A negative value indicates subsidence, while a positive value indicates protrusion. However, this is merely an example; a depth map can express depth with various values ​​based on various criteria.

[0074] For example, the processor (140) may process an input image and then obtain depth information based on the processed image. Here, the image processing may be digital image processing including at least one of image enhancement, image restoration, image transformation, image analysis, image understanding, image compression, image decoding, or scaling.

[0075] For example, one or more processors (140) may perform various preprocessing before acquiring depth information for an input image. However, for convenience of explanation, the input image and the preprocessed image are not distinguished and are referred to as images below.

[0076] One or more processors (140) according to an example of the present disclosure can obtain an image including an object through a depth camera and obtain shape information of the object through the obtained image.

[0077] FIG. 5 is a drawing for explaining a display device providing objects and virtual objects according to one embodiment of the present disclosure.

[0078] Referring to FIG. 5, a display device (100) according to an example of the present disclosure can provide a mirror function and a display function.

[0079] For example, the display device (100) can provide an object (1) to a user positioned in front of the display device (100) through a mirror function, and display a virtual object (2) through a display function.

[0080] For example, an object (1) may be located in the same space as the display device (100) (e.g., located in front of the mirror display (110)) and correspond to a thing or user reflected by the mirror display (110), and a virtual object (2) may correspond to a thing or user located in a different space from the display device (100), or a non-existent graphic object.

[0081] According to an example of the present disclosure, if an image acquired through a depth camera (120) is a 2D image, and one or more processors (140) identify the object (1) as only a 2D object based on the 2D image, the display device (100) cannot provide interaction between the object (1) and the virtual object (2).

[0082] Here, the interaction may include whether a collision (or contact) occurs between the object (1) and the virtual object (2) according to the shape of the object (1) and the shape of the virtual object (2), whether one of the object (1) and the virtual object (2) overlaps the other, or the arrangement relationship (or sorting order) of the object (1) and the virtual object (2) according to a first distance between the object (1) and the display device (100) and a second distance between the virtual object (2) and the display device (100).

[0083] For example, if one or more processors (140) identify the object (1) only as a 2D object, there is a problem in that interaction between the object (1) and the virtual object (2) cannot be provided by considering the depth information of the object (1) from a viewpoint other than the viewpoint through the depth camera (120).

[0084] For example, there is a problem in that one or more processors (140) cannot identify the shape (or surface) of an object (1) from a viewpoint other than the viewpoint through the depth camera (120), identify whether a collision (or contact) has occurred between the object (1) and the virtual object (2) according to the shape of the object (1), and provide feedback (e.g., vibration) according to the occurrence of the collision.

[0085] One or more processors (140) according to an example of the present disclosure can acquire an image including an object (1) through a depth camera (120) and acquire shape information of the object (1) through the image. One or more processors (140) according to an example can appropriately provide interaction between virtual objects (2) using the shape information of the object (1). Here, the shape information of the object (1) can include surface information of the object (1).

[0086] One or more processors (140) according to an example of the present disclosure can acquire an image of an object (1) reflected on a mirror display (110) through a depth camera (120), and acquire depth information of the object (1) based on the image.

[0087] According to an example, one or more processors (140) can obtain skeleton information of an object (1) based on depth information, and obtain shape information of the object (1) based on the skeleton information.

[0088] Here, the shape information of the object (1) may include depth information corresponding to each of multiple view points for the object (1), which can express the object (1) as a 3D object.

[0089] According to an example, one or more processors (140) can align a virtual object (2) with an object (1) based on shape information of an object (1) and shape information of a virtual object (2) and provide the result through a mirror display (110).

[0090] For example, one or more processors (140) can determine whether a part of the virtual object (2) touches (or collides with, or contacts) the object (1) based on shape information of the object (1) and shape information of the virtual object (2), and render the virtual object (2) so that a part of the virtual object (2) touches the object (1).

[0091] That is, one or more processors (140) can render the virtual object (2) so that a part of the virtual object (2) comes into contact with the object (1) according to the shape information of the object (1) so that the virtual object (2) does not overlap the object (1) (so that the virtual object (2) does not overlap the object (1)).

[0092] For example, one or more processors (140) can display a virtual object (2) as an image reflected by a mirror display (110) located in the same space as the display device (100).

[0093] Hereinafter, for convenience of explanation, a configuration in which one or more processors (140) render a virtual object (2) based on shape information of an object (1) and shape information of a virtual object (2) is referred to as a configuration in which a virtual object (2) is adjusted (aligned) to an object (1).

[0094] FIG. 6 is a drawing for explaining depth information according to one embodiment of the present disclosure.

[0095] According to the conventional method, an image acquired through a depth camera (120) is a 2D image, and one or more processors (140) can only acquire depth information (10) corresponding to the view point of the depth camera (120) based on the 2D image, and shape information of an object (1) including depth information corresponding to each of a plurality of view points for the object (1) cannot be acquired.

[0096] One or more processors (140) according to one embodiment of the present disclosure can obtain depth information (10) of an object (1) based on an image obtained through a depth camera (120).

[0097] According to an example, one or more processors (140) can obtain skeleton information (20) of an object (1) based on depth information (10).

[0098] According to an example, one or more processors (140) can obtain shape information (30) of the object (1) by predicting (estimating) the shape of the object (1) as a symmetrical structure (or cylinder) based on a skeleton (bone, frame).

[0099] For example, one or more processors (140) can divide the object (1) into a plurality of parts based on the depth information (10) and identify skeletons for each of the plurality of parts. For example, one or more processors (140) can identify the central axis of each of the plurality of parts as a skeleton. Then, one or more processors (140) can identify (or predict) the shape of each of the plurality of parts according to a symmetrical structure (or cylinder) based on the skeletons for each of the plurality of parts.

[0100] According to an example, one or more processors (140) can obtain shape information (30) of an object (1) including the shape of each of a plurality of parts.

[0101] Referring to FIG. 6, one or more processors (140) can obtain skeleton information (20) based on depth information (10) of an arm portion, and identify the shape of the arm portion according to a symmetrical structure according to the following mathematical expression 1 based on the skeleton.

[0102] [Mathematical Formula 1]

[0103]

[0104] Here, (x, y, z) is the depth information of the object (1) (or, depth information of the shadow area of ​​the object (1)) from a viewpoint other than the viewpoint of the depth camera (120) (e.g., shadow area), ((x0 to x n ), (y0 to y n ), (z0 to z n )) refers to the depth information of the object (1) obtained based on the image.

[0105] FIG. 7 is a drawing for explaining a display device that obtains shape information based on skeleton information according to one embodiment of the present disclosure.

[0106] Referring to FIG. 7, one or more processors (140) according to an example of the present disclosure may input an image or depth information (10) including an object (1) into a first neural network model to obtain skeleton information (20). According to an example, the skeleton information (20) may include a plurality of skeletons for each part included in the object (1).

[0107] According to an example, one or more processors (140) can identify the shape of each of a plurality of parts according to a symmetrical structure based on a plurality of skeletons for each part included in the skeleton information (20) obtained from the first neural network model.

[0108] According to an example, one or more processors (140) can obtain shape information (30) of an object (1) including the shape of each of a plurality of parts.

[0109] According to an example, the first neural network model may be a model trained to output skeleton information (20) corresponding to an image when an image is input, using a plurality of sample images and skeleton information corresponding to each of the plurality of sample images.

[0110] For example, the first neural network model may be a model trained to identify an object included in an image when an image is input, identify multiple parts included in the object, and identify a skeleton for each of the multiple parts.

[0111] FIG. 8 is a diagram illustrating a neural network model for obtaining shape information according to an embodiment of the present disclosure.

[0112] One or more processors (140) according to an example of the present disclosure may input an image acquired through a depth camera (120) into a second neural network model to acquire shape information (30) of an object included in the image.

[0113] A second neural network model according to an example of the present disclosure may be a Generative Adversarial Networks (GAN)-based model. A GAN is composed of a generator (G) that generates virtual data samples and a discriminator (D) that determines whether an input data sample is real data. A GAN refers to a machine learning model built through adversarial training between the generator and the discriminator. The generator (G) (hereinafter, the second neural network model) is a model that learns to minimize the difference (Loss) between the shape information (30) (i.e., depth information corresponding to each of a plurality of viewpoints) that it has generated (or predicted) and the actual shape of the object (1). The discriminator (D) is a model that identifies the difference value between the shape information (30) of the object (1) and the actual shape of the object (1).

[0114] However, this is just an example, and the second neural network model may be a NeRF (Neural Radiance Fields)-based model.

[0115] A NeRF-based model according to an example may be a model that generates an image including an object (1) from a new viewpoint (i.e., a viewpoint other than the viewpoint of the depth camera (120)) based on an image acquired by photographing the object (1).

[0116] For example, the display device (100) includes at least one depth camera (120), and when at least one 2D image having an object (1) as a subject is received through at least one depth camera (120), the NeRF-based model can generate an image including the object (1) at a new viewpoint or a 3D object corresponding to the object (1) based on the at least one 2D image.

[0117] For example, a NeRF-based model may be a model that learns a fully-connected network (FC) that predicts RGB values ​​and the density of an object when five-dimensional data including (x, y, z), which is the spatial location of an object (or subject), and (θ, Φ), which is the viewing direction (e.g., the viewpoint of a depth camera (120)) of the object, is input. Here, the density of an object is the inverse of transparency; when the density increases, the object becomes opaque (other objects located behind the object according to the arrangement relationship (or sorting order) cannot be seen), and when the density decreases, the object may become transparent.

[0118] The artificial intelligence-related function according to the present disclosure is operated through one or more processors (140) and memory (130) of the display device (100).

[0119] One or more processors (140) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an NPU (Neural Processing Unit), but are not limited to the examples of the processors described above.

[0120] CPUs are general-purpose processors capable of performing not only general calculations but also artificial intelligence calculations. Their multi-layered cache structure allows for the efficient execution of complex programs. CPUs are advantageous for serial processing, enabling organic linking of previous and subsequent calculation results through sequential calculations. General-purpose processors are not limited to the examples described above, except where specifically identified as CPUs.

[0121] A GPU is a processor designed for large-scale computations, such as floating-point operations used in graphics processing. It integrates a large number of cores to perform large-scale computations in parallel. In particular, GPUs may be advantageous over CPUs in parallel processing methods, such as convolution operations. Furthermore, GPUs can be used as coprocessors to supplement the functions of CPUs. Processors for large-scale computations are not limited to the examples described above, except in cases where they are specifically referred to as GPUs.

[0122] An NPU is a processor specialized in artificial intelligence computation using artificial neural networks, and each layer of the artificial neural network can be implemented in hardware (e.g., silicon). Since an NPU is designed specifically according to the company's specifications, it has less freedom than a CPU or GPU, but can efficiently process the AI ​​computations requested by the company. Meanwhile, as a processor specialized in artificial intelligence computation, an NPU can be implemented in various forms, such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), or a Vision Processing Unit (VPU). Except as specifically stated as an NPU, an AI processor is not limited to the examples described above.

[0123] Additionally, one or more processors (140) may be implemented as a System on Chip (SoC). In this case, the SoC may further include, in addition to one or more processors, a memory, and a network interface such as a bus for data communication between the processor and the memory.

[0124] When a plurality of processors are included in a SoC (System on Chip) included in a display device (100), the display device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the plurality of processors. For example, the display device (100) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors. However, this is merely an example, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor.

[0125] Additionally, the display device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in a single processor. In particular, the display device (100) can perform artificial intelligence operations, such as convolution operations and matrix multiplication operations, in parallel by utilizing multiple cores included in the processor.

[0126] One or more processors are controlled to process input data according to predefined operation rules or artificial intelligence models stored in memory (130). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0127] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0128] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), a neural recurrent network (NeRF), a deep Q-network (Deep Q-Network), and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0129] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0130] FIG. 9 is a drawing for explaining a display device communicating with an external device according to one embodiment of the present disclosure.

[0131] A display device (100) according to an example of the present disclosure may further include a communication interface, and the display device (100) may communicate with another display device (hereinafter, an external device (100')) through the communication interface.

[0132] It goes without saying that the communication interface can be implemented as various interfaces depending on the implementation example of the display device (100). For example, the communication interface can communicate with an external device (100'), an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc. According to one example, the communication interface can communicate with another electronic device, an external server, and / or a remote control device.

[0133] In particular, according to an example, one or more processors (140) can transmit shape information of an object (1) to an external device (100') and receive shape information of a virtual object (2) from the external device (100').

[0134] For example, an external device (100') may be located in the same space as the external device (100') and may obtain shape information of a user reflected by a mirror display (110') provided in the external device (100'). Subsequently, the external device (100') may transmit shape information of the user (or an object, etc.) to the display device (100). For convenience of explanation, the user reflected by the mirror display (110') provided in the external device (100') will be collectively referred to as a virtual object (2).

[0135] One or more processors (140) can control the mirror display (110) to display a virtual object (2) based on shape information of the virtual object (2) received from an external device (100').

[0136] According to an example, one or more processors (140) can display a virtual object (2) aligned with the object (1) based on shape information of the object (1) and shape information of the virtual object (2).

[0137] For example, one or more processors (140) can identify a first distance between an object (1) and a display device (100) based on shape information of the object (1). One or more processors (140) can identify a second distance between a virtual object (2) and a display device (100) based on shape information of the virtual object (2).

[0138] In one example, one or more processors (140) can position one of the object (1) and the virtual object (2) behind the other based on the first distance and the second distance.

[0139] For example, one or more processors (140) can position at least one portion (or a part of the virtual object (2)) among a plurality of portions included in the virtual object (2) in front of the object (1), and position the remaining portions (or the remainder of the virtual object (2)) in back of the object (1).

[0140] Meanwhile, one or more processors (140) transmit shape information of an object (1) to an external device (100'), and the external device (100') can, of course, align the object (1) with the virtual object (2) based on the shape information of the object (1) and the shape information of the virtual object (2) and provide it through a mirror display (110').

[0141] According to an example of the present disclosure, one or more processors (140) can display a virtual object (2) on a mirror display (110) so that a part of the virtual object (2) comes into contact with the object (1) based on shape information of the object (1) and shape information of the virtual object (2).

[0142] For example, one or more processors (140) may identify each of the object (1) and the virtual object (2) as a 3D object, not a 2D object, and may display the virtual object (2) so that a part of the virtual object (2) comes into contact with the object (1) based on surface information of the object (1), rather than overlapping the object (1) and the virtual object (2) (or displaying the virtual object (2) so as to pass through or penetrate the object (1)).

[0143] FIG. 10 is a drawing for explaining a virtual object aligned with an object according to one embodiment of the present disclosure.

[0144] Referring to FIG. 10, one or more processors (140) may display a virtual object (2) by aligning it with (or adjusting it to) the object (1) based on shape information of the object (1) and shape information of the virtual object (2) received from an external device (100'), rather than displaying the virtual object (2) by simply covering the object (1) reflected by the mirror display (110) without adjustment.

[0145] For example, one or more processors (140) can obtain a 3D object corresponding to the object (1) based on depth information corresponding to each of a plurality of viewpoints for the object (1) included in the shape information of the object (1). The 3D object can also include pose information of the object (1).

[0146] For example, one or more processors (140) can obtain a 3D object corresponding to a virtual object (2) based on shape information of the virtual object (2).

[0147] In one example, one or more processors (140) can identify whether contact occurs based on the pose of the object (1) and the pose of the virtual object (2) in a new viewpoint (or an area where depth information is not acquired through the depth camera (120)) other than the viewpoint according to the depth camera (120).

[0148] FIG. 11 is a drawing for explaining a display device that provides feedback according to one embodiment of the present disclosure.

[0149] One or more processors (140) according to an example of the present disclosure may communicate with a wearable device (200) via a communication interface. Here, the wearable device (200) refers to a device made of a flexible material (e.g., silicone rubber, fiber, etc.) and capable of being worn by a user or in contact with a part of the user's body. For example, various types of devices such as watches, clothing (e.g., a vibration suit), shoes, gloves, glasses, hats, accessories (e.g., rings), etc. that can be worn on the body by a human or an animal may be included in the wearable device (200). However, this is merely an example and, of course, the present invention is not limited thereto.

[0150] According to an example of the present disclosure, one or more processors (140) can control the wearable device (200) to provide tactile feedback to an area on the object (1) where a part of the virtual object (2) has come into contact with the object (1) when contact between a part of the virtual object (2) and the object (1) is identified on the mirror display (110).

[0151] For example, as illustrated in FIG. 11, when a user (i.e., object (1)) is wearing a wearable device (200) on a shoulder area, and when changed shape information of a virtual object (2) is received from an external device (100') according to the movement of another user represented by a virtual object (2), the virtual object (2) can be realigned to the object (1) according to the changed shape information of the virtual object (2). According to an example, when a part of the virtual object (2) comes into contact with the shoulder area of ​​the object (1), one or more processors (140) can transmit a control signal to the wearable device (200) to provide feedback (e.g., vibration) to the shoulder area of ​​the user.

[0152] For example, when the posture of a user of an external device (100') (i.e., a user different from the user of the display device (100)) changes, the depth camera (120') of the external device (100') can obtain an image including the user whose posture has changed, and can obtain changed shape information of the user (hereinafter, virtual object (2)).

[0153] For example, an external device (100') transmits changed shape information to a display device (100), and one or more processors (140) can realign a virtual object (2) to an object (1) according to the changed shape information.

[0154] For example, by realigning the virtual object (2) to the object (1) according to the changed shape information of the virtual object (2) in real time or at preset time intervals, the display device (100) can provide the user of the display device (100) with the effect that the user of the external device (100') (i.e., another user) is located in the same space and the other user is reflected on the mirror display (110) of the display device (100).

[0155] In addition, the display device (100) further includes a speaker, and a microphone equipped in the external device (100') obtains the voice of another user, sounds generated in the space where the external device (100') is located, and transmits them to the display device (100), and the display device (100) can output the voice and sounds received from the external device (100'). In addition, the display device (100) further includes a microphone, and the microphone equipped in the display device (100) obtains the voice of a user, sounds generated in the space where the display device (100) is located, and transmits them to the external device (100'), and the external device (100') can output the voice and sounds received from the display device (100).

[0156] FIG. 12 is a drawing for explaining a display device communicating with an external device according to one embodiment of the present disclosure.

[0157] According to one example of the present disclosure, the display device (100) and the external device (100') may communicate with each other through an external server (1000).

[0158] In one example, one or more processors (140) may interact with a virtual object (2) when a user's input to the virtual object (2) is identified.

[0159] For example, if the user's posture changes and the object (1) reflected on the mirror display (110) touches the virtual object (2), or if contact between the object (1) and the virtual object (2) is identified, interaction with the virtual object (2) can be performed.

[0160] For example, if the virtual object (2) is a graphic object, one or more processors (140) can execute an action (or function) corresponding to the graphic object.

[0161] For example, if a virtual object (2) corresponds to a user of an external device (100'), one or more processors (140) transmit a signal to the external device (100') that controls a wearable device worn by the user of the external device (100') to provide tactile feedback, and the external device (100') can control the wearable device worn by the user of the external device (100') according to the received signal.

[0162] For example, one or more processors (140) may adjust at least one of the position, viewpoint, size, or ratio of the virtual object (2) on the mirror display (110) according to a user's input to the virtual object (2).

[0163] For example, if the user's input is an input for rotating a virtual object (2), one or more processors (140) can change the viewpoint of the virtual object (2) (or rotate the virtual object (2)) based on the shape information of the virtual object (2).

[0164] For example, if the user's input is an input that changes the sorting order of the virtual object (2), one or more processors (140) can position the virtual object (2) in front or behind the object (1) according to the user's input.

[0165] For example, if the user's input is an input that changes the size of the virtual object (2) (e.g., pinch to reduce the size, spread to increase the size, etc.), one or more processors (140) can change the size of the virtual object (2) according to the user's input.

[0166] Here, the user's input may include a touch input to the mirror display (110) (i.e., an input with physical contact with the mirror display (110), an input through an object (1) reflected on the mirror display (110) corresponding to the user's changed posture (i.e., an input without physical contact with the mirror display (110), etc.). For example, one or more processors (140) may identify a hand part (e.g., a thumb and an index finger, etc.) in an object (1) reflected on the mirror display (110), and identify an input based on the pose, position, shape, etc. of the identified hand part.

[0167] FIG. 13 is a flowchart for explaining a method for controlling a display device according to an embodiment of the present disclosure.

[0168] A control method according to an example obtains depth information of an object by acquiring an image of an object reflected on a mirror display (S1310).

[0169] Skeleton information of an object is acquired based on the depth information of the object (S1320).

[0170] Based on the acquired skeleton information, shape information of the object is acquired (S1330).

[0171] Based on the shape information of the object and the shape information of the virtual object, the virtual object is aligned with the object and provided through a mirror display (S1340).

[0172] According to an example, the skeleton information of an object includes a plurality of skeletons for each part included in the object, and the step S1330 of obtaining shape information may include a step of identifying the shape of each of the plurality of parts according to a symmetrical structure based on the plurality of skeletons for each part, and a step of obtaining shape information of an object including the shape of each of the plurality of parts.

[0173] Step S1330 of obtaining shape information according to an example of the present disclosure includes a step of obtaining shape information of an object included in the image by inputting an image into a neural network model, and the neural network model may be a model trained to output shape information of an object including depth information corresponding to each of a plurality of view points for the object from depth information of the object when an image including the object is input.

[0174] Step S1340, which provides an example of the present disclosure, may include a step of displaying a virtual object by positioning a part of the virtual object in front of the object and positioning the remainder of the virtual object in the back of the object based on shape information of the object and shape information of the virtual object.

[0175] Step S1340, according to an example of the present disclosure, may include a step of displaying a virtual object on a mirror display so that a part of the virtual object comes into contact with the object based on shape information of the object and shape information of the virtual object.

[0176] The control method of the present disclosure may further include a step of controlling a wearable device worn by the object to provide tactile feedback to an area on the object where a portion of the virtual object has come into contact, when contact between a portion of the virtual object and the object is identified on the mirror display.

[0177] The control method of the present disclosure may further include a step of receiving shape information of a virtual object from an external device and a step of transmitting shape information of the object to the external device.

[0178] The control method of the present disclosure further includes a step of performing an interaction with the virtual object based on the input when a user's input for the virtual object is identified, and the transmitting step includes a step of transmitting changed shape information of the object according to the user's input to an external device, and the user's input may include a touch input for a part of the virtual object.

[0179] Step S1340, which provides an example of the present disclosure, includes a step of realigning a virtual object to an object according to changed shape information received from an external device according to an input of another user for the object, wherein the virtual object corresponds to another user reflected on a mirror display provided in the external device, and the object can correspond to the user.

[0180] A control method according to an example of the present disclosure may further include a step of adjusting at least one of a position, a viewpoint, a size, or a ratio of a virtual object on a mirror display based on a user's input.

[0181] However, it goes without saying that the various embodiments of the present disclosure can be applied not only to display devices but also to various types of electronic devices that include mirror functions and display functions.

[0182] Meanwhile, the various embodiments described above may be implemented in a computer-readable recording medium or similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments, such as the procedures and functions described herein, may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0183] Meanwhile, computer instructions for performing processing operations of the display device (100) according to the various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in the non-transitory computer-readable medium are executed by a processor of a specific device, they cause the specific device to perform processing operations in the display device (100) according to the various embodiments described above.

[0184] A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0185] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. Mirror display; A depth camera capable of taking pictures of objects reflected on the above mirror display; and Obtaining an image including the object through the depth camera to obtain depth information of the object, Obtain skeleton information of the object based on the depth information of the object, Obtain shape information of the object based on the acquired skeleton information, A display device including one or more processors that align the virtual object with the object based on shape information of the object and shape information of the virtual object and provide the same through the mirror display.

2. In paragraph 1, The skeleton information of the above object is: Contains multiple skeletons for each part included in the above object, One or more of the above processors, Identify the shape of each of the plurality of parts according to the symmetrical structure based on the skeleton of each of the plurality of parts, A display device that obtains shape information of the object including the shape of each of the plurality of parts.

3. In paragraph 1, One or more of the above processors, By inputting the above image into a neural network model, shape information of the object included in the image is obtained, The above neural network model is, A display device, wherein the model is trained to output shape information of the object including depth information corresponding to each of a plurality of view points for the object from depth information of the object when the image including the object is input.

4. In paragraph 1, One or more of the above processors, A display device that displays the virtual object by positioning a part of the virtual object in front of the object and the remainder of the virtual object in the back of the object based on shape information of the object and shape information of the virtual object.

5. In paragraph 1, One or more of the above processors, A display device that displays a virtual object on the mirror display so that a part of the virtual object comes into contact with the object based on shape information of the object and shape information of the virtual object.

6. In paragraph 5, further comprising a communication interface; One or more of the above processors, Controlling the communication interface so that the object communicates with a wearable device worn by the object; When contact between a part of the virtual object and the object is identified on the mirror display, A display device that controls the wearable device to provide tactile feedback to an area where a part of the virtual object is in contact with the object.

7. In paragraph 1, further comprising a communication interface; One or more of the above processors, Receive shape information of the virtual object from an external device through the above communication interface, A display device that controls the communication interface to transmit shape information of the object to the external device.

8. In paragraph 7, One or more of the above processors, When a user's input for the virtual object is identified, an interaction is performed with the virtual object based on the input, Transmits the changed shape information of the object according to the user's input to the external device, The above user input is, A display device including a touch input for a portion of the virtual object.

9. In paragraph 8, One or more of the above processors, When the changed shape information of the virtual object is received from the external device according to the input of another user for the object, the virtual object is re-aligned to the object according to the changed shape information. The above virtual object corresponds to the other user reflected on the mirror display provided in the external device, The above object is a display device corresponding to the user.

10. In paragraph 1, One or more of the above processors, A display device that adjusts at least one of the position, viewpoint, size, or ratio of the virtual object on the mirror display based on a user's input.

11. A method for controlling a display device including a mirror display, A step of obtaining an image of an object reflected on the mirror display and obtaining depth information of the object; A step of obtaining skeleton information of the object based on depth information of the object; A step of obtaining shape information of the object based on the obtained skeleton information; and A control method comprising: a step of aligning the virtual object with the object based on shape information of the object and shape information of the virtual object and providing the same through the mirror display.

12. In paragraph 11, The skeleton information of the above object is: Contains multiple skeletons for each part included in the above object, The step of obtaining the above shape information is: A step of identifying the shape of each of the plurality of parts according to a symmetrical structure based on the skeleton of each of the plurality of parts; and A control method, comprising: a step of obtaining shape information of the object including the shape of each of the plurality of parts; 13. In paragraph 11, The step of obtaining the above shape information is: A step of inputting the image into a neural network model to obtain shape information of the object included in the image; The above neural network model is, A control method, wherein when an image including the object is input, the model is trained to output shape information of the object including depth information corresponding to each of a plurality of view points for the object from depth information of the object.

14. In paragraph 11, The steps provided above are: A control method, comprising: a step of displaying the virtual object by positioning a part of the virtual object in front of the object and the remainder of the virtual object in the back of the object based on shape information of the object and shape information of the virtual object.

15. In paragraph 11, The steps provided above are: A control method, comprising: a step of displaying the virtual object on the mirror display so that a part of the virtual object comes into contact with the object based on shape information of the object and shape information of the virtual object.

Images