Image output device including transparent display and projector and control method thereof

Abstract

Disclosed is an image output device. The image output device comprises: a transparent display; a projector; and one or more processors for inputting content into a first neural network model to acquire a foreground image including an object and a partial image in which the object has been excluded from the content, inputting the partial image into a second neural network model to acquire a background image, controlling the transparent display to display the foreground image, and controlling the projector to project the background image.

Description

Transparent display and projector, video output device and control method thereof

[0001] The present invention relates to an image output device and a control method thereof, and more particularly, to an image output device including a transparent display and a projector and a control method thereof.

[0002] Recently, various types of display devices are being developed and distributed.

[0003] In addition to conventional TVs, there is an increasing trend of users watching content using various types of display devices such as transparent displays and projectors.

[0004] However, transparent displays are only used for limited purposes such as exhibitions, and projectors are also used for limited purposes due to the difficulty in users adjusting the projection position of the content.

[0005] There has been a demand for a method that appropriately combines the strengths of transparent displays and projectors, making them universally usable beyond limited purposes, maximizing the user experience and increasing the immersion and three-dimensionality when viewing content.

[0006] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0007] According to an embodiment for achieving the above-described object of the present disclosure, an image output device includes a transparent display, a projector, and one or more processors for inputting content into a first neural network model to obtain a foreground image including an object and a partial image excluding the object from the content, inputting the partial image into a second neural network model to obtain a background image, controlling the transparent display to display the foreground image, and controlling the projector to project the background image, wherein the first neural network model is a neural network model trained to, when the content is input, separate the object from the content to obtain the foreground image and the partial image including the remainder of the content excluding the object, and the second neural network model is a neural network model trained to, when the partial image is input, fill in an area excluding the object to obtain the background image.

[0008] According to an embodiment for achieving the above-described object of the present disclosure, a method for controlling an image output device includes the steps of inputting content into a first neural network model to obtain a foreground image including an object and a partial image excluding the object from the content, the step of inputting the partial image into a second neural network model to obtain a background image, the step of controlling a transparent display to display the foreground image, and the step of controlling a projector to project the background image, wherein the first neural network model is a neural network model trained to, when the content is input, separate the object from the content to obtain the foreground image and the partial image including the remainder of the content excluding the object, and the second neural network model is a neural network model trained to, when the partial image is input, fill in an area excluding the object to obtain the background image.

[0009] According to an embodiment for achieving the above-described object of the present disclosure, there is provided a computer-readable recording medium including a program for executing a method for controlling an image output device, the method for controlling an image output device including a step of inputting content into a first neural network model to obtain a foreground image including an object and a partial image excluding the object from the content, a step of inputting the partial image into a second neural network model to obtain a background image, a step of controlling a transparent display to display the foreground image, and a step of controlling a projector to project the background image, wherein the first neural network model is a neural network model trained to, when the content is input, separate the object from the content to obtain the foreground image and the partial image including the remainder of the content excluding the object, and the second neural network model is a neural network model trained to, when the partial image is input, fill in an area excluding the object to obtain the background image.

[0010] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0011] FIG. 1 is a drawing for explaining an image output device according to an embodiment of the present disclosure.

[0012] FIG. 2 is a block diagram showing the configuration of an image output device according to an embodiment of the present disclosure.

[0013] FIG. 3 is a drawing for explaining a foreground image and a background image according to an embodiment of the present disclosure.

[0014] FIG. 4 is a drawing for explaining a transparent display for displaying a foreground image and a projector for displaying a background image according to an embodiment of the present disclosure.

[0015] FIG. 5 is a drawing for explaining out-painting according to an embodiment of the present disclosure.

[0016] FIG. 6 is a drawing for explaining a projector including a motor according to an embodiment of the present disclosure.

[0017] FIG. 7 is a drawing for explaining the distance between a wall and a transparent display according to an embodiment of the present disclosure.

[0018] FIG. 8 is a drawing for explaining an ambient image according to an embodiment of the present disclosure.

[0019] FIG. 9 is a drawing for explaining a background image according to an embodiment of the present disclosure.

[0020] FIG. 10 is a flowchart for explaining a method for controlling an image output device according to an embodiment of the present disclosure.

[0021] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0022] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0023] The embodiments of the present disclosure may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the present disclosure to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the scope of the disclosed concepts and techniques. In describing the embodiments, detailed descriptions of related known technologies will be omitted if they are deemed to obscure the main point.

[0024] Terms such as "first" and "second" may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0025] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0027] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description of the present disclosure, and similar parts are designated with similar reference numerals throughout the specification.

[0028] FIG. 1 is a drawing for explaining the configuration of an image output device according to one embodiment of the present disclosure.

[0029] Referring to FIG. 1, the image output device (100) can be a device of various types that outputs an image.

[0030] In particular, the image output device (100) may include a transparent display (110) and a projector (120).

[0031] According to an embodiment, the transparent display (110) may be implemented as a TV, but is not limited to and may be applied to any device with a display function, such as a video wall, a large format display (LFD), a digital signage, a digital information display (DID), a projector display, etc.

[0032] For example, a transparent display (110) uses transparent materials for electrical wiring, such as TFT (Thin Film Transistor), to make areas other than the part expressing color transparent, and allows a user (1) to see objects behind the transparent display (110) as if they were transparent glass.

[0033] According to an embodiment, the transparent display (110) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, the transparent display (110) may be implemented as a display of various forms, such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a Micro LED, a MINI LED, a PDP (Plasma Display Panel), a QD (Quantum Dot) display, a QLED (Quantum Dot Light-Emitting Diodes), etc.

[0034] According to an embodiment, the self-luminous element or non-luminous element included in the transparent display (110) may be a transparent element.

[0035] For example, the transparent display (110) may include an OLED element in which both the anode and cathode are implemented as transparent based on the principle that metals having a thickness less than a preset thickness (e.g., 10 nm) become transparent. According to an example, the transparent display (110) may have high light transmittance (or transmittance) (e.g., transmittance of 45% or more) in a wavelength range of 380 to 780 nm.

[0036] According to an embodiment, the transparent display (110) includes a transparent MicroLED, which may include a MicroLED disposed on a transparent substrate such as glass or a transparent polymer.

[0037] The transparent display (110) according to the embodiment of the present disclosure may include a driving circuit, a backlight, etc. implemented with a-si TFT, LTPS (Low Temperature Poly Silicon) TFT, OTFT (Organic TFT), etc.

[0038] In some embodiments, the transparent display (110) may include an edge-type backlight to provide a foreground image (10).

[0039] Here, the edge-type backlight is arranged on at least one of the multiple sides of the light guide plate and can output light toward the light guide plate. The light output by the edge-type backlight can be reflected forward through the light guide plate to the center of the transparent display (110).

[0040] According to an embodiment, the projector (120) can project a background image (20) onto a wall or screen.

[0041] For example, the projector (120) may be implemented as an LCD projector or a DLP (digital light processing) projector using a DMD (Digital Micro-Mirror Device).

[0042] According to an embodiment, a transparent display (110) displays a foreground image (10), a projector (120) displays a background image (20), and an image output device (100) displays content, and the image output device (100) can provide a three-dimensional effect, immersion, etc. to a user (1) viewing the content.

[0043] FIG. 2 is a block diagram showing the configuration of an image output device according to an embodiment of the present disclosure.

[0044] Referring to FIG. 2, the image output device (100) includes a transparent display (110), a projector (120), and one or more processors (130).

[0045] According to an embodiment, the transparent display (110) may be a display panel that displays a foreground image (10), and at the same time, may be glass that transmits a portion of light incident on the transparent display (110) so that the user (1) can see the back of the transparent display (110).

[0046] According to an embodiment, a projector (120) is spaced apart from the front of a transparent display (110) as shown in FIG. 1, and can project a background image (20) toward the transparent display (110) so as to be overlaid on a foreground image (10) displayed by the transparent display (110).

[0047] According to an embodiment, a portion of the light projected by the projector (120) and incident on the transparent display (110) passes through the transparent display (110), so the background image (20) projected by the projector (120) can pass through the transparent display (110) and be formed on a wall or screen placed at the rear of the transparent display (110).

[0048] A detailed description of the foreground image (10) and background image (20) will be provided later.

[0049] One or more processors (130) are electrically connected to a memory (not shown) and control the overall operation of the image output device (100).

[0050] According to an embodiment of the present disclosure, one or more processors (130) may be implemented as a digital signal processor (DSP), a microprocessor, or a timing controller (TCON) for processing a digital signal. However, the present disclosure is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, or an artificial intelligence (AI) processor, or may be defined by the corresponding terminology. In addition, one or more processors (130) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). One or more processors (130) may perform various functions by executing computer executable instructions stored in a memory.

[0051] The artificial intelligence-related functions according to the present disclosure are operated via one or more processors (130) and memory. According to an embodiment, the one or more processors (130) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an NPU (Neural Processing Unit), but are not limited to the examples of the processors described above.

[0052] CPUs are general-purpose processors capable of performing not only general calculations but also artificial intelligence calculations. Their multi-layered cache structure allows for the efficient execution of complex programs. CPUs are advantageous for serial processing, enabling organic linking of previous and subsequent calculation results through sequential calculations. General-purpose processors are not limited to the examples described above, except where specifically identified as CPUs.

[0053] A GPU is a processor designed for large-scale computations, such as floating-point operations used in graphics processing. It integrates a large number of cores to perform large-scale computations in parallel. In particular, GPUs may be advantageous over CPUs in parallel processing methods, such as convolution operations. Furthermore, GPUs can be used as coprocessors to supplement the functions of CPUs. Processors for large-scale computations are not limited to the examples described above, except in cases where they are specifically referred to as GPUs.

[0054] An NPU is a processor specialized in artificial intelligence computation using artificial neural networks, and each layer of the artificial neural network can be implemented in hardware (e.g., silicon). Since an NPU is designed specifically according to the company's specifications, it has less freedom than a CPU or GPU, but can efficiently process the AI ​​computations requested by the company. Meanwhile, as a processor specialized in artificial intelligence computation, an NPU can be implemented in various forms, such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), or a Vision Processing Unit (VPU). Except as specifically stated as an NPU, an AI processor is not limited to the examples described above.

[0055] Additionally, one or more processors (130) may be implemented as a System on Chip (SoC). In this case, the SoC may further include, in addition to one or more processors (130), a memory, and a network interface such as a bus for data communication between one or more processors (130) and the memory.

[0056] When a plurality of processors are included in the SoC (System on Chip) included in the image output device (100), the image output device (100) can perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the plurality of processors. For example, the image output device (100) can perform operations related to artificial intelligence by using at least one of a GPU, NPU, VPU, TPU, or hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors. However, this is merely an example for the convenience of explanation, and it goes without saying that operations related to artificial intelligence can be processed by using a general-purpose processor such as a CPU.

[0057] In addition, the image output device (100) can perform operations related to functions related to artificial intelligence by utilizing multi-cores (e.g., dual cores, quad cores, etc.) included in one or more processors (130). In particular, the image output device (100) can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel by utilizing multi-cores included in one or more processors (130).

[0058] One or more processors (130) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in memory. The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0059] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0060] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0061] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0062] According to an embodiment, one or more processors (130) may input content into a first neural network model to obtain a foreground image (10) including an object and a partial image excluding the object from the content.

[0063] According to an embodiment, one or more processors (130) may input a partial image into a second neural network model to obtain a background image (20).

[0064] A detailed explanation of this will be provided with reference to Fig. 3.

[0065] FIG. 3 is a drawing for explaining a foreground image and a background image according to an embodiment of the present disclosure.

[0066] Referring to FIG. 3, one or more processors (130) can input content (A) into a first neural network model to obtain a foreground image (10) including an object and a partial image excluding the object from the content.

[0067] According to an embodiment, the first neural network model may be a neural network model trained to separate an object from content (A) and obtain a partial image including a foreground image (10) and the remainder excluding the object.

[0068] According to an embodiment, the first neural network model can obtain a foreground image (10) by separating a character object that can be controlled by a user's (1) operation, a main object that can be interacted with, an object larger than a preset size, an object located in a preset area (e.g., a center area), etc. from a plurality of objects included in the content (A) based on depth map information, object map information, etc. of the content (A).

[0069] An object map may refer to an image channel that represents the shape and appearance of a predefined object or an object with a priority higher than a threshold value among multiple objects included in the content (A).

[0070] However, it is not limited, and the first neural network model can obtain a foreground image (10) by separating an object from the content (A) based on RGB Histogram, Color Histogram, saliency map information (or, saliency mask), or peak highlight map information.

[0071] Saliency Map information is an image channel that represents an area where pixel values ​​change rapidly within content (A), and may be an image channel that represents an object within content (A) by separating it from the background.

[0072] Peak Highlight Map information includes High Contrast information and may be an image channel that increases the gap between the brightest and darkest areas within the content (A) and separates objects from the background within the content (A).

[0073] According to an embodiment, the first neural network model can separate an object from content (A) and obtain a partial image excluding the object.

[0074] According to an embodiment, one or more processors (130) may input a partial image into a second neural network model to obtain a background image (20).

[0075] According to an embodiment, the second neural network model may be a neural network model trained to obtain a background image (20) by filling in an area where an object is excluded when a partial image is input.

[0076] For example, when a partial image excluding an object is input, the second neural network model can obtain a background image (20) by performing in-painting processing to fill in the area where the object is excluded based on the surrounding area of ​​the area where the object is excluded.

[0077] For example, the second neural network model can identify the context of the surrounding area from which the object is excluded based on the texture, color, pattern, etc. of the surrounding area from which the object is excluded.

[0078] According to an embodiment, the second neural network model can output a background image (20) by generating pixels to naturally fill in an area where an object is excluded based on the context.

[0079] FIG. 4 is a drawing for explaining a transparent display for displaying a foreground image and a projector for displaying a background image according to an embodiment of the present disclosure.

[0080] Referring to FIG. 4, one or more processors (130) can control a transparent display (110) to display a foreground image.

[0081] According to an embodiment, one or more processors (130) may control a projector (120) to display a background image.

[0082] As illustrated in FIG. 4, the projector (120) can project the background image (20) toward the transparent display (110) so that the background image (20) is overlaid on the foreground image (10) displayed by the transparent display (110).

[0083] Since a portion of the light projected by the projector (120) and incident on the transparent display (110) passes through the transparent display (110), the background image (20) projected by the projector (120) is formed on a wall or screen placed at the back of the transparent display (110), and can provide the user (1) with content (A) with a three-dimensional effect and immersion.

[0084] FIG. 5 is a drawing for explaining out-painting according to an embodiment of the present disclosure.

[0085] According to an embodiment, when a partial image excluding an object is input, the second neural network model performs an in-painting process to fill in an area where an object is excluded based on a surrounding area of ​​the area where the object is excluded, and performs an out-painting process to expand the partial image in at least one direction among up, down, left, and right, thereby obtaining a background image (20).

[0086] For example, the second neural network model can obtain a background image (20) that includes a new area by extending a partial image beyond the existing boundary, as opposed to in-painting processing.

[0087] For example, the second neural network model can analyze the style, texture, color, pattern, etc. of a partial image to create a new region that is natural and consistent with the partial image.

[0088] For example, when a partial image excluding an object is input, the second neural network model can obtain a background image (20) by expanding the left region based on the style, texture, color, pattern, etc. of the left edge region. However, this is an example for convenience of explanation, and the second neural network model can also obtain a background image (20) by expanding each of the right edge region, the upper edge region, and the lower edge region in addition to the left edge region.

[0089] FIG. 6 is a drawing for explaining a projector including a motor according to an embodiment of the present disclosure.

[0090] Referring to FIG. 6, one or more processors (130) may receive sensing data from a sensor.

[0091] For example, a sensor is provided in a transparent display (110) or a projector (120) and can acquire sensing data by detecting the position of the user (1), the position of the head of the user (1), or the viewpoint of the user (1).

[0092] According to an embodiment, when sensing data is received from a sensor, one or more processors (130) can adjust the projection position of the projector (120) based on the position of the user (1), the head position of the user (1), etc. included in the sensing data.

[0093] For example, if a user (1) is positioned relatively to the left of the center of the front of a transparent display (110), the background image (20) must be projected relatively to the right due to the distance between the transparent display (110) displaying the foreground image (10) and the wall (or screen) on which the background image (20) is displayed, so that the foreground image (10) and the background image (20) can be appropriately overlaid without distortion.

[0094] According to an embodiment, one or more processors (130) can adjust the projection position of the projector (120) by controlling a motor so that the background image (20) is projected to the right when the user (1) is positioned to the left based on the detection data.

[0095] As another example, if the user (1) is positioned relatively to the right of the center of the front of the transparent display (110), the background image (20) must be projected relatively to the left so that the foreground image (10) and the background image (20) can be appropriately overlaid.

[0096] According to an embodiment, one or more processors (130) can adjust the projection position of the projector (120) by controlling a motor so that the background image (20) is projected to the left when the user (1) is positioned to the right based on the detection data.

[0097] As another example, when a user (1) is positioned at the center of the front of a transparent display (110), the background image (20) must be projected relatively toward the center so that the foreground image (10) and the background image (20) can be properly overlaid, and therefore, one or more processors (130) can control a motor to adjust the projection position of the projector (120) so that the background image (20) is projected toward the center.

[0098] According to an embodiment, when a user (1) is positioned relatively lower than the center of the front of a transparent display (110), the background image (20) must be projected relatively upward due to the distance between the transparent display (110) displaying the foreground image (10) and the wall (or screen) on which the background image (20) is displayed, so that the foreground image (10) and the background image (20) can be properly overlaid without distortion.

[0099] According to an embodiment, one or more processors (130) can adjust the projection position of the projector (120) by controlling a motor so that the background image (20) is projected upward when the user (1) is positioned downward based on the detection data.

[0100] As another example, if the user (1) is positioned relatively upward with respect to the center of the front of the transparent display (110), the background image (20) must be projected relatively to the left so that the foreground image (10) and the background image (20) can be appropriately overlaid.

[0101] According to an embodiment, one or more processors (130) can adjust the projection position of the projector (120) by controlling a motor so that the background image (20) is projected to the left when the user (1) is positioned to the right based on the detection data.

[0102] According to an embodiment, one or more processors (130) control a motor provided in the projector (120), and the motor may include a PTZ (Pan, Tilt, Zoom) motor. Here, the PTZ motor may include a motor that can rotate in a pan (pan: horizontal direction, left and right direction) direction and a motor that can rotate in a tilt (tilt: vertical direction, up and down) direction.

[0103] According to an embodiment, one or more processors (130) can adjust the projection position of the projector (120) by rotating the PTZ motor in at least one of four directions (e.g., up, down, left, right).

[0104] According to an embodiment, the sensor can detect a foreground image (10) displayed by a transparent display (110) and a background image (20) projected by a projector (120).

[0105] According to an embodiment, one or more processors (130) can control a motor to adjust the projection position of the projector (120) so that the foreground image (10) and the background image (20) are naturally connected, i.e., seamlessly connected, without a space between the foreground image (10) and the background image (20) based on the detection data received from the sensor.

[0106] For example, the sensor includes a camera, and the camera can capture a foreground image, a background image, and a user (1) to obtain sensing data.

[0107] According to an embodiment, one or more processors (130) may adjust the projection position of the projector (120) based on the sensing data so that there is no space between the display area of ​​the foreground image (10) and the projection area of ​​the background image (20), i.e., so that it is seamless.

[0108] Additionally, one or more processors (130) may perform calibration of the transparent display (110) and the projector (120) based on an area where the foreground image (10) included in the detection data is displayed and an area where the background image (20) is projected. For example, one or more processors (130) may perform lens shift, color matching, edge blending, keystone correction, leveling correction (horizontal correction), focus correction, etc.

[0109] FIG. 7 is a drawing for explaining the distance between a wall and a transparent display according to an embodiment of the present disclosure.

[0110] Referring to FIG. 7, one or more processors (130) can identify a distance (D) between a wall (or screen) and a transparent display (110) and adjust the size of a foreground image (10) including an object based on the distance (D).

[0111] For example, as the distance (D) between the wall (or screen) on which the background image (20) is formed and the transparent display (110) that displays the foreground image (10) increases, the size of the object included in the foreground image (10) relative to the background included in the background image (20) decreases relatively, so the image output device (100) may have a problem of distorting the content (A) and providing it to the user (1).

[0112] According to an embodiment, one or more processors (130) may identify a distance (D) between a wall and the transparent display (110) through a sensor or the like provided in the transparent display (110), and adjust the size of the foreground image (10) or the size of the background image (20) based on the distance (D).

[0113] For example, as the distance (D) between the wall and the transparent display (110) increases, one or more processors (130) can increase the size of an object to display a foreground image (10) through the transparent display (110), or reduce the size of a background image (20) to project it through the projector (120).

[0114] According to an embodiment, one or more processors (130) can adjust the size of the object and the size of the background image (20) to be closer to the original size of the content (A) as the distance (D) between the wall and the transparent display (110) decreases.

[0115] FIG. 8 is a drawing for explaining an ambient image according to an embodiment of the present disclosure.

[0116] Referring to FIG. 8, one or more processors (130) can control a transparent display (110) to display content (A).

[0117] According to an embodiment, one or more processors (130) may input content (A) into a third neural network model to obtain an ambient image (20') and control a projector (120) to project the ambient image (20').

[0118] According to an embodiment, the third neural network model is a neural network model trained to obtain an ambient image (20') based on the edge of the content (A), and the ambient image (20') may include at least one of a first region corresponding to colors of a left edge region of the content (A), a second region corresponding to colors of a right edge region, a third region corresponding to colors of an upper edge region, or a fourth region corresponding to colors of a lower edge region.

[0119] According to an embodiment, the image output device (100) can provide content (A) through a transparent display (110) and at the same time, provide a sense of immersion to the user (1) by projecting an ambient image (20') including a color corresponding to the content (A) through a projector (120).

[0120] For example, one or more processors (130) can obtain an ambient image (20') that dynamically changes according to the color of the content (A) through a third neural network model, and control a projector (120) to project the ambient image (20') to increase the immersion of a user (1) viewing the content (A).

[0121] According to an embodiment, the color of the content (A) may include a main color of the content (A) (e.g., a color corresponding to the highest frequency among a plurality of colors included in the background image (20), an average color of the background image (20), a color corresponding to an object included in the foreground image (10), etc.), colors corresponding to each of the upper edge area, the lower edge area, the left edge area, and the right edge area of ​​the content (A), etc.

[0122] FIG. 9 is a drawing for explaining a background image according to an embodiment of the present disclosure.

[0123] Referring to FIG. 9, one or more processors (130) input a partial image excluding an object into a second neural network model, perform out-painting processing to obtain a background image (20), and control a projector (120) to project the remainder (20') excluding the content (A) from the background image (20).

[0124] For example, one or more processors (130) may input a partial image excluding an object into a second neural network model and perform out-painting processing to expand the partial image in at least one direction among up, down, left, and right.

[0125] According to an embodiment, one or more processors (130) may extend a partial image beyond an existing boundary to obtain an image (20') composed only of a new region generated by a second neural network model, and when a transparent display (110) displays content (A), at the same time, a projector (120) may be controlled to project an image (20') composed only of a new region generated by the second neural network model.

[0126] According to an embodiment, one or more processors (130) may control a PTZ motor to adjust the projection position of the projector (120) so that the content (A) displayed by the transparent display (110) and the image (20') composed only of a new area generated by the second neural network model are naturally connected (i.e., seamlessly connected) based on the sensed data received from the sensor.

[0127] According to an embodiment, one or more processors (130) control a transparent display (110) to display a foreground image (10) and may also control a projector (120) to project content (A).

[0128] According to an embodiment, the projection position of the projector (120) can be adjusted by controlling the PTZ motor so that an object included in the content (A) projected by the projector (120) and an object included in the foreground image (10) displayed by the transparent display (110) are overlaid based on the detection data received from the sensor.

[0129] Returning to FIG. 2, for convenience of explanation, the image processing device (100) is described as including a transparent display (110) and a projector (120), but is not limited thereto. The image processing device (100) may include only a transparent display (110) and may communicate with a projector implemented as an external device to perform various embodiments of the present disclosure. Alternatively, the image processing device (100) may include only a projector (120) and may communicate with a transparent display implemented as an external device to perform various embodiments of the present disclosure.

[0130] For example, if the image processing device (100) includes a transparent display (110) and performs various embodiments of the present disclosure by communicating with a projector implemented as an external device, the transparent display (110) may include one or more processors (130) and a communication interface.

[0131] A communication interface according to an embodiment of the present disclosure can transmit and receive various types of data and information by performing communication with an external device, an external server, etc.

[0132] For example, the communication interface can transmit and receive various types of data and information to and from external devices (e.g., projectors), external storage media (e.g., USB memory), external servers (e.g., cloud servers, web hard drives), etc. through communication methods / communication standards such as AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), Thunderbolt™, MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0133] FIG. 10 is a flowchart for explaining a method for controlling an image output device according to an embodiment of the present disclosure.

[0134] According to an embodiment, a method for controlling an image output device inputs content into a first neural network model to obtain a foreground image including an object and a partial image excluding the object from the content (S1010).

[0135] A partial image is input into a second neural network model to obtain a background image (S1020).

[0136] Control the transparent display to display the foreground image (S1030).

[0137] Control the projector to project a background image (S1040).

[0138] According to an embodiment, the first neural network model may be a neural network model trained to obtain a foreground image and a partial image including the remainder of the content excluding the object by separating the object from the content when content is input, and the second neural network model may be a neural network model trained to obtain a background image by filling in the area where the object is excluded when a partial image is input.

[0139] In an embodiment, step S1040 of controlling a projector includes a step of controlling a projector spaced apart from the front of the transparent display to project a background image toward the transparent display so that a foreground image is overlaid on the background image, wherein the background image can be projected on the rear of the transparent display.

[0140] According to an embodiment, the projector includes a PTZ (Pan, Tilt, Zoom) motor, and the step S1040 of controlling the projector may include a step of controlling the PTZ motor so that a projection position of the projector projecting the background image is adjusted based on at least one of a distance between a wall on which the background image is displayed and the transparent display or a position of the user.

[0141] According to an embodiment, the projector includes a sensor and a PTZ motor, and the step S1040 of controlling the projector may include a step of adjusting a projection position of the projector by controlling the PTZ motor so that a display area of ​​a foreground image and a projection area of ​​a background image are seamlessly connected based on detection data received from the sensor.

[0142] According to an embodiment, the second neural network model may be a neural network model trained to obtain a background image by performing at least one of an in-painting process that fills an area where an object is excluded or an out-painting process that expands a partial image in at least one direction of up, down, left, and right.

[0143] According to an embodiment, the control method may further include a step of obtaining the remainder of the background image excluding the content after performing out-painting processing on the partial image through a second neural network model, a step of controlling a transparent display to display the content, and a step of controlling a projector to project the remainder of the background image excluding the content.

[0144] According to an embodiment, a control method includes a step of inputting content into a third neural network model to obtain an ambient image, a step of controlling a transparent display to display the content, and a step of controlling a projector to project the ambient image, wherein the third neural network model is a neural network model learned to obtain an ambient image based on an edge of the content, and the ambient image may include at least one of a first region corresponding to colors of a left edge region of the content, a second region corresponding to colors of a right edge region, a third region corresponding to colors of an upper edge region, or a fourth region corresponding to colors of a lower edge region.

[0145] According to an embodiment, the control method further includes a step of adjusting the size of an object based on a distance between a wall on which a background image is displayed and a transparent display, and the step S1030 of controlling the transparent display may include a step of displaying a foreground image including the adjusted-size object through the transparent display.

[0146] According to an embodiment, the step S1010 of obtaining a foreground image and a partial image may include a step of obtaining a foreground image including a character object controllable according to a user operation or a main object interactable according to a user operation among the plurality of objects when a plurality of objects are identified in the content, and the step of obtaining a background image may include a step of obtaining a background image based on a partial image excluding the character object or the main object.

[0147] According to an embodiment, the projector includes a sensor, and the step S1040 of controlling the projector may include the step of controlling the projector to project content and the step of adjusting the projection position of the projector so that an object included in the content projected by the projector and an object included in a foreground image are overlaid based on sensed data received from the sensor.

[0148] However, it goes without saying that the various embodiments of the present disclosure can be applied not only to image output devices but also to various types of electronic devices capable of image output.

[0149] Meanwhile, the various embodiments described above may be implemented in a computer-readable recording medium or similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments, such as the procedures and functions described herein, may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0150] Meanwhile, computer instructions for performing processing operations of an electronic device according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in the non-transitory computer-readable medium are executed by a processor of a specific device, the computer instructions cause the specific device to perform processing operations in an image output device according to various embodiments described above.

[0151] A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0152] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the video output device, transparent display; Projector; and By inputting the content into the first neural network model, a foreground image including the object and a partial image excluding the object from the content are obtained, The above partial image is input into the second neural network model to obtain a background image, Controlling the transparent display to display the foreground image; one or more processors controlling the projector to project the background image; The above first neural network model is, When the above content is input, a neural network model trained to separate the object from the content and obtain the partial image including the foreground image and the remainder of the content excluding the object, The above second neural network model is, An image output device, which is a neural network model trained to obtain the background image by filling in the area where the object is excluded when the above partial image is input.

2. In paragraph 1, The above projector, The foreground image is projected toward the transparent display so that the foreground image is overlaid on the background image, and the background image is spaced apart from the front of the transparent display. The above background image is a video output device that is projected onto the back of the transparent display.

3. In paragraph 2, The above projector, Includes PTZ (Pan, Tilt, Zoom) motor; One or more of the above processors, An image output device that controls the PTZ motor so that the projection position of the projector that projects the background image is adjusted based on at least one of the distance between the wall on which the background image is displayed and the transparent display or the position of the user.

4. In paragraph 1, The above projector, sensor; and PTZ motor; One or more of the above processors, An image output device that controls the PTZ motor to adjust the projection position of the projector so that the display area of ​​the foreground image and the projection area of ​​the background image are seamlessly connected based on the detection data received from the sensor.

5. In paragraph 1, The above second neural network model is, An image output device, wherein the neural network model is trained to obtain the background image by performing at least one of an in-painting process that fills an area where the object is excluded or an out-painting process that expands the partial image in at least one direction of up, down, left, and right.

6. In paragraph 5, One or more of the above processors, After performing the out-painting process on the partial image through the second neural network model, the remainder excluding the content is obtained from the background image, Controlling the transparent display to display the above content, An image output device that controls the projector to project the remainder of the background image except for the content.

7. In paragraph 1, One or more of the above processors, By inputting the above content into the third neural network model, an ambient image is obtained, Controlling the transparent display to display the above content, Controlling the projector to project the above ambient image, The above third neural network model is, A neural network model trained to obtain the ambient image based on the edge of the above content, The above ambient image is, An image output device comprising at least one of a first region corresponding to colors of the left edge region of the content, a second region corresponding to colors of the right edge region, a third region corresponding to colors of the upper edge region, or a fourth region corresponding to colors of the lower edge region.

8. In paragraph 1, One or more of the above processors, Adjust the size of the object based on the distance between the wall on which the background image is displayed and the transparent display, An image output device that displays the foreground image including the object whose size has been adjusted through the transparent display.

9. In paragraph 1, One or more of the above processors, When multiple objects are identified in the above content, the foreground image including a character object controllable according to user operation or a main object interactable according to user operation among the multiple objects is acquired, An image output device that obtains the background image based on a partial image excluding the character object or the main object.

10. In paragraph 1, The above projector, including a sensor; One or more of the above processors, Controlling the transparent display to display the foreground image; Control the projector to project the above content, An image output device that adjusts the projection position of the projector so that the object included in the content projected by the projector and the object included in the foreground image are overlaid based on the detection data received from the sensor.

11. In a method for controlling a video output device, A step of inputting content into a first neural network model to obtain a foreground image including an object and a partial image excluding the object from the content; A step of inputting the above partial image into a second neural network model to obtain a background image; A step of controlling a transparent display to display the foreground image; and A step of controlling a projector to project the above background image; The above first neural network model is, When the above content is input, a neural network model trained to separate the object from the content and obtain the partial image including the foreground image and the remainder of the content excluding the object, The above second neural network model is, A control method, wherein when the above partial image is input, the neural network model is trained to obtain the background image by filling the area where the object is excluded.

12. In paragraph 11, The steps for controlling the above projector are: A step of controlling the projector spaced apart from the front of the transparent display to project the background image toward the transparent display so that the foreground image is overlaid on the background image; The above background image is, A control method projected onto the rear of the above transparent display.

13. In paragraph 12, The above projector, Includes PTZ (Pan, Tilt, Zoom) motor; The steps for controlling the above projector are: A control method comprising: a step of controlling the PTZ motor so that the projection position of the projector projecting the background image is adjusted based on at least one of a distance between the wall on which the background image is displayed and the transparent display or a position of the user; 14. In paragraph 11, The above projector, sensor; and PTZ motor; The steps for controlling the above projector are: A control method comprising: a step of controlling the PTZ motor to adjust the projection position of the projector so that the display area of ​​the foreground image and the projection area of ​​the background image are seamlessly connected based on the detection data received from the sensor.

15. A non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor of an image output device, cause the image output device to perform an operation, wherein the operation is: A step of inputting content into a first neural network model to obtain a foreground image including an object and a partial image excluding the object from the content; A step of inputting the above partial image into a second neural network model to obtain a background image; A step of controlling a transparent display to display the foreground image; and A step of controlling a projector to project the above background image; The above first neural network model is, When the above content is input, a neural network model trained to separate the object from the content and obtain the partial image including the foreground image and the remainder of the content excluding the object, The above second neural network model is, A non-transitory computer-readable storage medium, wherein the neural network model is trained to obtain the background image by filling in the area where the object is excluded when the partial image is input.

Images