Image projection device and operating method thereof

Abstract

The present disclosure relates to an image projection device and an operating method thereof. The image projection device according to an embodiment of the present disclosure may comprise: a light source assembly including at least one light source; a display element which outputs an image on the basis of light output from the light source assembly; an optical lens which projects the image output from the display element onto a real-world background; and a control unit, wherein the control unit determines a shape corresponding to a predetermined area of the real-world background onto which the image is projected, processes an image to be projected, on the basis of a result of processing of the shape by using a polygon mesh, and controls the display element so that a predetermined image corresponding to the processed image is output.

Description

Video projection device and its operating method

[0001] The present disclosure relates to an image projection device and an operating method thereof, and more specifically, to an image projection device capable of projecting an image optimized for a real background on which the image is projected, and an operating method thereof.

[0002] With the recent increase in consumption of high-quality, high-capacity multimedia content, there is a growing demand for larger and higher-quality screens. Among display devices, video projectors use optical devices to project images externally, offering the advantage of easily implementing large screens compared to other display devices.

[0003] Conventional video projection devices are typically configured to project images onto a flat surface, such as a screen installed directly in front of the user or a wall surface corresponding to the screen. Recently, research is being conducted to apply augmented reality (AR), a technology that adds virtual information to the real world, to video projection devices, projecting AR content onto various objects and spaces.

[0004] Meanwhile, when projecting images using augmented reality, an external device processes the image, considering the composition and size of the object on which the image is projected, and then the image projection device receives and outputs the processed image from the external device. However, when projecting a pre-processed image, the problem of the image projected onto the real background becoming distorted occurs as the position of the object on which the image is projected or the position or direction of the image projection device changes.

[0005] The present disclosure aims to solve the above-mentioned and other problems.

[0006] Another purpose is to provide an image projection device and an operating method thereof that can project images without distortion on various real backgrounds.

[0007] Another purpose is to provide an image projection device and an operating method thereof that can process an image in response to changes in the real background on which the image is projected.

[0008] Another purpose is to provide an image projection device and an operating method thereof that allow a user to freely set a real background on which an image is projected.

[0009] Another purpose is to provide an image projection device and an operating method thereof that can automatically identify a real background on which an image is projected and process the image.

[0010] In order to achieve the above object, an image projection device according to one embodiment of the present disclosure includes a light source assembly including at least one light source; a display element that outputs an image based on light output from the light source assembly; an optical lens that projects the image output from the display element onto a real background; and a control unit, wherein the control unit determines a shape corresponding to a predetermined area of ​​the real background on which the image is projected, processes an image as a projection target based on a result of processing the shape using a polygon mesh, and controls the display element so that a predetermined image corresponding to the processed image is output.

[0011] In order to achieve the above object, an operating method of an image projection device according to one embodiment of the present disclosure may include an operation of determining a shape corresponding to a predetermined area of ​​a real background on which an image is projected through an optical lens; an operation of processing an image as a projection target based on a result of processing the shape using a polygon mesh; and an operation of outputting a predetermined image corresponding to the processed image.

[0012] The effects of the image projection device and its operating method according to the present disclosure are described as follows.

[0013] According to at least one embodiment of the present disclosure, an image can be projected onto various real backgrounds without distortion as the image is processed in response to the real background in the image projection device.

[0014] According to at least one embodiment of the present disclosure, by using a polygon mesh, an image can be processed in response to changes in a real background on which the image is projected.

[0015] According to at least one embodiment of the present disclosure, a user can freely set a real-world background on which an image is projected.

[0016] According to at least one embodiment of the present disclosure, an image can be processed by automatically identifying a real background onto which an image is projected using a camera.

[0017] Further scope of the applicability of the present disclosure will become apparent from the detailed description below. However, since various modifications and variations within the spirit and scope of the present disclosure will become apparent to those skilled in the art, it should be understood that the detailed description and specific examples, such as preferred embodiments of the present disclosure, are given by way of example only.

[0018] FIG. 1 is a diagram illustrating an imaging system according to one embodiment of the present disclosure.

[0019] Figure 2 is an internal block diagram of the image projection device of Figure 1.

[0020] Figure 3 is an internal block diagram of the signal processing device of Figure 2.

[0021] FIG. 4 is an example of the structure of an optical device of an image projection device according to one embodiment of the present disclosure.

[0022] Fig. 5 is an example of an internal block diagram of the remote control device of Fig. 1.

[0023] FIGS. 6 and 7 are flowcharts of an operation method of an image projection device according to one embodiment of the present disclosure.

[0024] FIGS. 8A to 16 are drawings for reference in explaining the operation of an image projection device according to one embodiment of the present disclosure.

[0025] FIG. 17 is a flowchart of an operation method of an image projection device according to another embodiment of the present disclosure.

[0026] FIG. 18 is a drawing for reference in explaining the operation of an image projection device according to another embodiment of the present disclosure.

[0027] Hereinafter, the present disclosure will be described in detail with reference to the drawings. In the drawings, portions irrelevant to the description are omitted to clearly and concisely describe the present disclosure, and the same reference numerals are used for identical or extremely similar portions throughout the specification.

[0028] The suffixes "module" and "part" used in the following description are given solely for the convenience of writing this specification and do not impart any particularly significant meaning or role to the components themselves. Therefore, the terms "module" and "part" may be used interchangeably.

[0029] In this application, it should be understood that terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] Additionally, while terms such as "first" and "second" may be used in this specification to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another.

[0031] Hereinafter, an optical device may refer to a device that outputs visible light. Such an optical device may be applied to an image projection device. Alternatively, it may also be applied to a lighting device.

[0032] Hereinafter, the image projection device may be referred to as a projector. Meanwhile, the image projection device described in this disclosure may also be installed as a component within another device. For example, it may be installed within a mobile terminal, or it may be incorporated into home appliances such as air conditioners, refrigerators, cooking appliances, and robot vacuum cleaners, or it may be installed within vehicles such as automobiles.

[0033] FIG. 1 is a diagram illustrating an imaging system according to one embodiment of the present disclosure.

[0034] Referring to FIG. 1, the image system (10) may include an image projection device (100). The image projection device (100) may project an image onto a real background (20). Hereinafter, an image projected from the image projection device (100) may be referred to as a projection image. In FIG. 1, the real background (20) is exemplified as a screen having a flat surface, but is not limited thereto. A user may view a projection image projected onto the real background (20).

[0035] The video system (10) may include a remote control device (200). The remote control device (200) may be connected to the video projection device (100) in a communication manner and provide various control signals to the video projection device (100). At this time, the remote control device (200) may include a device that establishes a communication network with the video projection device (100) and transmits various control signals to the video projection device (100) through the established communication network, or receives signals related to various operations processed in the video projection device (100) from the video projection device (100). In the present disclosure, the remote control device (200) is described as an example of a space remote control, but is not limited thereto.

[0036] The image projection device (100) can be connected to only a single remote control device (200) or can be connected to two or more remote control devices (200) simultaneously, and can change the projected image or adjust the state of the projected image based on the control signal provided from each remote control device (200).

[0037] Figure 2 is an internal block diagram of the image projection device of Figure 1.

[0038] Referring to FIG. 2, the image projection device (100) may include a memory (120), a camera (130), a communication interface (140), a signal processing device (170), an image output device (180), and / or a power supply unit (190).

[0039] The memory (120) may store programs for each signal processing and control within the signal processing device (170), or may store signal-processed image or data signals. For example, the memory (120) may store application programs designed for the purpose of performing various tasks that can be processed by the signal processing device (170), and may selectively provide some of the stored application programs upon request from the signal processing device (170).

[0040] Programs stored in the memory (120) are not particularly limited as long as they can be executed by the signal processing device (170).

[0041] Although the memory (120) of FIG. 2 is provided separately from the signal processing device (170), the scope of the present invention is not limited thereto, and the memory (120) may be included within the signal processing device (170).

[0042] The memory (120) may include at least one of volatile memory (e.g., DRAM, SRAM, SDRAM, etc.) or non-volatile memory (e.g., flash memory, hard disk drive (HDD), solid-state drive (SSD), etc.).

[0043] The camera (130) may include one or more cameras. The camera (130) may be embedded in the image projection device (100) or may be separately positioned. The camera (130) may be positioned facing the direction in which the image is projected from the image projection device (100). The camera (130) may capture an image of the real background on which the image is projected. Image information captured by the camera (130) may be input to the signal processing device (170).

[0044] The communication interface (140) can serve as an interface with any external device or network connected to the image projection device (100) via wire or wirelessly. The communication interface (140) can receive data or power from such external devices and transmit it to each component within the image projection device (100), and can enable data within the image projection device (100) to be transmitted to the external device.

[0045] The communication interface (140) can receive a wireless signal from a mobile terminal (not shown). Here, the wireless signal can include various types of data, such as a voice call signal, a video call signal, text data, or image data. Meanwhile, the communication interface (140) can receive a control signal from a remote control device (200).

[0046] The communication interface (140) may be equipped with a short-range communication device (not shown). Short-range communication technologies such as Bluetooth, RFID (Radio Frequency Identification), infrared communication (IrDA, infrared Data Association), UWB (Ultra-Wideband), ZigBee, and NFC (Near Field Communication) may be used.

[0047] The signal processing device (170) may include at least one processor. Here, the processor may be a general processor such as a central processing unit (CPU). Of course, the processor may be a dedicated device such as an ASIC or another hardware-based processor.

[0048] The signal processing device (170) can control the overall operation of the image projection device (100). The signal processing device (170) can control the operation of each unit within the image projection device (100). The signal processing device (170) can be referred to as a control unit (170).

[0049] The signal processing device (170) can control a video image stored in the memory (120) or a video image received from the outside through the communication interface (140) to be output to the outside as a projection image.

[0050] The signal processing device (170) can control the image output device (180). The image output device (180) can be equipped with a driving device (185) and / or an optical device (210). The signal processing device (170) can control the optical device (210) or the driving device (185) that outputs visible light such as R, G, and B. For example, the signal processing device (170) can output R, G, and B signals corresponding to a video image to the optical device (210) or the driving device (185).

[0051] The driving device (185) can drive the optical device (210). For example, the driving device (185) can drive a light source within the optical device (210).

[0052] The optical device (210) may include optical components such as a light source and a lens for light output, particularly visible light output. In particular, the optical device (210) according to the embodiment of the present disclosure can easily implement color separation and synthesis of light. This will be described later with reference to FIG. 4 and below.

[0053] The power supply unit (190) can supply power required for the operation of each component by receiving external power or internal power under the control of the signal processing device (170).

[0054] The power supply unit (190) supplies power to the entire image projection device (100). In particular, it can supply power to a signal processing device (170) that can be implemented in the form of a system on chip (SOC), an image output device (180) for image display, and an audio output unit (not shown) for audio output.

[0055] Meanwhile, although not illustrated in FIG. 2, the image projection device (100) may include an input unit. The input unit may be provided on one side of the main body of the image projection device (100). For example, the input unit may include a touch pad, a physical button, etc.

[0056] The input unit can receive various user commands related to the operation of the image projection device (100) and transmit a control signal corresponding to the input command to the signal processing device (170).

[0057] Meanwhile, the block diagram of the image projection device (100) illustrated in FIG. 2 is only a block diagram for one embodiment of the present disclosure, and each component of the block diagram may be integrated, added, or omitted depending on the specifications of the image projection device (100) actually implemented.

[0058] That is, two or more components may be combined into a single component, or a single component may be subdivided into two or more components, as needed. Furthermore, the functions performed by each block are intended to illustrate embodiments of the present invention, and their specific operations or devices do not limit the scope of the present invention.

[0059] Figure 3 is an internal block diagram of the control unit of Figure 2.

[0060] Referring to FIG. 3, a signal processing device (170) according to one embodiment of the present invention may include a demultiplexing unit (310), an image processing unit (320), a processor (330), an OSD generation unit (340), a mixer (345), a frame rate conversion unit (350), and / or a formatter (360). In addition, an audio processing unit (not shown) and a data processing unit (not shown) may be further included.

[0061] The demultiplexer (310) can demultiplex an input stream.

[0062] The image processing unit (320) can perform image processing of a demultiplexed image signal. To this end, the image processing unit (320) may be equipped with an image decoder (325) and a scaler (335).

[0063] The video decoder (325) can decode a demultiplexed video signal, and the scaler (335) can perform scaling so that the resolution of the decoded video signal can be output on the display (180).

[0064] The video decoder (325) can be equipped with decoders of various specifications.

[0065] The processor (330) can control the overall operation within the image projection device (100) or the signal processing device (170). In addition, the processor (330) can control the operation of the demultiplexing unit (310), the image processing unit (320), the OSD generation unit (340), etc. within the signal processing device (170).

[0066] The OSD generation unit (340) can generate an OSD signal based on user input or on its own. The OSD generation unit (340) can generate a pointer based on a pointing signal input from a remote control device (200). The OSD generation unit (340) can include a pointing signal processing unit (not shown) that generates the pointer. The pointing signal processing unit (not shown) can also be provided separately rather than being included within the OSD generation unit (240).

[0067] The mixer (345) can mix the OSD signal generated by the OSD generation unit (340) and the decoded image signal processed by the image processing unit (320). The mixed image signal can be provided to the frame rate conversion unit (350).

[0068] The frame rate converter (FRC) (350) can convert the frame rate of an input video. Meanwhile, the frame rate converter (350) can also output the video as is without a separate frame rate conversion.

[0069] Meanwhile, the formatter (360) can receive a mixed signal from the mixer (345), i.e., an OSD signal and a decoded image signal, and perform signal conversion for input to the image output device (180). For example, it can output a low voltage differential signal (LVDS).

[0070] Meanwhile, the block diagram of the signal processing device (170) illustrated in FIG. 3 is a block diagram for one embodiment of the present disclosure. Each component of the block diagram may be integrated, added, or omitted depending on the specifications of the signal processing device (170) actually implemented.

[0071] In particular, the frame rate converter (350) and the formatter (360) are not provided within the signal processing device (170), but may be provided separately, or may be provided separately as one module.

[0072] FIG. 4 is an example of the structure of an optical device of an image projection device according to one embodiment of the present disclosure.

[0073] Referring to FIG. 4, the optical device (210) may include a light source assembly (410) that outputs light, a condenser lens (420), a diffuser (430) that diffuses light passing through the condenser lens (420), a reflective mirror (440), a display element (450), a total internal reflection (TIR) ​​prism (460), an actuator (470), and / or an optical lens (480).

[0074] The light source assembly (410) may include at least one light source. For example, the light source assembly (410) may include a plurality of laser diodes that output red light, blue light, and green light, respectively. The light source assembly (410) may include at least one dichroic filter that reflects a portion of incident light and transmits the other portion. In this case, as the light output from the light source included in the light source assembly (410) is transmitted or reflected through the dichroic filter, color separation and synthesis of the light may be achieved.

[0075] The condenser lens (420) can change the path of light output from the light source assembly (410). The path of light of multiple colors output from the light source assembly (410) and passing through the condenser lens (420) can be directed toward the focus of the condenser lens (420).

[0076] The reflective mirror (440) can reflect light passing through the diffuser (430) in a direction toward the display element (450).

[0077] The display element (450) can output a projection image based on light reflected by the reflective mirror (440). According to one embodiment, the display element (450) can include a digital micro-mirror device (DMD). The digital micro-mirror device can be configured by arranging a plurality of microscopic mirrors. In this case, each of the plurality of mirrors can function as a pixel to selectively reflect the light reflected by the reflective mirror (440) to implement a projection image. Since light loss is minimized by the operation of the micro-mirror, light efficiency is improved and color reproducibility is excellent, so that a 4K projection image or an 8K projection image can be output. Meanwhile, in the present disclosure, the display element (450) is described as an example of a digital micro-mirror device, but is not limited thereto. For example, the display element (450) can include a transparent display panel such as a liquid crystal display panel.

[0078] The optical lens (480) may include a projection lens. The optical lens can adjust the size of the projected image.

[0079] Meanwhile, although not shown in the drawing, the optical device (210) may further include a collimator lens, a fly-eye lens, an illumination lens, etc. Accordingly, the configuration of the optical device (210) can be implemented simply and high brightness performance can be secured.

[0080] Fig. 5 is an example of an internal block diagram of the remote control device of Fig. 1.

[0081] Referring to FIG. 5, the remote control device (200) may include a wireless communication unit (220), a user input unit (230), a sensor unit (240), an output unit (250), a power supply unit (260), a storage unit (270), and / or a signal processing unit (280).

[0082] The wireless communication unit (220) can transmit and receive signals with the image projection device (100).

[0083] In this embodiment, the remote control device (200) may be equipped with an RF module (221) capable of transmitting and receiving signals with the image projection device (100) according to RF (Radio frequency) communication standards. In addition, the remote control device (200) may be equipped with an IR module (223) capable of transmitting and receiving signals with the image projection device (100) according to IR (Infrared radiation) communication standards.

[0084] The remote control device (200) can transmit a signal including information about the movement of the remote control device (200) to the image projection device (100) through the RF module (221). The remote control device (200) can receive a signal transmitted by the image projection device (100) through the RF module (221).

[0085] The remote control device (200) can transmit commands regarding power on / off, size of projection image, etc. to the image projection device (100) through the IR module (223).

[0086] The user input unit (230) may be composed of a keypad, buttons, a touch pad, a touch screen, etc. The user can input commands related to the image projection device (100) using the remote control device (200) by operating the user input unit (230).

[0087] When the user input unit (230) has a hard key button, the user can input a command related to the image projection device (100) to the remote control device (200) through a push operation of the hard key button.

[0088] When the user input unit (230) is equipped with a touch screen, the user can input commands related to the image projection device (100) using the remote control device (200) by touching the soft keys of the touch screen.

[0089] Meanwhile, the user input unit (230) may be equipped with various types of input means that can be operated by the user, such as a scroll key or a jog key, and this embodiment does not limit the scope of the present invention.

[0090] The sensor unit (240) may be equipped with a gyro sensor (241) or an acceleration sensor (243). The gyro sensor (241) can sense the movement of the remote control device (200).

[0091] The gyro sensor (241) can sense information about the operation of the remote control device (200) based on the x, y, and z axes. The acceleration sensor (243) can sense information about the movement speed of the remote control device (200). Meanwhile, the sensor unit (240) may further include a distance measuring sensor capable of sensing the distance from the display (180).

[0092] The output unit (250) can output an image or sound corresponding to the operation of the user input unit (230) or to a signal transmitted from the image projection device (100). Through the output unit (250), the user can recognize whether the user input unit (230) is being operated or whether the image projection device (100) is being controlled.

[0093] The output unit (250) may include an LED module (251) including at least one light-emitting element (e.g., an LED (Light Emitting Diode)), a vibration module (253) that generates vibration, a sound output module (255) that outputs sound, and / or a display module (257) that outputs an image.

[0094] The power supply unit (260) can supply power to each component provided in the remote control device (200). The power supply unit (260) can include at least one battery (not shown).

[0095] The power supply unit (260) can prevent unnecessary power consumption by stopping the power supply to each component provided in the remote control device (200) when movement of the remote control device (200) is not detected for a predetermined period of time through the sensor unit (240).

[0096] The power supply unit (260) can resume power supply to each component equipped in the remote control device (200) when a predetermined event occurs. For example, the power supply unit (260) can resume power supply to each component when a predetermined key equipped in the remote control device (200) is operated. For example, the power supply unit (260) can resume power supply to each component equipped in the remote control device (200) when movement of the remote control device (200) is detected through the sensor unit (240).

[0097] The storage unit (270) can store various types of programs, application data, etc. required for the control or operation of the remote control device (200).

[0098] When the remote control device (200) wirelessly transmits and receives signals through the image projection device (100) and the RF module (221), the remote control device (200) and the image projection device (100) can transmit and receive signals through a predetermined frequency band. The signal processing device (280) of the remote control device (200) can store and reference information about the frequency band, etc., through which signals can be wirelessly transmitted and received between the remote control device (200) and the image projection device (100) paired therewith in the storage unit (270).

[0099] The signal processing device (280) may include at least one processor, and may control the overall operation of the remote control device (200) using the processor included therein.

[0100] The signal processing device (280) can transmit a control signal corresponding to a predetermined key operation of the user input unit (230) or a control signal corresponding to the movement of the remote control device (200) sensed by the sensor unit (240) to the image projection device (100) via the wireless communication unit (220).

[0101] The image projection device (100) can wirelessly transmit and receive signals with the remote control device (200). The image projection device (100) can receive a signal transmitted by the remote control device (200) according to the IR communication standard through the communication interface (140).

[0102] The image projection device (100) can calculate the coordinate values ​​(x, y) of the pointer corresponding to the remote control device (200) by correcting hand shake or errors from a signal corresponding to the operation of the remote control device (200) received through the communication interface (140).

[0103] A signal of a remote control device (200) input to an image projection device (100) can be transmitted to a signal processing device (170) of the image projection device (100). The signal processing device (170) of the image projection device (100) can check information about the operation and key operation of the remote control device (200) from the signal transmitted from the remote control device (200) and control the image projection device (100) in response thereto.

[0104] As another example, the remote control device (200) can calculate pointer coordinate values ​​corresponding to the operation and output them to the image projection device (100). In this case, the image projection device (100) can transmit information about the received pointer coordinate values ​​to the signal processing device (170) without a separate shake or error correction process.

[0105] FIGS. 6 and 7 are flowcharts of an operation method of an image projection device according to one embodiment of the present disclosure.

[0106] Referring to FIG. 6, the image projection device (100) can, in operation S610, generate a shape corresponding to an area where an image is projected on a real background (hereinafter, “projection area”). Here, the shape corresponding to the projection area (hereinafter, “projection shape”) may be a polygonal shape such as a triangle or a square. According to one embodiment, the image projection device (100) can project a user interface (UI) image including the projection shape. At this time, the user can set the shape, size, etc. of the projection shape using the remote control device (200).

[0107] Referring to FIGS. 8A and 8B, the image projection device (100) can output a UI image (800) to a real background (30) on which an image is projected. The UI image (800) can include a shape object (810) including a plurality of shapes, a tool object (820) including a tool for adjusting the color, shape, direction, position, etc. of the shape, etc. For example, the tool object (820) can include a tool for adjusting the color of the shape (821), a tool for displaying an image in an area corresponding to the shape (822), a tool for rotating the shape (823), a tool for symmetrically moving the shape (824), a tool for copying the shape (825), a tool for positioning the shape in front of another shape (826), a tool for positioning the shape behind another shape (827), a tool for deleting the shape (828), a tool for completing the settings for the shape (829), etc.

[0108] The user can move or rotate the remote control device (200) up and down, left and right, forward and backward. At this time, the pointer (205) included in the UI image (800) can be displayed in response to the movement of the remote control device (200). Since the pointer (205) moves and is displayed according to the movement of the remote control device (200) in three-dimensional space, the remote control device (200) can be called a space remote control or a 3D pointing device. Meanwhile, the movement speed or movement direction of the pointer (205) can correspond to the movement speed or movement direction of the remote control device (200).

[0109] A user can select one of the shapes included in the shape object (810) using a pointer (205). When the user selects a square using the pointer (205), a first shape (830) that is a square can be displayed through the UI image (800).

[0110] The user can adjust the positions of the vertices (831 to 834) of the first figure (830) using the pointer (205). For example, the user can select one of the vertices (831 to 834) of the first figure (830) using the pointer (205), and then adjust the position of the pointer (205) to adjust the position of the selected vertex. At this time, the image projection device (100) can determine the positions of the vertices (831 to 834) of the first figure (830) based on the position of the pointer (205). In addition, the image projection device (100) can determine the shape, position, size, etc. of the first figure (830) based on the positions of the vertices (831 to 834) of the first figure (830).

[0111] When a user selects a tool (829) to complete the settings for a shape on the UI screen (800) using a pointer (205), the image projection device (100) can determine the shape of the projected shape.

[0112] Referring again to FIG. 6, the image projection device (100) can determine, in operation S620, whether a projection shape is determined.

[0113] The image projection device (100) can process an image, which is a projection image, using a polygon mesh in operation S630. Here, the polygon mesh can mean a set of polygons such as triangles and squares and vertices that constitute an object. For example, the image projection device (100) can process an image, which is a projection image, based on the result of processing a projection shape using the polygon mesh. In the present disclosure, processing an image, which is a projection image, using a polygon mesh including a plurality of triangles will be described as an example. In this regard, the description will be made with reference to FIG. 7.

[0114] Referring to FIG. 7, the image projection device (100) can generate a plurality of first triangles constituting a projection shape in operation S710. For example, the image projection device (100) can determine the number of the plurality of first triangles in response to a preset resolution. In this case, the greater the resolution, the greater the number of the plurality of first triangles.

[0115] According to one embodiment, the image projection device (100) can determine the coordinates of the vertices of a plurality of first triangles constituting the projection figure based on linear interpolation. The image projection device (100) can determine the coordinates of a plurality of points located on the lines connecting the vertices of the projection figure. At this time, the image projection device (100) can determine the coordinates of the plurality of points according to a preset resolution. For example, when the preset resolution is 2, the image projection device (100) can determine the coordinates of a plurality of points dividing each of the line segments into two. For example, when the preset resolution is 3, the image projection device (100) can determine the coordinates of a plurality of points dividing each of the line segments into three.

[0116] The image projection device (100) can generate a plurality of rectangles constituting the projection figure by connecting a plurality of points located on the vertices and / or line segments of the projection figure. At this time, the image projection device (100) can generate a plurality of first triangles constituting the projection figure by dividing each of the plurality of rectangles constituting the projection figure into two triangles using a diagonal.

[0117] Referring to FIG. 9, when the projection shape (900) is a square, the image projection device (100) can determine the coordinates of a plurality of points located on the lines connecting the vertices (901 to 904) of the projection shape (900). For example, the coordinates of two points (911, 912) located on the first line segment connecting the first vertex (901) and the second vertex (902), two points (921, 922) located on the second line segment connecting the second vertex (902) and the third vertex (903), two points (931, 932) located on the third line segment connecting the third vertex (903) and the fourth vertex (904), and two points (941, 942) located on the fourth line segment connecting the first vertex (901) and the fourth vertex (904) can be determined.

[0118] The image projection device (100) can create a plurality of rectangles that constitute a projection shape by using a plurality of additional line segments that connect two points (911, 912) located on a first line segment and two points (931, 932) located on a third line segment, and connect two points (921, 922) located on a second line segment and two points (941, 942) located on a fourth line segment. At this time, the vertices (901 to 904) of the projection figure (900), the plurality of points (911, 912, 921, 922, 931, 932, 941, 942) located on the first to fourth line segments, and the plurality of points (951 to 954) where the plurality of additional line segments intersect may correspond to the vertices of the plurality of squares constituting the projection figure (900).

[0119] The image projection device (100) can generate a plurality of first triangles constituting the projection figure (900) by dividing each of the plurality of squares constituting the projection figure (900) into two triangles using a diagonal. For example, the first square (960) constituting the projection figure (900) can be divided into two triangles (961, 962) by a diagonal connecting the first vertex (901) and the third vertex (951).

[0120] Referring back to FIG. 7, the image projection device (100) may determine texture coordinates corresponding to the vertices of each of the plurality of first triangles constituting the projection shape in operation S720. For example, the image projection device (100) may determine texture coordinates for the coordinates of the vertices of each of the plurality of first triangles using a UV mapping technique. Meanwhile, in the present disclosure, determining texture coordinates using a UV mapping technique is described as an example, but is not limited thereto. For example, various mapping techniques such as Mip Mapping, Normal Mapping, Bump Mapping, Displacement Mapping, Procedural Mapping, Light Mapping, Reflection Mapping, and Refraction Mapping may be used.

[0121] The image projection device (100) can perform coordinate transformation on the image, which is a projection image, in operation S730. For example, the image projection device (100) can transform the coordinates of pixels of the image, which is a projection image, using affine transformation.

[0122] At this time, the image projection device (100) can convert the coordinates of the pixels of the image, which is a projection image, based on the coordinates and texture coordinates of the vertices of the first triangles corresponding to the pixels of the image, which is a projection image, among the plurality of first triangles constituting the projection shape. For example, the image projection device (100) can determine the coordinates and texture coordinates of the vertices of the first triangles corresponding to the pixels of the image, which is a projection image, based on the plurality of second triangles constituting the image, which are projection images, corresponding to the plurality of first triangles constituting the projection shape.

[0123] According to one embodiment, the image projection device (100) can convert the coordinates of pixels of an image, which is a projection image, based on the following mathematical expression 1.

[0124]

[0125] In the above mathematical expression 1, (x, y) may be the coordinates of a pixel of an image which is a projected image, and (x', y') may be the coordinates of a transformed pixel. Meanwhile, m11, m12, m21, m22, dx, and dy included in the transformation matrix may be determined based on the following mathematical expression 2.

[0126]

[0127] In the above mathematical expression 2, (sx0, sy0), (sx1, sy1), and (sx2, sy2) may be coordinates of vertices of the first triangle constituting the projected shape, and (p0x, p0y), (p1x, p1y), and (p2x, p2y) may be texture coordinates corresponding to the vertices of the first triangle constituting the projected shape.

[0128] Referring to FIGS. 10A to 10C, the image projection device (100) can output a projection image corresponding to the shape of the projection figure (1000). The user can set the shape of the projection figure (1000) using the remote control device (200). The shape of the projection figure (1000) can be changed corresponding to the user changing the positions of the vertices (1001 to 1004) of the projection figure (1000) using the remote control device (200).

[0129] In response to a change in the shape of the projection figure (1000), the coordinates of a plurality of first triangles (1050) constituting the projection figure (1000) may be changed. In addition, in response to a change in the shape of the projection figure (1000), the texture coordinates corresponding to a plurality of first triangles (1050) constituting the projection figure (1000) may be changed. At this time, the transformed coordinates for pixels of the projection image may be changed in response to a change in the shape of the projection figure (1000).

[0130] Meanwhile, referring to FIGS. 11A to 11C, as the resolution increases, the number of the plurality of first triangles (1150) constituting the projection figure (1000) may increase. As the number of the plurality of first triangles (1150) constituting the projection figure (1000) increases, the coordinates of the vertices of the plurality of first triangles (1150) may change more precisely in response to changes in the shape of the projection figure (1000). In addition, as the number of the plurality of first triangles (1150) constituting the projection figure (1000) increases, the transformed coordinates for the pixels of the projection image may change more precisely.

[0131] Referring again to FIG. 6, the image projection device (100) can output an image processed using a polygon mesh in operation S640. For example, the image projection device (100) can project an image whose coordinates have been transformed using affine transformation toward a projection area.

[0132] Referring to FIGS. 12 and 13, the coordinates of pixels of the projection image (1200) can be converted to correspond to the shape of the projection shape (900). At this time, the shape (1210) of the projection image (1200) can correspond to the shape of the projection shape (900).

[0133] As the user sets the shape of the first figure (830) displayed on the UI image (800) to correspond to the first wall surface (31) of the real background (30), the shape of the projected figure (900) can correspond to the first wall surface (31). In addition, the shape of the projected image (1200) corresponding to the shape of the projected figure (900) can also correspond to the first wall surface (31) of the real background (30). Therefore, when the image projection device (100) projects the projected image (1200) toward the real background (30), the projected image (1200) can be displayed on the first wall surface (31) of the real background (30) without distortion.

[0134] Meanwhile, referring to FIGS. 14a and 14b, a user can select one of the shapes included in a shape object (810) using a pointer (205). When the user selects a hexagon using the pointer (205), a second shape (840), which is a hexagon, can be displayed through a UI image (800).

[0135] The user can adjust the positions of the vertices (841 to 846) of the second figure (840) using the pointer (205). For example, the user can select one of the vertices (841 to 846) of the second figure (840) using the pointer (205), and then adjust the position of the pointer (205) to adjust the position of the selected vertex. At this time, the image projection device (100) can determine the positions of the vertices (841 to 846) of the second figure (840) based on the position of the pointer (205). In addition, the image projection device (100) can determine the shape, position, size, etc. of the second figure (840) based on the positions of the vertices (841 to 846) of the second figure (840).

[0136] According to one embodiment, a user can divide a second shape (840) into a plurality of detailed shapes by using a tool object (820) included in a UI image (800). For example, the user can divide the second shape (840) into a plurality of detailed shapes by using a diagonal line connecting two (842, 845) of a plurality of vertices (841 to 846) of the second shape (840) by using the tool object (820). In this case, the user can set a plurality of images to be displayed in each of the plurality of detailed shapes.

[0137] Referring to FIGS. 15 and 16, as the user sets the shape of the second figure (840) displayed on the UI image (800) to correspond to the first wall surface (31) and the second wall surface (32) of the real background (30), the shape of the projected figure may correspond to the first wall surface (31) and the second wall surface (32). In addition, one shape among the plurality of detailed figures may correspond to the first wall surface (31), and the other shape may correspond to the second wall surface (32).

[0138] Meanwhile, the projection image (1500) may be composed of a plurality of different images (1510, 1520). For example, the projection image (1500) may be composed of a first image (1510) and a second image (1520) each stored in the memory (120). For example, the projection image (1500) may be composed of a first image (1510) received from a first external device and a second image (1520) received from a second external device.

[0139] At this time, the image projection device (100) can convert the coordinates of the pixels of the first image (1510) to correspond to one shape among the plurality of detailed shapes, and can convert the coordinates of the pixels of the second image (1520) to correspond to another shape among the plurality of detailed shapes. The shape of the first image (1510) projected toward the first wall surface (31) can correspond to the first wall surface (31), and the shape of the second image (1520) projected toward the second wall surface (32) can correspond to the second wall surface (32). Therefore, when the image projection device (100) projects the projection image (1500) toward the real background (30), the first image (1510) can be displayed without distortion on the first wall surface (31) of the real background (30), and the second image (1520) can be displayed without distortion on the second wall surface (32).

[0140] Figure 17 is a flowchart illustrating an operating method of an image projection device according to another embodiment of the present disclosure. Any details that overlap with those described in Figures 6 to 16 will be omitted for detailed explanation.

[0141] Referring to FIG. 17, the image projection device (100) can capture a projection area through a camera (130) in operation S1710.

[0142] The image projection device (100) can extract features of the projection area based on an image acquired by photographing the projection area in operation S1720. For example, the image projection device (100) can identify an object included in the projection area. For example, the image projection device (100) can extract feature points of an object included in the projection area, such as edges, surfaces, and vertices that constitute the object.

[0143] According to one embodiment, the image projection device (100) can extract features of a projection area using a learning model learned through machine learning. Machine learning can refer to a field that defines various problems dealt with in the field of artificial intelligence and studies methodologies for solving them. An artificial neural network (ANN) is a model used in machine learning and can refer to a general model with problem-solving capabilities that is composed of artificial neurons (nodes) that form a network by combining synapses. An artificial neural network can be defined by a connection pattern between neurons in different layers, a learning process that updates model parameters, and an activation function that generates an output value. An artificial neural network can include an input layer, an output layer, and optionally one or more hidden layers. Each layer includes one or more neurons, and an artificial neural network can include synapses that connect neurons to neurons. In an artificial neural network, each neuron can output a function value of an activation function for input signals, weights, and biases received through synapses. Model parameters refer to parameters determined through learning, including the weights of synaptic connections and the biases of neurons. Hyperparameters refer to parameters that must be set before learning in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function. Among artificial neural networks, machine learning implemented with a deep neural network (DNN) that includes multiple hidden layers is also called deep learning, and deep learning is a subset of machine learning. Hereinafter, machine learning is used to include deep learning.

[0144] Object detection models using machine learning include the single-stage YOLO (You Only Look Once) model and the two-stage Faster R-CNN (Regions with Convolution Neural Networks) model.

[0145] The YOLO (You Only Look Once) model is a model that can predict objects and their locations within an image with just one look. The YOLO (You Only Look Once) model divides the original image into grids of equal size. For each grid, it predicts the number of bounding boxes in a predefined shape centered around the grid center, and calculates a confidence level based on this prediction. Then, whether the image contains an object or is just a background is included, and locations with high object confidence are selected to identify the object category.

[0146] The Faster R-CNN (Regions with Convolution Neural Networks) model is a model that can detect objects faster than the RCNN model and the Fast RCNN model. The Faster R-CNN (Regions with Convolution Neural Networks) model will be described in detail. First, a feature map is extracted from an image using a CNN (Convolution Neural Network) model. Based on the extracted feature map, multiple regions of interest (RoIs) are extracted. RoI pooling is performed for each region of interest. RoI pooling is a process of setting a grid to match a predetermined size of H x W for the feature map onto which the region of interest is projected, and extracting the largest value for each cell included in each grid to extract a feature map of size H x W. A feature vector is extracted from the feature map of size H x W, and object identification information can be obtained from the feature vector.

[0147] The image projection device (100) can determine a projection shape corresponding to the projection area in operation S1730. For example, the image projection device (100) can determine the shape, position, size, etc. of the projection shape based on the vertices, edges, etc. of objects included in the projection area.

[0148] Referring to FIG. 18, the image projection device (100) can capture a real background (30) through a camera (130) to obtain an image (1800) of a projection area. At this time, the image projection device (100) can extract features of the projection area based on the image (1800) of the projection area. The image projection device (100) can extract surfaces (1801 to 1806), edges (1811 to 1818), vertices (1821 to 1825), etc., that constitute the projection area from the image (1800) of the projection area.

[0149] The image projection device (100) can determine a projection shape based on features extracted from the projection area. For example, the image projection device (100) can determine a projection shape according to the shape of the first surface (1801) corresponding to the first wall surface (31), based on corners (1811 to 1813), vertices (1821, 1822), etc. constituting the first wall surface (31).

[0150] Meanwhile, when the projection image is composed of multiple different images, the image projection device (100) can determine multiple detailed shapes based on features extracted for the projection area. For example, the image projection device (100) can determine a first detailed shape according to the shape of the first surface (1801) corresponding to the first wall surface (31), based on corners (1811 to 1813), vertices (1821, 1822), etc. constituting the first wall surface (31), and can determine a second detailed shape according to the shape of the second surface (1802) corresponding to the second wall surface (32), based on corners (1811 1814, 1815), vertices (1821, 1823), etc. constituting the second wall surface (32).

[0151] Referring again to FIG. 17, the image projection device (100) can process an image, which is a projection image, using a polygon mesh in operation S1740.

[0152] The image projection device (100) can output an image processed using a polygon mesh in operation S1750.

[0153] As described above, according to at least one embodiment of the present disclosure, since the image is processed in correspondence to the real background (30) in the image projection device (100), the image can be projected onto various real backgrounds (30) without distortion.

[0154] Additionally, according to at least one embodiment of the present disclosure, by using a polygon mesh, an image can be processed in response to changes in a real background (30) on which the image is projected.

[0155] Additionally, according to at least one embodiment of the present disclosure, a user can freely set a real background (30) on which an image is projected.

[0156] Additionally, according to at least one embodiment of the present disclosure, the image can be processed by automatically identifying the real background (30) on which the image is projected using a camera (130).

[0157] Referring to FIGS. 1 to 18, an image projection device (100) according to one aspect of the present disclosure includes a light source assembly (410) including at least one light source; a display element (440) that outputs an image based on light output from the light source assembly (410); an optical lens (480) that projects the image output from the display element (440) onto a real background (30); and a control unit (170). The control unit (170) determines a shape corresponding to a predetermined area of ​​the real background (30) on which the image is projected, processes an image that is a projection target based on a result of processing the shape using a polygon mesh, and controls the display element (440) so that a predetermined image corresponding to the processed image is output.

[0158] In addition, according to one aspect of the present disclosure, the control unit (170) can output a user interface (UI) image (800) that sets the shape through the display element (440), and determine the shape based on a user input that adjusts a vertex of the shape included in the UI image (800).

[0159] In addition, according to one aspect of the present disclosure, a camera (130) is further included that is positioned toward the real background (30), and the control unit (170) can extract features of the real background (30) based on an image acquired through the camera (130) and determine the shape based on the extracted features.

[0160] In addition, according to one aspect of the present disclosure, the control unit (170) can generate a plurality of triangles constituting the shape, determine a plurality of texture coordinates corresponding to the plurality of triangles, and convert coordinates of pixels of the image based on the plurality of texture coordinates.

[0161] Additionally, according to one aspect of the present disclosure, the control unit (170) can determine the coordinates of the vertices of the plurality of triangles based on linear interpolation.

[0162] In addition, according to one aspect of the present disclosure, the control unit (170) can determine the plurality of texture coordinates corresponding to the coordinates of the vertices of each of the plurality of triangles using a UV mapping technique.

[0163] In addition, according to one aspect of the present disclosure, the control unit (170) can transform the coordinates of pixels of the image using an affine transformation based on the coordinates of the vertices of the plurality of triangles and the plurality of texture coordinates.

[0164] In addition, according to one aspect of the present disclosure, when the shape is composed of a plurality of detailed shapes, the control unit (170) may process each of the plurality of detailed shapes using the polygon mesh, process a first image based on a result of processing a first detailed shape, process a second image based on a result of processing a second detailed shape, and control the display element (440) so that the predetermined images corresponding to the processed first image and the processed second image are output.

[0165] An operating method of an image projection device (100) according to one aspect of the present disclosure may include an operation of determining a shape corresponding to a predetermined area of ​​a real background (30) on which an image is projected through an optical lens (480); an operation of processing an image as a projection target based on a result of processing the shape using a polygon mesh; and an operation of outputting a predetermined image corresponding to the processed image.

[0166] In addition, according to one aspect of the present disclosure, the operation of determining the shape may include an operation of outputting a user interface (UI) image that sets the shape; and an operation of determining the shape based on a user input that adjusts a vertex of the shape included in the UI image (800).

[0167] In addition, according to one aspect of the present disclosure, the operation of determining the shape may include an operation of extracting a feature of the real background (30) based on an image acquired through a camera (130) positioned toward the real background (30); and an operation of determining the shape based on the extracted feature.

[0168] In addition, according to one aspect of the present disclosure, the operation of processing the image as the projection target may include an operation of generating a plurality of triangles constituting the shape; an operation of determining a plurality of texture coordinates corresponding to the plurality of triangles; and an operation of converting coordinates of pixels of the image based on the plurality of texture coordinates.

[0169] Additionally, according to one aspect of the present disclosure, the operation of generating the plurality of triangles may include an operation of determining coordinates of vertices of the plurality of triangles based on linear interpolation.

[0170] Additionally, according to one aspect of the present disclosure, the operation of determining the plurality of texture coordinates may include an operation of determining the plurality of texture coordinates corresponding to the coordinates of the vertices of each of the plurality of triangles using a UV mapping technique.

[0171] Additionally, according to one aspect of the present disclosure, the operation of transforming the coordinates of pixels of the image may include an operation of transforming the coordinates of pixels of the image using an affine transformation based on the coordinates of vertices of the plurality of triangles and the plurality of texture coordinates.

[0172] The attached drawings are only intended to facilitate understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present disclosure.

[0173] Meanwhile, the operating method of the present disclosure can be implemented as processor-readable code on a processor-readable recording medium. A processor-readable recording medium includes all types of recording devices that store data that can be read by a processor. Examples of processor-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage devices, etc., and also include those implemented in the form of a carrier wave, such as transmission via the Internet. Furthermore, the processor-readable recording medium can be distributed across network-connected computer systems, so that the processor-readable code can be stored and executed in a distributed manner.

[0174] In addition, although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. A light source assembly comprising at least one light source; A display element that outputs an image based on light output from the above light source assembly; An optical lens that projects the image output from the display element onto a real background; and Including a control unit, The above control unit, Determine a shape corresponding to a predetermined area of ​​the above-mentioned real background on which the above-mentioned image is projected, Based on the result of processing the above shape using a polygon mesh, the image, which is the projection target, is processed, An image projection device characterized by controlling the display element so that a predetermined image corresponding to the processed image is output.

2. In paragraph 1, The above control unit, Outputting a user interface (UI) image that sets the shape through the above display element, An image projection device characterized in that the shape is determined based on a user input that adjusts a vertex of the shape included in the UI image.

3. In paragraph 1, Further comprising a camera positioned towards the above-mentioned real background, The above control unit, Based on the image acquired through the above camera, the features of the real background are extracted, An image projection device characterized in that the shape is determined based on the extracted features.

4. In paragraph 1, The above control unit, Generate multiple triangles that make up the above shape, Determine a plurality of texture coordinates corresponding to the plurality of triangles, An image projection device characterized in that the coordinates of pixels of the image are converted based on the plurality of texture coordinates.

5. In paragraph 4, The above control unit, An image projection device characterized in that the coordinates of the vertices of the plurality of triangles are determined based on linear interpolation.

6. In paragraph 4, The above control unit, An image projection device characterized in that the plurality of texture coordinates corresponding to the coordinates of the vertices of each of the plurality of triangles are determined using a UV mapping technique.

7. In paragraph 4, The above control unit, An image projection device characterized in that the coordinates of pixels of the image are transformed using an affine transformation based on the coordinates of the vertices of the plurality of triangles and the plurality of texture coordinates.

8. In paragraph 1, The above control unit, When the above shape is composed of multiple detailed shapes, each of the multiple detailed shapes is processed using the polygon mesh, Based on the result of processing the first detailed shape, the first image is processed, Based on the result of processing the second detailed shape, the second image is processed, An image projection device characterized by controlling the display element so that the predetermined image corresponding to the processed first image and the processed second image is output.

9. In the operating method of the video projection device, The act of determining a shape corresponding to a given area of ​​a real background onto which an image is projected through an optical lens; An operation of processing an image as a projection target based on the result of processing the shape using a polygon mesh; and An operating method of an image projection device, characterized by including an operation of outputting a predetermined image corresponding to the processed image.

10. In paragraph 9, The action that determines the above shape is, An action of outputting a user interface (UI) image that sets the above shape; and An operating method of an image projection device, characterized by including an action of determining the shape based on a user input that adjusts a vertex of the shape included in the UI image.

11. In paragraph 9, The action that determines the above shape is, An operation of extracting features of a real background based on an image acquired through a camera positioned toward the real background; and An operating method of an image projection device, characterized by including an operation of determining the shape based on the extracted features.

12. In paragraph 9, The operation of processing the image, which is the projection target, is as follows: An action for generating a plurality of triangles constituting the above shape; An operation for determining a plurality of texture coordinates corresponding to the plurality of triangles; and An operating method of an image projection device, characterized by including an operation of converting coordinates of pixels of the image based on the plurality of texture coordinates.

13. In paragraph 12, The operation of generating the above multiple triangles is: An operating method of an image projection device, characterized by including an operation of determining coordinates of vertices of the plurality of triangles based on linear interpolation.

14. In paragraph 12, The operation of determining the above multiple texture coordinates is: An operating method of an image projection device, characterized by including an operation of determining a plurality of texture coordinates corresponding to the coordinates of each vertex of the plurality of triangles using a UV mapping technique.

15. In paragraph 12, The operation of converting the coordinates of the pixels of the above image is, An operating method of an image projection device, characterized by including an operation of transforming coordinates of pixels of the image using an affine transformation based on coordinates of vertices of the plurality of triangles and the plurality of texture coordinates.

Images