Image projection device and control method thereof

Abstract

An image projection device and a control method thereof may be provided. Specifically, an image projection device and a control method thereof may be provided, the image projection device comprising: a first camera; a second camera; a light source; an image processor; and a processor, wherein the processor acquires characteristic information of a surface on which an image is projected, acquires viewpoint information related to a viewpoint at which a user views the image, generates a virtual image on the basis of the characteristic information and the viewpoint information, and projects the image on the surface on the basis of the virtual image and the user's position.

Description

Image projection device and control method

[0001] Embodiments of the present disclosure relate to an image projection device and a control method thereof.

[0002] An image projection device can be an electronic device that projects an image onto a surface separate from the image projection device. For example, the image projection device can be a beam projector that projects a screen or video onto a screen or wall separate from the image projection device. Image projection devices can project large images onto any surface other than a display, allowing multiple viewers to view the image without being restricted by space.

[0003] An image projection device can project an image assuming that the surface onto which the image is projected is a flat, uniform white surface. For example, the image projection device can project an image assuming a flat, uniform white screen or wall. If the surface onto which the image is projected is folded, if the surface onto which the image is projected has at least one curve, or if the surface onto which the image is projected contains at least one color, distortion may occur in the image projected by the image projection device. Accordingly, if the image projection device projects an image onto a wall with a curved portion, a curved surface such as a curtain, or a colored surface, distortion may occur in the projected image.

[0004] An image projection device according to one embodiment of the present disclosure includes a first camera, a second camera, a light source, an image processor, a memory including one or more computer programs, and at least one processor communicatively connected to the first camera, the second camera, the light source, the image processor, and the memory, wherein the one or more computer programs include instructions executable by a computer, and the instructions, when executed by the at least one processor, can cause the image projection device to obtain characteristic information of a surface on which an image is projected, obtain viewpoint information related to a viewpoint from which a user views the image, generate a virtual image based on the characteristic information and the viewpoint information, and project the image onto a surface based on the virtual image and a position of the user.

[0005] A method for controlling an image projection device according to one embodiment of the present disclosure may include an operation in which the image projection device acquires characteristic information of a surface on which an image is projected, an operation in which the image projection device acquires viewpoint information related to a viewpoint from which a user views an image, an operation in which the image projection device generates a virtual image based on the characteristic information and the viewpoint information, and an operation in which the image projection device projects an image on a surface based on the virtual image and the user's position.

[0006] FIG. 1 is a drawing showing an image projection device according to one embodiment of the present disclosure projecting an image.

[0007] FIG. 2 is a drawing showing an image projection device according to one embodiment of the present disclosure projecting an image.

[0008] Figure 3 is a drawing showing that an image projected by an image projection device is distorted.

[0009] Figure 4 is a drawing showing the cause of distortion of an image projected by an image projection device.

[0010] FIG. 5 is a block diagram illustrating an image projection device according to one embodiment of the present disclosure.

[0011] FIG. 6 is a flowchart illustrating a method for controlling an image projection device according to one embodiment of the present disclosure.

[0012] FIG. 7 is a flowchart illustrating a method for an image projection device according to one embodiment of the present disclosure to obtain characteristic information.

[0013] FIG. 8 is a flowchart illustrating a method for an image projection device according to one embodiment of the present disclosure to obtain viewpoint information.

[0014] FIG. 9 is a flowchart illustrating a method for projecting an image by an image projection device according to one embodiment of the present disclosure.

[0015] FIG. 10 is a drawing showing an image projection device according to one embodiment of the present disclosure calculating a distance.

[0016] FIG. 11 is a diagram illustrating an image projection device according to one embodiment of the present disclosure generating a virtual image.

[0017] The terms used in this disclosure will be briefly explained, and one embodiment of the present disclosure will be specifically described.

[0018] The terms used in this disclosure are selected from widely used, current terms, taking into account the functions of one embodiment of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of the disclosure.

[0019] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like described herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0020] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, one embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted to clearly describe one embodiment of the present disclosure, and similar parts are designated with similar drawing reference numerals throughout the present disclosure.

[0021] All functions or operations described in the present disclosure may be processed by a single processor or a combination of processors. The single processor or the combination of processors may include a circuit that performs processing. The single processor or the combination of processors may include at least one of an application processor (AP), a central processing unit (CPU), a communication processor (CP), a graphics processing unit (GPU), a neural processing unit (NPU), an AI chip, a Wi-Fi chip, a Bluetooth chip, a GPS chip, a near field communication (NFC) chip, a sensor controller, a touch controller, a fingerprint controller, a display driver integrated circuit (DDIC), an audio codec chip, a USB controller, a camera controller, an image processing IC, a microprocessor unit, and a system on a chip (SoC).

[0022] FIG. 1 is a drawing showing an image projection device (110) according to one embodiment of the present disclosure projecting an image.

[0023] The image projection device (110) may be an electronic device that projects an image onto a surface spaced apart from the image projection device (110). For example, the image projection device may be a beam projector that projects a screen or video onto a screen or wall spaced apart from the image projection device (110) to display the image. The image projection device (110) may project a large-sized image onto any surface other than a display, thereby allowing multiple viewers to view the image with less spatial constraints. In particular, the image projection device (110) according to one embodiment of the present disclosure may improve the quality of an image projected onto a surface in various environments.

[0024] An image projection device (110) according to one embodiment of the present disclosure may include a plurality of built-in projectors (111). The image projection device (110) may include at least one built-in camera. The image projection device (110) may have a projection area of ​​180 degrees or more and 360 degrees or less. The image projection device (110) may be placed on a ceiling or wall inside a house. For example, the image projection device (110) may be attached to a ceiling inside a house. For example, the image projection device (110) may also be attached to a side wall inside a house.

[0025] An image projection device (110) according to one embodiment of the present disclosure can project an image onto a surface having a folded portion. For example, an image projection device (110) according to one embodiment of the present disclosure can project an image onto a wall having folded portions by an upper surface, a lower surface, a left surface, and a right surface, thereby projecting an image including a main portion (120), a first edge portion (121), a second edge portion (122), a third edge portion (123), and a fourth edge portion (124). The image projection device (110) can reduce distortion caused by the folded portion in the projected image.

[0026] An image projection device (110) according to one embodiment of the present disclosure can seamlessly stitch images projected from a plurality of built-in projectors (111). An image projection device (110) according to one embodiment of the present disclosure can correct a projected image by reflecting the user's position in real time.

[0027] FIG. 2 is a drawing showing an image projection device (110) according to one embodiment of the present disclosure projecting an image.

[0028] An image projection device (110) according to one embodiment of the present disclosure may be placed on the floor of an indoor space. For example, the image projection device (110) may be installed on the floor of a home. For example, the image projection device (110) may be installed on a desk, table, or chest of drawers in the home.

[0029] An image projection device (110) according to one embodiment of the present disclosure can project an image onto at least one surface. For example, the image projection device (110) can project a first image (210) onto a front wall and a second image (220) onto a left wall. The image projection device (110) can project different images onto multiple surfaces. The image projection device (110) can project images by considering the positions and angles of each of the multiple surfaces.

[0030] An image projection device (110) according to one embodiment of the present disclosure can project an image onto a surface isolated from various external environments. For example, the image projection device (110) can project an image onto a surface in various lighting environments and in various color environments. The image projection device (110) can compensate for image distortion caused by the external environment. Since the image projection device (110) can project an image in various external environments, limitations caused by the external environment when utilizing the image projection device (110) can be reduced.

[0031] In particular, the image projection device (110) according to one embodiment of the present disclosure can project images onto various surfaces. For example, the image projection device (110) can project images onto curved surfaces such as walls with curved sections, curtains, or colored surfaces. The image projection device (110) can compensate for image distortion caused by the surface on which the image is projected. The image projection device (110) can project images onto surfaces other than flat and uniform white surfaces, thereby further increasing the usability of the image projection device (110).

[0032] Figure 3 is a drawing showing that an image projected by an image projection device is distorted.

[0033] The image projection device may be a non-linear device that projects an image in a form that is most visually appealing to the user. The image projection device may project the image onto a surface by considering that the human eye perceives the color and brightness of the image in a non-linear form and manner. The image projection device may include a plurality of image processors for processing the image to be projected onto the surface. The image projection device may process the image by assuming that the image to be projected is projected onto a flat, uniform white surface.

[0034] When an image projection device projects an image onto an uneven surface, distortion may occur in the projected image. For example, when an image projection device projects an image onto a curved and transparent surface, such as a curtain, the main image (310) that was originally intended to be projected and the distorted image (320) projected by the surface may be displayed as overlapping. In addition, when an image projection device projects an image onto a wall with a curved portion, an uneven surface, or a colored surface, the image that was originally intended to be projected may be projected onto the surface in a distorted manner.

[0035] The image projection device (110) according to the present disclosure can apply a method for solving the problem shown in FIG. 3.

[0036] Figure 4 is a drawing showing the cause of distortion of an image projected by an image projection device.

[0037] When an image projection device projects an image onto an uneven surface (410), the image projected may be distorted by the uneven surface (410) itself depending on the region. The distance from the image projection device to the uneven surface (410) may vary depending on the region of the uneven surface (410). For example, the distance from the image projection device to a concave portion of the uneven surface (410) may be longer than the distance from the image projection device to a convex portion of the uneven surface (410). Accordingly, the image projected onto the uneven surface (410) may be distorted such that it is reduced in the concave portion and enlarged in the convex portion.

[0038] The projected image may appear differently depending on the user's position from which the image is viewed. When the image projection device projects an image onto an uneven surface (410), the distortion occurring in the projected image may be more noticeable depending on the user's position. For example, if the user's position is adjacent to a curve on the uneven surface (410), the distortion occurring in the projected image may be more noticeable.

[0039] The image projection device (110) according to the present disclosure can apply a method for solving the problem shown in FIG. 4.

[0040] FIG. 5 is a block diagram illustrating an image projection device (110) according to one embodiment of the present disclosure. The image projection device (110) according to one embodiment may include a first camera (510), a second camera (520), a light source (530), an image processor (540), and a processor (550).

[0041] The first camera (510) can capture images of a surface on which the image projection device (110) projects images. The first camera (510) can obtain surface condition information. For example, the first camera (510) can obtain information related to whether the surface is flat or folded, information related to the curvature of the surface, and information related to the color of the surface. The first camera (510) can be an RGB camera or an infrared camera.

[0042] The second camera (520) can capture a surface on which the image projection device (110) projects an image. The second camera (520) can obtain depth information of the surface. For example, the second camera (520) can obtain information related to the distance from the image projection device (110) to the surface. The second camera (520) can capture a user. The second camera (520) can obtain location information of the user. The second camera (520) can obtain information related to the user's gaze looking at the surface. The second camera (520) can be a ToF (Time of Flight) camera.

[0043] The light source (530) can project an image projected by the image projection device (110) onto a surface. The light source (530) can project optical information capable of displaying an image on the surface. The light source (530) can project a pattern image for measuring the shape of the surface and the distance to the surface onto the surface. The light source (530) can include at least one projector. For example, the light source can include a plurality of projectors that project light containing optical information in different directions. The light source (530) can include a sensor module that measures the distance to the surface. For example, the light source (530) can include a Light Detection And Ranging (LIDAR) sensor.

[0044] The image processor (540) can generate an image to be projected onto a surface. The image processor (540) can generate an image based on surface state information and surface depth information. The image processor (540) can generate an image considering the user's position. The image processor (540) can generate an image considering whether the surface is folded, the curvature of the surface, and the color of the surface. The image processor (540) can compensate for distortion caused by whether the surface is folded, the curvature of the surface, and the color of the surface. The image processor (540) can determine the viewpoint of the image considering how the image is viewed from the user's position.

[0045] In one embodiment, the image processor (540) can calculate geometric structure information of a surface onto which an image is projected. The geometric structure information can include three-dimensional shape information of the surface, curvature information of the surface, and color information of the surface. The geometric structure information can include information related to the shape of the surface that cannot be acquired by the first camera (510) or the second camera (520). By calculating the geometric structure information of the surface in the image processor (540), the image processor (540) can consider distortion phenomena that are visible in the projected image when viewed from the user's perspective but cannot be recognized by the first camera (510) or the second camera (520).

[0046] The processor (550) can control the first camera (510), the second camera (520), the light source (530), and the image processor (540). The processor (550) can control the first camera (510) to capture a surface on which an image is to be projected. The processor (550) can receive surface state information acquired by the first camera (510). The processor (550) can control the second camera (520) to capture a surface on which an image is to be projected or a user. The processor (550) can receive depth information of the surface acquired by the second camera (510) and position information of the user. The processor (550) can control the light source (530) to project optical information onto the surface. The processor (550) can receive information related to the shape of the surface acquired by the light source and the distance to the surface. The processor (550) can transmit acquired surface status information, surface depth information, and user location information to the image processor (540). The processor (550) can receive an image generated by the image processor (540). The processor (550) can control the light source (530) to project the generated image onto the surface.

[0047] FIG. 6 is a flowchart illustrating a control method of an image projection device (110) according to one embodiment of the present disclosure.

[0048] In operation 610, an image projection device (110) according to an embodiment can obtain characteristic information of a surface on which an image is projected. The characteristic information of the surface can include information related to folds, curvatures, and colors on the surface. The surface on which an image is projected can have folds, such as corners between walls, curves, or colors, such as curtains. The image projection device (110) can identify folds, curvatures, and colors on the surface on which an image is projected. The image projection device (110) can obtain information related to folds, curvatures, and colors on the surface.

[0049] In operation 620, an image projection device (110) according to an embodiment can obtain viewpoint information related to a time point at which a user views an image. The viewpoint information may include a time point at which the user views an image from the user's location. The image projection device (110) can detect the user's location. For example, the image projection device (110) can capture the user's location. The image projection device (110) can obtain information related to the user's location. For example, the image projection device (110) can obtain the user's location as coordinate information. The image projection device (110) can calculate a time point at which the user views an image from the user's location.

[0050] In operation 630, an image projection device (110) according to an embodiment can generate a virtual image based on characteristic information and viewpoint information. The image projection device (110) can generate a virtual image to compensate for distortion that occurs in an image when projected onto a surface based on the characteristic information. For example, the image projection device (110) can generate a virtual image in which a bent portion is processed seamlessly to compensate for a distortion phenomenon in which an image is folded when projected onto a surface with a bent portion. For example, the image projection device (110) can generate a virtual image in which a bent portion is smoothly processed to compensate for a distortion phenomenon in which an image is folded when projected onto a surface with a curved portion. For example, the image projection device (110) can generate a virtual image in which a color effect opposite to the color of the surface is added to compensate for a distortion phenomenon in which an image is folded when projected onto a colored surface.

[0051] In operation 640, an image projection device (110) according to an embodiment can project an image onto a surface based on a virtual image and a user's position. The image projection device (110) can generate an image to be projected onto the surface based on the generated virtual image. The image projection device (110) can consider a viewpoint when viewing the image projected onto the surface from the user's position. For example, the image projection device (110) can calculate a distance from the user's position to the surface and an angle from the user's position to view the surface. The image projection device (110) can correct an image to be projected onto the surface by reflecting the distance from the user's position to the surface and the angle from the user's position to view the surface. The image projection device (110) can project the corrected image onto the surface.

[0052] FIG. 7 is a flowchart illustrating a method for an image projection device according to one embodiment of the present disclosure to obtain characteristic information.

[0053] In operation 710, an image projection device (110) according to an embodiment can project a pattern image onto a surface. The pattern image can be any image for obtaining surface condition information. For example, the pattern image can be an image including a dot pattern and a stripe pattern for identifying whether the surface is folded. For example, the pattern image can be an image including a wave pattern for identifying whether the surface is curved. The image projection device (110) can capture the pattern image projected onto the surface. The image projection device (110) can compare the pattern image originally intended to be projected with the pattern image actually projected onto the surface. The image projection device (110) can obtain surface condition information based on the comparison result.

[0054] In operation 720, an image projection device (110) according to an embodiment can correct the image projection device (110) based on a pattern image. The image projection device (110) can obtain surface condition information based on the pattern image. The image projection device (110) can predict the degree of distortion when an image is projected onto a surface based on the surface condition information. The image projection device (110) can correct the image projection device (110) based on the prediction result. For example, the image projection device (110) can adjust the intensity of light projected by the image projection device (110), the angle of the projected light, and the brightness of the projected light so as to compensate for the predicted degree of image distortion. For example, the image projection device (110) can control at least some of a plurality of projectors in the image projection device (110) to adjust the light projected on a portion of the image distorted by the surface.

[0055] In operation 730, an image projection device (110) according to an embodiment can obtain depth information of a surface based on a pattern image and a correction result. The image projection device (110) can capture a pattern image projected onto a surface after adjusting the pattern image and the light projected by the image projection device (110). The image projection device (110) can measure a distance from the image projection device (110) to the surface based on the pattern image correction result.

[0056] In operation 740, an image projection device (110) according to an embodiment can obtain color information and curvature information of a surface based on a pattern image and depth information. The color information of the surface can include information related to what color the surface is. The curvature information of the surface can include information related to what type of curvature is formed on the surface. The image projection device (110) can obtain a shape in which a pattern image is projected onto the surface. The image projection device (110) can consider depth information in the pattern image projected onto the surface. The image projection device (110) can analyze how the pattern image is distorted by the surface based on the pattern image projected onto the surface and the depth information. The image projection device (110) can identify what color the surface is and what type of curvature is formed on the surface based on the analysis result. The image projection device (110) can obtain color information and curvature information of the surface based on the identification result.

[0057] FIG. 8 is a flowchart illustrating a method for an image projection device (110) according to one embodiment of the present disclosure to obtain viewpoint information.

[0058] In operation 810, an image projection device (110) according to an embodiment can localize a user's location. The image projection device (110) can obtain location information related to the user's location. For example, the image projection device (110) can obtain the user's location information by taking a picture of the location where the user is. For example, the image projection device (110) can obtain the user's location information using a sensor module. For example, the image projection device (110) can receive the user's location information obtained by another external device. The image projection device (110) can store the user's location information. The image projection device (110) can localize the user's location based on the stored user's location information. For example, the image projection device (110) can convert the user's location into a coordinate value based on the image projection device (110). The image projection device (110) can localize the user's location by converting the user's location into a coordinate value using the current location of the image projection device (110) as a reference point.

[0059] In operation 820, an image projection device (110) according to an embodiment can obtain a viewpoint from which a user views an image based on a localized user's location. The image projection device (110) can calculate a distance from the localized user's location to a surface on which an image is projected. The image projection device (110) can calculate a distance from the localized user's location to the surface on which an image is projected. For example, the image projection device (110) can calculate a distance from the user's location to the surface using a LIDAR sensor. The image projection device (110) can calculate a direction of a viewpoint from which a user views the surface on which an image is projected from the localized user's location. For example, the image projection device (110) can obtain a direction of the user's gaze toward the surface by photographing the direction of the user's gaze toward the surface from the user's location. For example, the image projection device (110) can receive a direction toward the surface from the user's location obtained by another external device. The image projection device (110) can obtain the point of view from which the user views the image based on the distance from the user's position to the surface and the direction from the user's position toward the surface. The image projection device (110) can predict how the image projected on the surface will appear at the point of view from which the user views the image.

[0060] In operation 830, an image projection device (110) according to an embodiment can generate a virtual field of view corresponding to an uncapturable area based on depth information and the user's position. The uncapturable area may be a portion of an image projected onto a surface that the image projection device (110) cannot capture. For example, the uncapturable area may be a portion of an image that is intended to be projected onto a surface but is not projected due to the state of the surface. For example, the uncapturable area may be a portion of an image projected onto a surface that is at an angle that makes it impossible for a camera included in the image projection device (110) to capture. The image projection device (110) can assume a case where the image is viewed from the user's position based on distance information from the user's position to the surface included in the depth information and the user's position. The image projection device (110) can predict how the image will appear in the uncapturable area based on the prediction result.

[0061] FIG. 9 is a flowchart illustrating a method for projecting an image by an image projection device (110) according to one embodiment of the present disclosure.

[0062] In operation 910, an image projection device (110) according to an embodiment can project an image onto a surface so that it appears optimally when viewed from a user's position. The image projection device (110) can generate a virtual field of view of how the image appears when viewed from the user's position. The image projection device (110) can project an image onto a surface based on the virtual field of view.

[0063] In operation 920, the image projection device (110) according to an embodiment can identify whether the user's location has changed. The image projection device (110) can periodically obtain location information related to the user's location. For example, the image projection device (110) can periodically detect the location of the user using a LIDAR sensor. For example, the image projection device (110) can periodically receive location information from an external device. If the user's location is maintained (operation 920 - NO), the image projection device (110) can proceed to operation 930. If the user's location changes (operation 920 - YES), the image projection device (110) can proceed to operation 940.

[0064] In operation 930, the image projection device (110) according to an embodiment can maintain the shape of the image being projected. The image projection device (110) can maintain the shape of the image being projected that is optimized for the current user's position when the user's position is maintained.

[0065] In operation 940, an image projection device (110) according to an embodiment can correct the shape of a projected image to correspond to a changed position of a user. When the position of the user changes, the image projection device (110) can generate a virtual field of view corresponding to the changed position of the user. The image projection device (110) can correct the shape of a projected image based on the newly generated virtual field of view.

[0066] FIG. 10 is a drawing showing an image projection device (110) according to one embodiment of the present disclosure calculating a distance.

[0067] The image projection device (110) can calculate the distance to the surface (1030) on which the image is projected. The image projection device (110) can perform system calibration to calculate the distance to the surface (1030). The system calibration can include internal calibration using each of the first module and the second module (1010, 1020) included in the image projection device (110) and external calibration using the spatial relationship between the first module and the second module (1010, 1020). The first module and the second module (1010, 1020) can be two viewpoints for measuring the distance from the image projection device (110) to the surface (1030). For example, the first module and the second module (1010, 1020) can be the first camera (510) and the second camera (520). For example, the first module and the second module (1010, 1020) may be a first camera (510) and a light source (530). For example, the first module and the second module (1010, 1020) may be a second camera (520) and a light source (530).

[0068] The image projection device (110) can project a pattern image onto a surface (1030) that projects an image using the first module and the second module (1010, 1020). The pattern image can undergo pixel localization. Pixel localization can be a process in which the pattern image reflected from the surface (1030) is separated and analyzed on a pixel basis on the first virtual surface and the second virtual surface (1011, 1021). The image projection device (110) can obtain state information of the surface (1030) and depth information of the surface (1030) based on the result of pixel localization of the pattern image. The image projection device (110) can calculate a distance to the surface (1030) based on the state information of the surface (1030) and the depth information of the surface (1030).

[0069] The image projection device (110) can calculate the user's viewpoint information based on the image projection device (110) on the coordinate axis. For example, the image projection device (110) can photograph the user and calculate the user's position as a coordinate value with the first module and the second module (1010, 1020) as the origin. For example, the image projection device (110) can photograph the user and identify the direction the user is facing and calculate the unit vector value. For example, the image projection device (110) can assume the direction the user is facing as the direction toward the surface (1030).

[0070] FIG. 11 is a drawing showing an image projection device (110) according to one embodiment of the present disclosure generating a virtual image.

[0071] An image projection device (110) can project an image from a viewpoint (1110) toward a surface. The viewpoint (1110) can include a camera (e.g., a first camera (510) and a second camera (520)) and a light source (530). The image projection device (110) can project an image from the viewpoint (1110) toward a minimum distance surface (1120). The minimum distance surface (1120) can be a minimum distance for an image to be projected onto a surface. The image projection device (110) can generate a virtual image (1240) based on the image projected toward the minimum distance surface (1120). The image projection device (110) can project an image from the viewpoint (1110) toward a maximum distance surface (1130). The maximum distance surface (1130) can be a maximum distance at which a user can view an image projected onto a surface. The image projection device (110) can create a virtual volume (1250) based on an image projected toward a maximum distance surface (1130).

[0072] The image projection device (110) can acquire depth information of at least one surface (1120, 1130) when projecting an image. For example, the image projection device (110) can acquire depth information of at least one surface (1120, 1130) by analyzing the pixel distribution of the projected image after projecting a pattern image.

[0073] The image projection device (110) can generate a virtual image corresponding to the user's viewpoint and project it onto at least one surface (1120, 1130). The image projection device (110) can generate a virtual image corresponding to the user's viewpoint using the acquired depth information. The image projection device (110) can additionally obtain the user's location information. The image projection device (110) can generate a virtual image to be projected onto at least one surface (1120, 1130) so as to have a shape optimized for the user's viewpoint.

[0074] The present disclosure aims to provide an image projection device and a control method thereof that reduces the problem of distortion of a projected image even when the surface is uneven and projects an image to suit a user's viewpoint.

[0075] An image projection device (110) according to one embodiment of the present disclosure includes a first camera (510), a second camera (520), a light source (530), an image processor (540), a memory including one or more computer programs, and at least one processor (550) communicatively connected to the first camera, the second camera, the light source, the image processor, and the memory, wherein the one or more computer programs include instructions executable by a computer, and the instructions, when executed by the at least one processor, can cause the image projection device to obtain characteristic information of a surface on which an image is projected, obtain viewpoint information related to a viewpoint from which a user views the image, generate a virtual image based on the characteristic information and the viewpoint information, and project the image onto a surface based on the virtual image and a position of the user.

[0076] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to project a pattern image onto a surface and to correct the image projection device (110) based on the pattern image.

[0077] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to obtain depth information of the surface based on the pattern image and the result of the correction, and to obtain color information and curvature information of the surface based on the pattern image and the depth information.

[0078] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to localize a location of a user.

[0079] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to generate a virtual field of view corresponding to an uncapturable area based on depth information and a user's location.

[0080] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to project an image onto a surface so that the image appears in an optimal shape when viewed from the user's position, detect whether the user's position has changed, and correct the shape of the projected image to correspond to the changed user's position.

[0081] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to perform system calibration to calculate a distance to a surface on which to project an image.

[0082] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to obtain state information of a surface and depth information of the surface based on a pixel localization result of a pattern image, and to calculate a distance to the surface based on the state information and the depth information.

[0083] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to produce viewpoint information of a user relative to the image projection device (110) on a coordinate axis.

[0084] The instructions according to one embodiment of the present disclosure, when executed by the at least one processor, may cause the image projection device to generate a virtual image based on an image projected toward a minimum distance surface, and to generate a virtual volume based on an image projected toward a maximum distance surface.

[0085] A method for controlling an image projection device according to one embodiment of the present disclosure may include an operation in which the image projection device acquires characteristic information of a surface on which an image is projected, an operation in which the image projection device acquires viewpoint information related to a viewpoint from which a user views an image, an operation in which the image projection device generates a virtual image based on the characteristic information and the viewpoint information, and an operation in which the image projection device projects an image on a surface based on the virtual image and the user's position.

[0086] A method for controlling an image projection device according to one embodiment of the present disclosure may include an operation in which the image projection device projects a pattern image onto a surface and an operation in which the image projection device corrects the image projection device based on the pattern image.

[0087] A method for controlling an image projection device according to an embodiment of the present disclosure may include an operation in which the image projection device acquires depth information of a surface based on a turn image and a result of correction, and an operation in which the image projection device acquires color information and curvature information of the surface based on a pattern image and depth information.

[0088] A method for controlling an image projection device according to one embodiment of the present disclosure may include an operation of the image projection device localizing a user's location.

[0089] A method for controlling an image projection device according to one embodiment of the present disclosure may include an operation in which the image projection device generates a virtual field of view corresponding to an uncapturable area based on depth information and a user's position.

[0090] A method for controlling an image projection device according to one embodiment of the present disclosure may include an operation of projecting an image onto a surface so that the image projection device appears in an optimal form when viewed from a user's position, an operation of detecting whether the user's position has changed, and an operation of correcting the form of the image projected by the image projection device to correspond to the changed user's position.

[0091] A method for controlling an image projection device according to one embodiment of the present disclosure may include an operation in which the image projection device performs system calibration to calculate a distance to a surface on which an image is projected.

[0092] A method for controlling an image projection device according to an embodiment of the present disclosure may include an operation of the image projection device acquiring state information of a surface and depth information of the surface based on a pixel localization result of a pattern image, and an operation of calculating a distance to the surface based on the state information and depth information.

[0093] A method for controlling an image projection device according to one embodiment of the present disclosure may include an operation in which the image projection device calculates information about a user's viewpoint relative to the image projection device on a coordinate axis.

[0094] A method for controlling an image projection device according to an embodiment of the present disclosure may include an operation of generating a virtual image based on an image projected by the image projection device toward a minimum distance surface and an operation of generating a virtual volume based on an image projected toward a maximum distance surface.

[0095] The image projection device and the control method thereof according to the present disclosure can compensate for distortion of a projected image that occurs when the surface is uneven, curved, or colored by projecting an image while taking into account the characteristics of the surface.

[0096] In addition, the image projection device and the control method thereof according to the present disclosure can provide an improved image from the user's perspective by projecting an image that takes into account the user's viewpoint, thereby projecting an image suited to the user's viewpoint.

[0097] A method according to an embodiment of the present disclosure may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present disclosure or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0098] Some embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism, and includes any information delivery media. Furthermore, some embodiments of the present disclosure may also be implemented as a computer program or computer program product containing computer-executable instructions, such as a computer program that is executed by a computer.

[0099] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0100] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0101] The contents and operations described in this disclosure may be implemented in hardware, software, or a combination of hardware and software.

[0102] Software implementing the contents described in this disclosure may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may store one or more computer programs. The one or more computer programs may include computer-executable instructions. The computer-executable instructions, when executed by one or more processors, may cause an image projection device to perform the operations described in this disclosure.

[0103] One or more computer programs containing instructions executable by a computer, as described herein, may be stored in memory. The memory may be a volatile or non-volatile storage space or storage device. For example, the memory may be at least one of a Read Only Memory (ROM), a Random Access Memory (RAM), a memory chip, a Compact Disc (CD), a Digital Versatile Disc (DVD), a magnetic disk, or a magnetic tape. However, the memory is not limited thereto, and the memory may be any of various volatile or non-volatile storage media suitable for storing one or more computer programs. The one or more computer programs stored in the memory may include instructions executable by the computer. The computer-executable instructions, when executed by one or more processors, may cause the image projection device to perform the operations described herein.

Claims

1. In the image projection device (110), Camera 1 (510); Second camera (520); Light source (530); Image processor (540); a memory containing one or more computer programs; and At least one processor (550) communicatively connected to the first camera, the second camera, the light source, the image processor, and the memory, The one or more computer programs above contain instructions executable by a computer, The above instructions, when executed by the at least one processor, cause the image projection device to: Obtain characteristic information of the surface on which the image is projected, Obtain time information related to the time at which the user views the above video, Generate a virtual image based on the above characteristic information and the above viewpoint information, An image projection device (110) that projects the image onto the surface based on the virtual image and the user's location.

2. In paragraph 1, The above instructions, when executed by the at least one processor, cause the image projection device to: Projecting a pattern image onto the surface, An image projection device (110) for correcting the image projection device (110) based on the above pattern image.

3. In paragraph 2, The above instructions, when executed by the at least one processor, cause the image projection device to: Obtaining depth information of the surface based on the above pattern image and the result of the above correction, An image projection device (110) that obtains color information and curvature information of the surface based on the pattern image and the depth information.

4. In any one of paragraphs 1 to 3, The above instructions, when executed by the at least one processor, cause the image projection device to: An image projection device (110) that localizes the location of the user.

5. In paragraph 4, The above instructions, when executed by the at least one processor, cause the image projection device to: An image projection device (110) that generates a virtual field of view corresponding to an area that cannot be captured based on the depth information and the user's location.

6. In any one of paragraphs 1 to 5, The above instructions, when executed by the at least one processor, cause the image projection device to: Projecting said image onto said surface so that it appears in an optimal form when viewed from said user's position; Detect whether the location of the above user has changed, An image projection device (110) that corrects the shape of the image projected to correspond to the changed position of the user.

7. In any one of paragraphs 1 to 6, The above instructions, when executed by the at least one processor, cause the image projection device to: An image projection device (110) that calculates the distance to the surface on which the image is projected by performing system correction.

8. In any one of paragraphs 1 to 7, The above instructions, when executed by the at least one processor, cause the image projection device to: Based on the pixel localization results of the pattern image, state information of the surface and depth information of the surface are obtained, An image projection device (110) that calculates the distance to the surface based on the above state information and the above depth information.

9. In any one of paragraphs 1 to 8, The above instructions, when executed by the at least one processor, cause the image projection device to: An image projection device (110) that calculates the user's viewpoint information based on the image projection device (110) on a coordinate axis.

10. In any one of paragraphs 1 to 9, The above instructions, when executed by the at least one processor, cause the image projection device to: Generate a virtual image based on an image projected toward a minimum distance surface, An image projection device (110) that creates a virtual volume based on an image projected toward a surface at a maximum distance.

11. In a method for controlling an image projection device, An operation of the image projection device to obtain characteristic information of a surface on which an image is projected; An operation of the above image projection device to obtain viewpoint information related to the time at which the user views the image; An operation in which the image projection device generates a virtual image based on the characteristic information and the viewpoint information; and A method comprising: the image projection device projecting the image onto the surface based on the virtual image and the location of the user.

12. In paragraph 11, The operation of the above image projection device projecting a pattern image onto the surface; and A method further comprising the operation of correcting the image projection device based on the pattern image.

13. In paragraph 12, An operation in which the image projection device acquires depth information of the surface based on the pattern image and the result of the correction; and A method further comprising an operation of the image projection device obtaining color information and curvature information of the surface based on the pattern image and the depth information.

14. In any one of paragraphs 11 to 13, A method, wherein the image projection device further comprises an operation of localizing a location of the user.

15. In paragraph 14, A method further comprising the operation of the image projection device generating a virtual field of view corresponding to an area that cannot be captured based on the depth information and the position of the user.

Images