Electronic device and method of controlling same

Abstract

This electronic device comprises an image projection unit, a camera, a memory storing instructions, and one or more processors including a processing circuit. The one or more processors are configured to, when the instructions are executed individually or collectively, instruct the image projection unit to output an image including a preset pattern to a projection surface, acquire a captured image of the projection surface through the camera, identify characteristic information including continuity information of lines included in the preset pattern in the captured image, identify at least a portion of the projection surface as an output area on the basis of the characteristic information, and instruct the image projection unit to project an input image to the identified output area.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for projecting an image and a method for controlling the same.

[0002] Recent advancements in electronic and optical technology have led to the widespread use of various projectors. A projector is an electronic device that projects light onto a projection surface (or screen) to form an image on the surface. Various projector-related technologies, such as image resizing, keystone correction, and screen adaptation, are also evolving.

[0003] An electronic device according to one or more embodiments comprises: an image projector; a camera; a memory storing instructions; and one or more processors including processing circuitry; wherein the one or more processors are configured to control the image projector to output an image including a preset pattern to a projection surface when the instructions are individually or collectively executed, acquire a captured image of the projection surface through the camera, identify characteristic information including continuity information of a line included in the preset pattern in the captured image, identify at least a portion of the projection surface as an output area based on the characteristic information, and control the image projector to project an input image to the identified output area.

[0004] According to one or more embodiments, the one or more processors may be configured to control the image projector to output the image including a plurality of grid areas defined by grid lines to the projection surface when the instructions are individually or collectively executed, identify continuity information of the grid lines in the captured image, and identify at least a portion of the projection surface as the output area based on the continuity information of the grid lines.

[0005] According to one or more embodiments, the one or more processors may be configured to identify characteristic information including at least one of texture complexity information including fine texture information of each grid area in the captured image, homogeneity information including brightness change information between adjacent grid areas adjacent to each grid area, or chroma information including color change information between adjacent grid areas adjacent to each grid area, when the instructions are individually or collectively executed, and identify the output area on the projection surface based on the line continuity information and at least one of the texture complexity information, the homogeneity information, or the chroma information.

[0006] According to one or more embodiments, the one or more processors may be configured to, when the instructions are individually or collectively executed, convert the captured image into a YUV image, identify at least one of the line continuity information, the texture complexity information, and the flatness information based on a Y signal included in the YUV image, and identify the chroma signal based on a UV signal included in the YUV image.

[0007] According to one or more embodiments, the one or more processors may be configured to, when the instructions are individually or collectively executed, separate the Y signal included in the YUV image into a reflectance component and an illumination component, apply a straight line estimation algorithm to the reflectance component to identify line continuity information corresponding to each grid area, identify texture complexity information corresponding to each grid area based on a difference between first contour information identified in the Y signal and second contour information identified in the reflectance component, and identify flatness information corresponding to each grid area based on whether a standard deviation of flatness values ​​identified in the reflectance component is greater than or equal to a threshold value.

[0008] According to one or more embodiments, the one or more processors may be configured to, when the instructions are individually or collectively executed, separate the UV signal included in the YUV image into a reflection component and an illumination component, and identify chroma information corresponding to each grid area based on a distance between the U component and the V component identified in the reflection component.

[0009] According to one or more embodiments, the one or more processors may be configured to identify the output area on the projection surface by applying different weights to each characteristic piece of information according to the importance of each characteristic piece of information obtained for each grid area when the instructions are individually or collectively executed.

[0010] According to one or more embodiments, the one or more processors may be configured to identify the output area in the projection surface by applying different weights to characteristic information based on at least one of a type or category of the input image when the instructions are individually or collectively executed.

[0011] According to one or more embodiments, the one or more processors may be configured to identify the output area on the projection surface by applying a dot product operation or a machine learning clustering algorithm to characteristic information obtained for each grid area when the instructions are individually or collectively executed.

[0012] According to one or more embodiments, the one or more processors may be configured to use at least one learned artificial intelligence model to identify at least one of the characteristic information or identify the output area when the instructions are individually or collectively executed.

[0013] According to one or more embodiments, the one or more processors may be configured to control the image projector to adjust the input image to correspond to a size of the identified output area and project the image onto the identified output area.

[0014] A method for controlling an electronic device according to one or more embodiments includes: a step of outputting an image including a preset pattern to a projection surface; a step of obtaining a photographed image of the projection surface; a step of identifying characteristic information including continuity information of a line included in the preset pattern in the photographed image; a step of identifying at least a portion of the projection surface as an output area based on the characteristic information; and a step of projecting an input image to the identified output area.

[0015] A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device according to one or more embodiments, cause the electronic device to perform an operation, the operation includes: outputting an image including a preset pattern to a projection surface; acquiring a photographed image of the projection surface; identifying characteristic information including continuity information of a line included in the preset pattern in the photographed image; identifying at least a portion of the projection surface as an output area based on the characteristic information; and projecting an input image onto the identified output area.

[0016] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0017] FIG. 1 is a diagram illustrating the operation of an electronic device according to one or more embodiments.

[0018] FIG. 2A is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0019] FIG. 2b is a drawing for explaining a detailed configuration of an electronic device according to one or more embodiments.

[0020] FIG. 3 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0021] FIG. 4 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments.

[0022] FIGS. 5A to 5C are drawings for explaining a method for providing a pattern image for surface searching according to one or more embodiments.

[0023] FIGS. 6A to 6D, FIGS. 7A and 7B are drawings for explaining the principle of a method for separating characteristics of a projection surface according to one or more embodiments.

[0024] FIG. 8 is a drawing for explaining a method for separating characteristics of a projection surface according to one or more embodiments.

[0025] FIG. 9 is a drawing for explaining a method for obtaining characteristic information of a projection surface according to one or more embodiments.

[0026] FIGS. 10A to 10C are drawings for explaining a method for obtaining characteristic information of a projection surface according to one or more embodiments.

[0027] FIGS. 11 and 12 are drawings for explaining line continuity information according to one or more embodiments.

[0028] FIGS. 13A to 13C are drawings for explaining chroma information according to one or more embodiments.

[0029] FIG. 14 is a drawing for explaining a method for identifying an output area according to one or more embodiments.

[0030] FIG. 15 is a diagram illustrating a method for determining a final output area according to one or more embodiments.

[0031] FIG. 16 is a diagram illustrating a method for determining a final output area according to one or more embodiments.

[0032] FIG. 17 is a diagram illustrating a method for determining a final output area according to one or more embodiments.

[0033] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0034] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions or cases of those skilled in the art, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0035] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0036] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0037] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0038] When it is said that a component (e.g., a first component) is “operatively or communicatively coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0039] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0040] In some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0041] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0042] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of "modules" or "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "module" or "part" that needs to be implemented as specific hardware.

[0043] As used herein, the terms "one embodiment," "an embodiment," "an exemplary embodiment," or similar expressions may indicate that a particular feature, structure, or characteristic may be included in at least one embodiment. Therefore, the appearances of the terms "in one embodiment," "in an embodiment," "in an exemplary embodiment," and the like throughout this specification may not necessarily refer to the same embodiment. The embodiments described herein are exemplary and are not limited thereto and may be implemented in various forms.

[0044] The specific order or hierarchy of the processes / flowcharts disclosed herein may be presented as an exemplary approach. The specific order or hierarchy of the processes / flowcharts may be modified according to design preferences. Furthermore, some blocks may be combined or omitted. The order of elements presented in the appended claims is exemplary and may not be limited to a specific order or hierarchy.

[0045] Embodiments described herein are illustrated through blocks shown in the drawings, and these blocks may be depicted as performing specific functions. These blocks may be referred to by names such as units, modules, etc., and may also be expressed in terms such as devices, logic, circuits, controllers, counters, comparators, generators, converters, etc. These blocks may be physically implemented as analog and / or digital circuits, and may include logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, etc.

[0046] The terms "one or more" and "one" may be used interchangeably herein, and "a" or similar expressions may be used when referring to only one item. For example, the expression "processor" may refer to a single processor or multiple processors. Furthermore, when a processor is described as performing a particular operation and is then referred to as performing additional operations, those operations may be performed by a single processor or by a combination of one or more processors.

[0047] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0048] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0049] FIG. 1 is a diagram illustrating the operation of an electronic device according to one or more embodiments.

[0050] According to one embodiment, the electronic device (100) can be implemented as a projector that projects an image onto a wall, a screen, or a screen, or various types of devices having an image projection function.

[0051] According to one embodiment, the electronic device (100) may provide an automatic screen setting technology that automatically sets the image projection screen. For example, the projection surface may have an irregular structure or a uniquely shaped surface. In this case, simply recognizing a predefined pattern (e.g., a wall corner) to determine the projection screen may not properly search for the projection screen. Furthermore, accurate search may not be performed in environments with dark or bright ambient lighting.

[0052] For example, the electronic device (100) can provide a projection screen at the most ideal location based on the characteristics of the projection surface. For example, the electronic device (100) can search for a flat area among the projectible area (10) as illustrated in FIG. 1 and provide a screen (20) of an ideal size at an ideal location. Hereinafter, this projection area search method is also referred to as surface searching.

[0053] Below, various embodiments are described that search for and provide the most suitable projection area by considering the surface characteristics of the projection surface (e.g., wall) and the influence of lighting.

[0054] FIG. 2A is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0055] According to FIG. 2A, the electronic device (100) may include an image projection unit (110), a memory (130), and one or more processors (140). The electronic device (100) may be implemented as a projector that projects an image onto a wall, a screen, or a screen, or various types of devices having an image projection function.

[0056] The image projection unit (110) can perform the function of projecting light to the outside to express an image and outputting the image onto a projection surface. Here, the projection surface may be a part of the physical space where the image is output. The image projection unit (110) may include various detailed components such as at least one light source among lamps, LEDs, and lasers, a projection lens, and a reflector.

[0057] The image projection unit (110) can project an image using one of various projection methods (e.g., CRT (cathode-ray tube) method, LCD (Liquid Crystal Display) method, DLP (Digital Light Processing) method, laser method, etc.). The image projection unit (110) can include at least one light source.

[0058] The image projection unit (110) can output images with a 4:3 screen ratio, a 5:4 screen ratio, or a 16:9 wide screen ratio depending on the purpose of the electronic device (100) or the user's settings, and can output images with various resolutions such as WVGA (854*480), SVGA (800*600), XGA (1024*768), WXGA (1280*720), WXGA (1280*800), SXGA (1280*1024), UXGA (1600*1200), and Full HD (1920*1080) depending on the screen ratio.

[0059] In addition, the image projector (110) can perform various functions for adjusting the projection image under the control of the processor (140). For example, the image projector (110) can perform a zoom function, a lens shift function, etc.

[0060] The camera (120) can be turned on and perform shooting according to a preset event. The camera (120) can convert the captured image into an electrical signal and generate image data based on the converted signal. For example, the subject can be converted into an electrical image signal through a semiconductor optical element (CCD; Charge Coupled Device), and the image signal thus converted can be amplified and converted into a digital signal and then signal processed.

[0061] The memory (130) can store data required for various embodiments. Depending on the purpose of data storage, the memory (130) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100). For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100). Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). In addition, in the case of memory that can be attached or detached to the electronic device (100'), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented in the form of.

[0062] For example, the memory (130) may store various information related to keystone correction and various information related to brightness correction. For example, the memory (130) may store various information obtained during the keystone correction process, such as a transformation matrix. For example, the memory (130) may store various information obtained during the brightness correction process, such as a brightness correction coefficient.

[0063] The one or more processors (120) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits comprising one or more processors, one or more of which may be configured to individually and / or collectively perform the various functions described herein in a distributed manner. As used herein, where "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms may encompass, for example, without limitation, a single processor performing the functions. Some of the recited functions may be performed by different processor(s) and other functions may be performed by different processor(s), and further, a single processor may perform all of the recited functions. Additionally, the one or more processors may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. The one or more processors (120) generally control the operation of the electronic device (100) by executing program instructions to achieve or perform various functions. Specifically, one or more processors (140) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (140) may be operatively connected to the display (130) and the memory (130). One or more processors (140) may be configured as one or more processors.

[0064] One or more processors (140) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (130).

[0065] The one or more processors (140) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (140) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (140) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing instructions stored in the memory.

[0066] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0067] One or more processors (140) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (140) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0068] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of description, one or more processors (140) will be referred to as a processor (140).

[0069] According to one embodiment, the processor (140) may control the image projector (110) to output an image of a preset pattern to a projection surface. According to one example, the image of the preset pattern may be an image including a plurality of grid areas identified by grid lines. For example, the plurality of grid areas may be square areas of a preset size, but are not limited thereto and may also be rectangular areas. For example, the preset size may be preset during the manufacturing of the electronic device (100) or may be set / changed according to a user command. For example, the preset size may be automatically determined according to the size, resolution, etc. of the image.

[0070] For convenience of explanation, the following description assumes that the image of the preset pattern is an image including multiple grid areas (hereinafter, grid image).

[0071] According to one embodiment, the processor (140) may acquire a captured image of the projection surface through the camera (120) and analyze the captured image to acquire characteristic information (or feature information or unique information) of the projection surface. According to one example, the characteristic information of the projection surface may include at least one characteristic information of different types. For example, the characteristic information of the projection surface may include at least one of line continuity information, texture information, homogeneity information, or chroma (or color uniformity) information.

[0072] According to one embodiment, the processor (140) may obtain characteristic information for each of a plurality of grid areas included in the captured image. According to one example, the processor (140) may obtain at least one of line continuity information, texture complexity information, flatness information, or chroma information for each of a plurality of grid areas included in the grid image. According to one example, the captured image may include a grid image output to the projection surface since it is an image captured on a projection surface on which a grid image is output.

[0073] According to one embodiment, the processor (140) can convert the captured image into a YUV image. For example, the captured image may be an RGB (Red, Green, Blue) domain image. Accordingly, the processor (140) can convert the RGB domain image or the YUV (Luminance, Chrominance) domain image. RGB is expressed as three values ​​representing the intensity of red, green, and blue components in each channel, and in the case of an 8-bit image, the channel has a value between 0 and 255, for example, (255, 0, 0) can represent red, (0, 255, 0) can represent green, and (0, 0, 255) can represent blue. YUV is a color space composed of three elements of luminance (Y) and chrominance (U, V), and Y is a brightness component and thus has luminance information, and U and V can have chrominance information for color. The U and V values ​​can represent, for each pixel, the difference between the color of that pixel and the colors of the surrounding pixels.

[0074] According to one embodiment, the processor (140) can convert an RGB image into a YUV image using at least one of a preset formula, rule, or algorithm.

[0075] The processor (140) can obtain characteristic information of the projection surface based on the YUV image.

[0076] For example, the processor (140) can identify continuity information, texture complexity, and flatness information of grid lines based on a Y signal (or Y domain signal) included in a YUV image.

[0077] In one example, the processor (140) can identify a chroma signal based on a UV signal included in a YUV image.

[0078] According to one embodiment, the processor (140) may process a YUV image to separate it into a reflectance component (or reflectance signal) and an illumination component (or illumination signal). The reason for separating the YUV image into a reflectance component and an illumination component is that the influence of lighting conditions must be removed in order to understand the actual color or material of the projection surface.

[0079] The specular component can describe the degree to which an object, such as a projection surface, reflects light of a specific wavelength. The specular component can vary depending on the color and texture of the projection surface.

[0080] The illumination component can represent the properties of light reflected on a projection surface. The illumination component can be determined by the type, intensity, angle, and spectral distribution of the light source.

[0081] For example, the processor (140) may perform image decomposition processing on a YUV image to separate it into a reflectance component and an illumination component. For example, image decomposition is based on the assumption that the image I can be expressed as the product of Reflectance R and Illumination L, and may be expressed as I(x,y)=R(x,y)*L(x,y). Here, I(x,y) is the brightness value of a pixel in the image, R(x,y) is Reflectance, L(x,y) is Illumination, and (x,y) is the image coordinate. However, the present invention is not limited thereto, and the YUV image can be separated into a reflectance component and an illumination component through Retinex Theory, Bayesian Framework, Deep Learning, Practical Algorithm, Single-Scale Retinex (SSR), filtering (e.g., Gaussian Filtering), etc.

[0082] According to one embodiment, the processor (140) may obtain characteristic information of a projection surface based on a reflection component separated from a YUV image. According to one example, the characteristic information of the projection surface may include at least one of line continuity information, texture information, flatness information, or chroma information.

[0083] Line continuity information may refer to information about the continuity of grid lines included in the captured image. Grid line continuity may refer to information about how closely the grid lines are to straight lines. For example, the grid lines may include at least one of a horizontal grid line and a vertical grid line.

[0084] For example, line continuity information can be obtained using a straight line estimation algorithm. For example, the straight line estimation algorithm may include at least one of Random Sample Consensus (RANSAC), Least Squares Method (LSM), Hough Transform, Least Absolute Deviations (LAD), Total Least Squares (TLS), or Orthogonal Distance Regression (ODR).

[0085] For example, the processor (140) may identify (or detect) each grid line in the captured image and obtain line continuity information, which may indicate how much of each grid line is an outlier or how much continuity of the line is maintained, from the identified straight lines through a straight line estimation algorithm. For example, the line continuity information may include a line continuity value corresponding to the coordinates of each grid area. The coordinates of each grid area may include coordinates indicating the location of each grid area, for example, (x, y) coordinates. The values ​​corresponding to the coordinates of each grid area may include a numerical value indicating how much of the grid lines corresponding to the four corners of each grid area are outliers.

[0086] Texture information may be visual pattern information indicating the expression characteristics of a projection surface included in a captured image. As an example, the texture information may include texture complexity information. Texture complexity indicates the structural and statistical characteristics of a texture, and may indicate how complex and diverse the texture pattern is. As an example, at least one of statistical methods, geometric methods, or transform-based methods may be used to obtain texture complexity.

[0087] According to one embodiment, the processor (140) may obtain texture information of each grid area based on the Y signal of the captured image. For example, the texture information may include texture values ​​corresponding to the coordinates of each grid area. According to one example, the processor (140) may obtain fine texture values ​​of each grid area and assign them to the coordinate values ​​of each grid area.

[0088] In this case, since the image signal acquired through the captured image is sensitive to the surrounding lighting environment, there is a possibility that the pattern or texture of the projection surface (e.g., a wall) may not be acquired from the captured image. Accordingly, the processor (140) can acquire a texture complexity value based on the difference value of the outline information and the reflection component detected in the Y signal. For example, the processor (130) can acquire the reflection component by separating the Y signal into the Reflectance and the Lighting component through Image Decomposition.

[0089] According to one example, the processor (140) can obtain the fine texture value of each grid area using an analysis function (or an analysis algorithm or an analysis rule). For example, the processor (140) can obtain the fine texture value of each grid area using Co-occurrence Matrices. Co-occurrence Matrices can obtain numerical features of textures using spatial relationships of similar gray tones, and can express, compare, and classify textures using the obtained numerical features. The following mathematical expressions 1 to 4 may be a subset of standard features that can be derived from normalized Co-occurrence Matrices.

[0090]

[0091]

[0092]

[0093]

[0094] Here, p[i. j] is the [i,j]th entry in the gray-tone spatial dependence matrix, and Ng can be the number of unique gray levels of the quantized image.

[0095] According to one embodiment, flatness information of each grid area can be acquired based on the Y signal of the captured image. For example, the flatness information can include a flatness value corresponding to the coordinates of each grid area. According to one example, the processor (140) can acquire the flatness value (e.g., contour detection) of each grid area and assign it to the coordinate values ​​of each grid area.

[0096] Flatness information may include information about the similarity of pixel brightness values ​​included in the grid output image. For example, the flatness information may include brightness variation information between adjacent grid areas for each grid area in the grid output image. In one example, at least one of standard deviation, gradient, or variance may be used to obtain the flatness information.

[0097] In this case, since the image signal acquired through the captured image is sensitive to the surrounding lighting environment, there is a possibility that the pattern or structure of the projection surface (e.g., wall) may not be acquired from the captured image. Accordingly, the processor (140) can acquire a flatness value based on the reflection component acquired from the Y signal. For example, the processor (130) can acquire the reflection component by separating the Y signal into the reflection component and the lighting component through image decomposition. For example, Reflectance can detect even the fine outline due to the texture or flatness characteristics of the wall by maximizing the local contrast.

[0098] For example, the processor (140) obtains the standard deviation (σ) of the flatness values ​​based on the Y signal values ​​of each grid area in the captured image. H ) has at least one threshold value (σ Thr2 ) can be used to calculate flatness information for each grid area based on whether the standard deviation is greater than or equal to 1. For example, the flatness characteristic increases as the standard deviation decreases. However, for the convenience of operation, the flatness characteristic of each grid area can be distinguished based on a threshold value. For example, the processor (140) can distinguish the flatness characteristic into two levels based on one threshold value, or can distinguish the flatness characteristic into three levels based on two threshold values.

[0099] Chroma information may include information about the similarity of pixel color values ​​included in the grid output image. For example, the chroma information may include color variation information between adjacent grid areas for each grid area in the grid output image. In one example, at least one of standard deviation, gradient, or variance may be used to obtain chroma information.

[0100] According to one embodiment, the processor (140) may obtain chroma information for each grid area based on the UV signal of the captured image. For example, the chroma information may include chroma values ​​corresponding to the coordinates of each grid area. According to one example, the processor (140) may obtain the chroma values ​​for each grid area and assign them to the coordinate values ​​of each grid area.

[0101] For example, the processor (140) may calculate a chroma value based on the distance between the U component and the V component (e.g., |U| + |V|) in the UV signal of the captured image. The chroma value calculated in this way may represent a color difference between grid areas. For example, the processor (140) may calculate the difference between the U and V components using the Euclidean distance, but is not limited thereto. For example, the Euclidean distance between two points (u1, v1) and (u2, v2) is It can be produced as follows.

[0102] For example, the processor (140) obtains the standard deviation (σ) of the chroma values ​​based on the UV signal values ​​of each grid area in the captured image. C ) is the threshold value (σ Thr3) can be used to calculate chroma information for each grid area based on whether the standard deviation is greater than or equal to the uniformity characteristic. For example, the smaller the standard deviation, the higher the uniformity characteristic. However, for the convenience of operation, the uniformity characteristic of each grid area can be distinguished based on a threshold value. For example, the processor (140) can distinguish the uniformity characteristic into two levels based on one threshold value, or can distinguish the uniformity characteristic into three levels based on two threshold values.

[0103] According to one embodiment, the processor (140) can identify an output area in a projection surface based on characteristic information of the projection surface.

[0104] In one example, the processor (140) can identify an output area in a projection plane based on at least one of line continuity information, texture complexity information, flatness information, or chroma information.

[0105] For example, the processor (140) may perform processing to reduce the amount of data of the characteristic information acquired for each grid area. For example, the characteristic information acquired for each grid area may be expressed as continuous signal values. The processor (140) may perform quantization or binarization processing to approximate the continuous values ​​of the signal to discrete values. Accordingly, the required amount of data may be reduced while minimizing information loss. For example, in the case of quantization, each characteristic information may be classified into multiple ranges based on multiple threshold values, and in the case of binarization, each characteristic information may be classified into two ranges based on a single threshold value. For example, the processor (140) may perform 4-bit quantization to express the characteristic values ​​as 16 values.

[0106] For example, the characteristic information may include at least one of line continuity information, texture complexity information, flatness information, or chroma information obtained for each grid area. However, for convenience of explanation, the following description assumes that line continuity information, texture complexity information, flatness information, and chroma information are all obtained for each grid area.

[0107] According to one embodiment, the processor (140) may identify an output area on the projection surface based on a value (hereinafter, a code value) calculated through quantization or binarization for each characteristic information. For example, the processor (140) may identify an output area on the projection surface based on a code value calculated through quantization or binarization for each grid area.

[0108] According to one embodiment, the processor (140) can identify an output area on the projection surface through an operation on the characteristic information calculated for each grid area. According to one example, the processor (140) can identify an output area on the projection surface through an operation on a code value for each characteristic information calculated for each grid area. For example, the processor (140) can identify an output area on the projection surface based on an operation value for each grid area calculated through a Dot Product for the code value for each characteristic information calculated for each grid area. For example, the Dot Product (inner product, scalar product) can be an operation that multiplies components at the same position of each vector and then adds all the results. For example, if the code values ​​of the line continuity information, the texture complexity information, the flatness information, and the chroma information calculated for the first grid area are a1, b1, c1, and d1, respectively, the processor (140) can identify a1*b1*c1*d1 as the first operation value of the first grid area, and if the code values ​​of the line continuity information, the texture complexity information, the flatness information, and the chroma information calculated for the second grid area are a2, b2, c2, and d2, respectively, the processor (140) can identify a2*b2*c2*d2 as the second operation value of the second grid area. For example, if the binarized code values ​​of the line continuity information, the texture complexity information, the flatness information, and the chroma information calculated for the first grid area are 1, 0, 1, and 0, respectively, the grid area in which the Dot Product result is 1 can be identified as a valid grid area that can be utilized as an output area.

[0109] According to one embodiment, the processor (140) may identify an output area on the projection surface by applying a preset algorithm to the characteristic information produced for each grid area. According to one example, the preset algorithm may include a machine learning clustering algorithm. The machine learning clustering algorithm may be an unsupervised learning technique that divides data into groups with similar characteristics. For example, the machine learning clustering algorithm may include a decision tree, a random forest, a K-means clustering, a hierarchical clustering, etc. For example, the processor (140) may classify a wall surface into multiple areas according to characteristics using the simplest machine learning algorithm, such as a decision tree. For example, the processor (140) may classify a wall surface into multiple areas using a random forest in a more robust manner. For example, the processor (140) can classify a wall surface into multiple regions using an ensemble method that combines multiple models. For example, the processor (140) can classify a wall surface into multiple regions using an ensemble method such as bagging or boosting.

[0110] According to one embodiment, the processor (140) may use distance information between each grid area in the clustering algorithm to identify the largest possible output area on the projection surface. According to one example, the processor (140) may classify grid areas with close distances as end terminals in the clustering algorithm. For example, the processor (140) may perform the clustering algorithm by setting the distance between all grid areas adjacent to a specific grid area as a first variable (x1), and the characteristic information values ​​corresponding to the specific grid area, for example, values ​​corresponding to line continuity information, texture complexity information, flatness information, and chroma information, as second to fifth variables (x2 to x5), respectively. For example, the processor (140) may calculate the distance between all grid areas adjacent to a specific grid area as the first variable based on the Manhattan Distance (or Taxi Distance or L1 Norm) as shown in Mathematical Expression 5 below. The Manhattan Distance is one of the methods for measuring the distance between two points and may calculate the sum of the distances moving along each axis.

[0111]

[0112] In one embodiment, the processor (140) can identify the output area at a specific ratio. This is because, when the image is played at a specific ratio, it can provide a viewing experience most similar to the original image. In one example, the processor (140) can perform a fast and accurate grid search operation under the constraint that the number of grid areas in terms of width and height remain constant.

[0113] According to one embodiment, the processor (140) may assign different priorities to the acquired characteristic information for each grid area based on its importance. For example, the processor (140) may apply different weights based on the priority of each characteristic piece of information. For example, the sum of the weights may be 1, but this is not necessarily limited to this.

[0114] For example, different priorities may be assigned in the order of line continuity information > flatness information > texture complexity information > chroma information, and relatively higher weights may be applied to feature information with higher priorities. For example, the weights may be managed as hyperparameters. Hyperparameters may be variables directly set by the user. However, hyperparameters may also be adjusted (or tuned) to optimize performance. For example, the processor (140) may try all combinations of predefined hyperparameter values ​​to find the optimal value. However, this is only an example, and the priorities of each feature information may be determined differently depending on the specifications of the electronic device (100), the projection environment, the user's projection environment preferences, etc.

[0115] According to one embodiment, when the processor (140) classifies a wall surface into multiple regions using a decision tree, the node with the largest number of grids included in the node among the nodes of the decision tree can be selected and identified as the output region.

[0116] According to one embodiment, when an output area is identified, the processor (140) adjusts the input image to correspond to the size of the output area.

[0117] FIG. 2b is a drawing for explaining a detailed configuration of an electronic device according to one or more embodiments.

[0118] According to FIG. 2b, the electronic device (100') includes an image projector (110), a camera (120), a memory (130), one or more processors (140), a sensor (150), a user interface (160), and a communication interface (170).

[0119] The image projector (110) can enlarge or reduce the image depending on the distance from the projection surface (projection distance). In other words, a zoom function can be performed depending on the distance from the projection surface. At this time, the zoom function may include a hardware method that adjusts the screen size by moving the lens and a software method that adjusts the screen size by cropping the image, etc. Meanwhile, when the zoom function is performed, the focus of the image needs to be adjusted. For example, the method of adjusting the focus may include a manual focus method, an electric method, etc.

[0120] In addition, the image projection unit (110) can automatically analyze the surrounding environment and projection environment without user input to provide zoom / keystone / focus functions. Specifically, the projection unit (111) can automatically provide zoom / keystone / focus functions based on the distance between the electronic device (100) and the projection surface detected through a sensor (Time of flight (ToF) sensor, depth camera, distance sensor, infrared sensor, light sensor, etc.), information about the space where the electronic device (100) is currently located, information about the amount of ambient light, etc.

[0121] The sensor (150) may include sensors such as a distance sensor, an acceleration sensor (or gravity sensor), a geomagnetic sensor, a gyro sensor, etc. The distance sensor is a component for detecting the distance from the projection surface. For example, the distance sensor may be implemented in various types such as an ultrasonic sensor, an infrared sensor, a LIDAR sensor, a RADAR sensor, a photodiode sensor, etc. For example, the distance sensor, an acceleration sensor (or gravity sensor), a geomagnetic sensor, and a gyro sensor may be used to obtain pitch information and / or yaw information.

[0122] In addition, the sensor (150) may include various types of sensors such as an image sensor, a touch sensor, a proximity sensor, a pressure sensor, a position sensor, etc.

[0123] The user interface (160) may be implemented by devices such as buttons, touch pads, mice, and keyboards, or may be implemented by a touch screen, remote control transmitter / receiver, etc. that can also perform the display function and operation input function described above. The remote control transmitter / receiver may receive a remote control signal from an external remote control device or transmit a remote control signal through at least one of infrared communication, Bluetooth communication, and Wi-Fi communication.

[0124] For example, the user interface (160) may receive user input for adjusting the magnification of a projected image, user input for selecting and / or changing at least one of the size or type of a sub-image, user input for selecting a menu on a UI screen, etc.

[0125] It goes without saying that at least one communication interface (170) (hereinafter, communication interface) can be implemented as various interfaces depending on the implementation example of the electronic device (100'). For example, the communication interface (170) can communicate with an external device (e.g., a user terminal), an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through communication methods such as various types of digital interfaces, AP-based Wi-Fi (Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0126] Meanwhile, according to an implementation example of the electronic device (100'), a speaker, a tuner, and a demodulator may be additionally included. The tuner (not shown) can receive an RF (Radio Frequency) broadcast signal by tuning a channel selected by a user or all pre-stored channels among RF broadcast signals received through an antenna. The demodulator (not shown) can receive and demodulate a digital IF signal (DIF) converted by the tuner, and can also perform channel decoding, etc. According to one embodiment, an input image received through the tuner can be processed through the demodulator (not shown) and then provided to the processor (140).

[0127] FIG. 3 is a flowchart illustrating a control method of an electronic device according to one or more embodiments. The operating method illustrated in FIG. 3 may be performed by either the electronic device (100) illustrated in FIG. 2a and / or the electronic device (100') illustrated in FIG. 2b. For convenience of explanation, the present disclosure will be described assuming that the method is performed by the electronic device (100).

[0128] According to FIG. 3, in operation 310, the electronic device (100) can output an image of a preset pattern to a projection surface. For example, the electronic device (100) can output an image including a plurality of grid areas identified by grid lines to the projection surface.

[0129] In operation 320, the electronic device (100) can obtain a photographed image of the projection surface.

[0130] In operation 330, the electronic device (100) can identify feature information including continuity information of lines included in a preset pattern in the captured image. For example, the electronic device (100) can identify continuity information of grid lines in the captured image.

[0131] In operation 340, the electronic device (100) may identify at least a portion of the projection surface where line continuity is maintained as an output area based on feature information. For example, the electronic device (100) may identify at least a portion of the projection surface where line continuity is maintained as an output area based on grid line continuity information.

[0132] In operation 350, the electronic device (100) may project an input image onto the identified output area. In one example, the electronic device (100) may adjust and project the input image to correspond to the size of the identified output area.

[0133] According to one embodiment, the electronic device (100) can identify characteristic information including at least one of texture complexity information including fine texture information of each grid area in the captured image, flatness information including brightness change information between adjacent grid areas adjacent to each grid area, or chroma information including color change information between adjacent grid areas adjacent to each grid area. The electronic device (100) can identify an output area on the projection surface based on line continuity information and at least one of texture complexity, flatness information, or chroma information.

[0134] According to one embodiment, the electronic device (100) can convert a captured image into a YUV image and identify at least one of line continuity information, texture complexity information, and flatness information based on a Y signal included in the YUV image. In addition, the electronic device (100) can identify a chroma signal based on a UV signal included in the YUV image.

[0135] According to one embodiment, the electronic device (100) may separate the Y signal included in the YUV image into a reflection component and an illumination component. According to one example, the electronic device (100) may identify line continuity information corresponding to each grid area by applying a straight line estimation algorithm to the reflection component. According to one example, the electronic device (100) may identify texture complexity information corresponding to each grid area based on the difference between the first contour information identified in the Y signal and the second contour information identified in the reflection component. According to one example, the electronic device (100) may identify flatness information corresponding to each grid area based on whether the standard deviation of the flatness value identified in the reflection component is greater than or equal to a threshold value.

[0136] According to one embodiment, the electronic device (100) can separate a UV signal included in a YUV image into a reflection component and an illumination component, and identify chroma information corresponding to each grid area based on the distance between the U component and the V component identified in the reflection component.

[0137] According to one embodiment, the electronic device (100) can identify an output area on a projection surface by applying different weights to each characteristic information according to the importance of each characteristic information obtained for each grid area.

[0138] According to one embodiment, the electronic device (100) may identify an output area on a projection surface by applying different weights to characteristic information based on at least one of the type or category of the input image. The type of the input image may include at least one of type information such as real-time broadcasting, OTT (Over The Top) content, VOD (Video On Demand) content, etc. The category of the input image may include at least one of category information such as movies, sports, documentaries, etc.

[0139] According to one embodiment, the electronic device (100) can utilize at least one learned artificial intelligence model for at least one of characteristic information identification or output area identification.

[0140] According to one example, the electronic device (100) can acquire characteristic information using a learned artificial intelligence model. For example, when a captured image is input, the artificial intelligence model can be trained to output characteristic information of a projection surface included in the captured image. For example, an artificial intelligence model trained individually for each type of characteristic information can be used. For example, an artificial intelligence model trained to output line continuity information, texture information, flatness information, and chroma information based on the input image can be used to acquire each characteristic information.

[0141] According to one example, the electronic device (100) can obtain information about the output area finally identified on the projection surface using the learned artificial intelligence model. For example, when a captured image is input, the artificial intelligence model can be trained to obtain characteristic information of the projection surface included in the captured image and output information about the output area based on the obtained characteristic information. For example, when characteristic information is input through the captured image, the artificial intelligence model can be trained to output information about the output area based on the input characteristic information. For example, the information about the output area can include coordinate information corresponding to the output area in the captured image. For example, the information about the output area can include a plurality of coordinate information and probability information corresponding to a plurality of output areas in the captured image. For example, the probability information can include priority information about the plurality of coordinate information.

[0142] For example, when an artificial intelligence model is trained, it means that a basic artificial intelligence model (e.g., an artificial intelligence model including any random parameters) is trained using a learning algorithm using a plurality of training data, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). Such learning may be performed through a separate server and / or system, but is not limited thereto, and may also be performed in the electronic device (100). Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0143] For example, the learned artificial intelligence model may be an on-device model included in the electronic device (100), but is not limited thereto. For example, the learned artificial intelligence model may be implemented on a server.

[0144] Meanwhile, in Fig. 3, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0145] FIG. 4 is a flowchart illustrating a control method of an electronic device according to one or more embodiments. The operating method illustrated in FIG. 4 may be performed by either the electronic device (100) illustrated in FIG. 2A and / or the electronic device (100') illustrated in FIG. 2B. For convenience of explanation, the present disclosure will be described assuming that the method is performed by the electronic device (100).

[0146] According to FIG. 4, when the electronic device (100) is turned on in operation 401, keystone correction can be performed in operation 402.

[0147] Keystone correction may be a function that corrects image distortion when an electronic device (100) is tilted. For example, keystone correction may be performed manually or automatically. Keystone correction may be performed according to various correction methods, such as quadrilateral correction and radial correction.

[0148] In operation 403, the electronic device (100) can output a frame image to a projection surface based on the keystone correction result.

[0149] A frame image may be an image representing an output area. The frame image may be an element for easily identifying the output area in an image captured by a camera (120). For example, the frame image may be an image in which the frame (or border) of the output area may be indicated with either a single color or a composite color. For example, the single color may be black or white, but is not limited thereto.

[0150] In operation 404, the electronic device (100) can output a pattern image including a preset pattern to a projection surface and capture the projection surface on which the pattern image is output. Subsequently, the electronic device (100) can convert the captured image, for example, an RGB image, into a YUV image and separate the YUV image into a Y signal and a UV signal.

[0151] In operation 405, the electronic device (100) can obtain a reflection component by separating the Y signal into reflectance and illumination components through image decomposition.

[0152] In operation 406, the electronic device (100) can detect an outline component from a Y signal separated from a YUV image.

[0153] In operation 407, the electronic device (100) can identify texture complexity information based on the contour component detected in the Y signal. For example, the electronic device (100) can obtain the texture complexity value based on the difference value of the contour information and the reflection component detected in the Y signal. For example, the electronic device (100) can obtain the reflection component by separating the Y signal into the reflectance and illumination component through image decomposition.

[0154] In operation 408, the electronic device (100) can quantize (or binarize) texture complexity information to obtain a quantized texture complexity value.

[0155] In operation 409, the electronic device (100) can perform surface searching on the projection surface based on the quantized texture complexity value.

[0156] In operation 410, the electronic device (100) can identify flatness information based on the Y signal. For example, the electronic device (100) can obtain a flatness value corresponding to each grid area based on a reflection component detected in the Y signal.

[0157] In operation 411, the electronic device (100) can quantize (or binarize) the flatness information to obtain a quantized flatness value.

[0158] In operation 412, the electronic device (100) can perform surface searching on the projection surface based on the quantized flatness value.

[0159] In operation 413, the electronic device (100) can identify continuity information of a line based on the Y signal. For example, the electronic device (100) can obtain a continuity value of a line corresponding to each grid area based on a reflection component detected in the Y signal.

[0160] In operation 414, the electronic device (100) can quantize (or binarize) the continuity information of the line to obtain the continuity value of the quantized line.

[0161] In operation 415, the electronic device (100) can perform surface searching on the projection plane based on the continuity value of the quantized line.

[0162] In operation 416, the electronic device (100) can obtain a reflection component by separating the UV signal into reflectance and illumination components through image decomposition.

[0163] In operation 417, the electronic device (100) can identify chroma information based on the UV signal. For example, the electronic device (100) can obtain a chroma value corresponding to each grid area based on a reflection component detected from the UV signal.

[0164] In operation 418, the electronic device (100) can quantize (or binarize) chroma information to obtain a quantized chroma value.

[0165] In operation 419, the electronic device (100) can perform surface searching on the projection surface based on the quantized chroma values.

[0166] In operation 420, the electronic device (100) can determine a final surface based on the surface searching result in operation 408, the surface searching result in operation 412, the surface searching result in operation 415, and the surface searching result in operation 419.

[0167] Meanwhile, in Fig. 4, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0168] FIGS. 5A to 5C are drawings for explaining a method for providing a pattern image for surface searching according to one or more embodiments.

[0169] According to one embodiment, the electronic device (100) may output a frame image (510) to a projection surface as illustrated in FIG. 5A. For example, the electronic device (100) may output the frame image to the projection surface based on a keystone-corrected output area. The frame image may be an element for easily identifying a keystone-corrected output area in a captured image.

[0170] According to one embodiment, the frame image may be a frame-shaped image in which the border (511) of the output area is indicated with either a single color (512, 514) or a composite color (513), as illustrated in FIG. 5B. For example, the single color may be black (512) or white (514), but is not limited thereto. For example, if the projector is used in a dark lighting environment, a black frame may be the easiest to recognize due to the color contrast with the surrounding colors when photographing with a camera (120). However, it may not be easy to recognize a black frame in a somewhat bright environment. A white frame may be the easiest to recognize if there is no interference from external light. A frame composed of composite colors compared to an achromatic frame may further improve the detection rate by responding to the surrounding lighting or chromatic environment.

[0171] According to one embodiment, the electronic device (100) can capture an image by using a camera (120) to capture a projection surface, for example, a wall, on which a pattern image is output. For example, the angle of view of the camera (120) may be wide enough to capture a pattern image.

[0172] According to one embodiment, the electronic device (100) can detect a rectangular frame area in a captured image. For example, the electronic device (100) can detect edges, corners, etc. included in the rectangular frame area.

[0173] For example, the electronic device (100) may apply thresholding to identify objects based on the color of each pixel in the captured image. For example, the electronic device (100) may detect separated object rectangles by utilizing contour detection and minimum Rect. Area search functions, calculate the area (e.g., width * height) of each detected rectangle, and select the rectangle with the largest area among all rectangles as the final result. For example, since the electronic device (100) may be affected by ambient lighting, color interference, etc., the electronic device (100) may utilize an improved frame shape detection algorithm through color histogram analysis. For example, if no rectangle is detected, the electronic device (100) may provide a notification to the user to position the image projection screen so that it is projected on a wall.

[0174] For example, even if a frame is not detected as a closed shape (e.g., a square), a pattern image (520) including a grid area with a specific interval can be output to the projector output image. For example, the pattern area can include a plurality of grid areas (520) within the frame (510), as illustrated in FIG. 5C. For example, an image including the frame (510) and an image including a plurality of grid areas (520) can be output simultaneously. For example, the electronic device (100) can output an image including the frame (510) and an image including a plurality of grid areas (520) by alpha blending them. Here, alpha blending refers to a method of mixing the background RGB value and the RGB value above it by assigning a new value called A (Alpha) to the color value RGB in order to create a transparent effect when overlapping another image on top of the image. For example, the Alpha value is divided into 0~255 values ​​or 0.0~1.0 values, and 0 can mean completely transparent, and the opposite, 255 (or the highest value such as 1.0) can mean fully opaque. Or 0 can mean completely opaque, and the opposite, 255 (or the highest value such as 1.0) can mean fully transparent. For example, when 8 bits are allocated to the Alpha value and can express values ​​from 0 to 255, the larger the value, the higher the ratio of the corresponding pixel, and the lower the value, the lower the ratio. For example, when mixing an image I1 including a frame (510) and an image I2 including a plurality of grid areas (520), the mixing operation can be expressed by a formula such as I1*Alpha + I2*(1-Alpah) or I1*(1-Alapha)+I2*alpha or I1*Alpah+I2.

[0175] FIGS. 6A to 6D, FIGS. 7A and 7B are drawings for explaining the principle of a method for separating characteristics of a projection surface according to one or more embodiments.

[0176] According to one embodiment, the image shown to the user as illustrated in FIG. 6a can be expressed as the product of Reflectance R and Illumination L.

[0177] According to one embodiment, the electronic device (100) can separate an illumination component (620) as shown in FIG. 6c and a reflection component (630) as shown in FIG. 6d from a captured image (610) as shown in FIG. 6b.

[0178] According to an example, the electronic device (100) can obtain a Y signal from a YUV image as illustrated in FIG. 7A, and perform image decomposition processing on the Y signal to obtain a reflection component from the Y signal. For example, the electronic device (100) can separate the Y signal included in the YUV image into an illumination component (720) and a reflection component (730).

[0179] For example, the electronic device (100) can obtain a reflection component from the Y signal through Retinex Decomposition processing.

[0180] For example, as illustrated in FIG. 7b, the electronic device (100) can separate the illumination component (L) from the Y signal through Retinex Decomposition processing, and obtain the reflection component (R) using the illumination component (L). Retinex is a compound word of "retina" and "cortex", and may be a method in which the retina and the cerebral cortex work together to process visual information. For example, the electronic device (100) can perform Retinex Decomposition processing through Logarithmic Transformation, Spatial Processing, and Combination of Multiple Scales. However, the present invention is not limited thereto, and the Y signal can be separated into the reflection component and the illumination component through Bayesian Framework, Deep Learning, Practical Algorithm, Single-Scale Retinex (SSR), filtering (e.g., Gaussian Filtering), etc.

[0181] FIG. 8 is a drawing for explaining a method for separating characteristics of a projection surface according to one or more embodiments.

[0182] According to one embodiment, as illustrated in FIG. 8, the electronic device (100) can capture an image by capturing a projection surface, for example, a wall surface (810).

[0183] According to one embodiment, the electronic device (100) can capture a projection surface and separate the captured image into an illumination component (820) and a reflection component (830). Thereafter, the electronic device (100) can obtain characteristic information of the projection surface based on the reflection component (830) representing the unique characteristics of the projection surface (810), and determine an output area for outputting an image on the projection surface (810) based on the obtained characteristic information.

[0184] FIG. 9 is a drawing for explaining a method for obtaining characteristic information of a projection surface according to one or more embodiments.

[0185] According to one embodiment, as illustrated in FIG. 9, the electronic device (100) can obtain a captured image (910) of a projection surface, for example, a wall surface.

[0186] According to one embodiment, the electronic device (100) can separate the captured image (910) into an illumination component (920) and a reflection component (930). For example, the electronic device (100) can convert the RGB captured image (910) into a YUV image and separate the YUV image into a Y signal and a UV signal. Thereafter, the electronic device (100) can separate the Y signal into an illumination component (920) and a reflection component (930).

[0187] After this, the electronic device (100) can obtain characteristic information of the projection surface based on the reflection component (930) representing the unique characteristics of the projection surface, and determine an output area for outputting an image on the projection surface based on the obtained characteristic information.

[0188] As shown in Fig. 9, the reflection component (942) obtained by separating the lighting component (941) corresponding to a specific area (911) in the captured image (910) can be confirmed to have texture edge information, etc., significantly emphasized or clearly improved.

[0189] FIGS. 10A to 10C are drawings for explaining a method for obtaining characteristic information of a projection surface according to one or more embodiments.

[0190] FIG. 10A is a diagram illustrating flatness information according to an embodiment. According to an example, the electronic device (100) can obtain a flatness map based on a Y signal in a YUV image. For example, the electronic device (100) can obtain a flatness map including flatness information for each grid area.

[0191] FIG. 10b is a diagram illustrating line continuity information according to an embodiment. According to an example, the electronic device (100) may obtain a line continuity map based on a Y signal in a YUV image. For example, the electronic device (100) may obtain a line continuity map including line continuity information for each grid area.

[0192] FIG. 10c is a diagram illustrating chroma information according to an embodiment. According to an example, the electronic device (100) may obtain a chroma map based on a UV signal from a YUV image. For example, the electronic device (100) may obtain a chroma map including chroma information for each grid area.

[0193] FIGS. 11 and 12 are drawings for explaining line continuity information according to one or more embodiments.

[0194] For example, line continuity information can be identified based on whether the linearity of grid lines in a captured image is maintained (e.g., whether a single line continues continuously). For example, as shown in FIG. 11, if there is a curvature as shown on the left in the projection plane, the linearity (the characteristic of appearing as a straight line) of lines (1-1, 2-1, 3-1...) (e.g., grid lines) in the captured image is broken as shown on the right. In this case, it can be identified that line continuity is not maintained in the corresponding projection plane area.

[0195] As an example, let's assume that the projection surface includes an object, for example, a vase, positioned in front of a wall, as illustrated in FIG. 12 . In this case, the depth between the vase and the wall may differ in the area where the vase is located in the captured image (1210) of the projection surface, resulting in line breaking. In this case, the electronic device (100) may determine an output area to output the projection image by avoiding the area.

[0196] FIGS. 13A to 13C are drawings for explaining chroma information according to one or more embodiments.

[0197] According to one embodiment, the electronic device (100) can convert a captured image (1310) including a grid area such as FIG. 13A into a YUV image (1320) such as FIG. 13B. The electronic device (100) can obtain chroma information (1330) including chroma values ​​of each grid area as illustrated in FIG. 13C based on a UV signal in the YUV image (1320). The chroma information can include color change information between adjacent grid areas for each grid area in the grid output image. According to one example, at least one of a standard deviation, a gradient, or a variance can be used to obtain chroma information. According to one example, the processor (140) can obtain a standard deviation (σ) of chroma values ​​obtained based on a UV signal value of each grid area in the captured image. C ) is the threshold value (σ Thr3 ) can be used to derive chroma information for each grid area based on whether it is abnormal or not.

[0198] FIG. 14 is a drawing for explaining a method for identifying an output area according to one or more embodiments.

[0199] According to one embodiment, the electronic device (100) can identify an output area on the projection surface by applying a preset algorithm to the characteristic information produced for each grid area. For example, as illustrated in FIG. 14, the electronic device (100) can classify a projection surface, for example, a wall surface, into a plurality of areas (c1, c2, c3, c4) based on characteristics using a machine learning algorithm such as a decision tree.

[0200] In one embodiment, the electronic device (100) may utilize distance information between each grid area in a clustering algorithm to identify the largest possible output area on the projection surface. In one example, the processor (140) may classify grid areas with close distances as end terminals in the clustering algorithm.

[0201] For example, the electronic device (100) may perform a clustering algorithm by setting the distance between all grid areas adjacent to a specific grid area as a first variable (x1), and the characteristic information values ​​corresponding to the specific grid area, for example, values ​​corresponding to line continuity information, texture complexity information, flatness information, and chroma information, as second to fifth variables (x2 to x5), respectively. For example, the electronic device (100) may calculate the distance between all grid areas adjacent to a specific grid area as the first variable (x1) based on the Manhattan distance as in the mathematical expression 1 above.

[0202] For example, each variable can be classified into two groups based on whether it is less than the threshold value s, as shown in Equation 6 below.

[0203]

[0204] For example, the electronic device (100) can determine the final output area based on the general formula of the decision tree as shown in the following mathematical expression 7.

[0205]

[0206] c1 and c2 are the mean values ​​of each class, and yi can be the characteristic information values ​​corresponding to each grid.

[0207] FIG. 15 is a diagram illustrating a method for determining a final output area according to one or more embodiments.

[0208] According to one embodiment, the electronic device (100) can output a frame image (1510) to a projection surface and output a pattern image (1520) including a grid area at a specific interval.

[0209] According to one embodiment, the electronic device (100) may obtain characteristic information (1530) based on a captured image of a projection surface on which a pattern image (1520) is output. For example, the characteristic information (1530) may include at least one of line continuity information, texture information, flatness information, or chroma information. According to one example, the electronic device (100) may use characteristic information selected by a user from among a plurality of characteristic information to determine an output area. According to one example, the electronic device (100) may use characteristic information corresponding to an input image type from among a plurality of characteristic information to identify an output area.

[0210] According to one embodiment, the electronic device (100) can divide the projection surface into a plurality of areas based on characteristic information (1530) (1540) and select a final output area among the plurality of areas (1550).

[0211] For example, the electronic device (100) calculates an integrated characteristic value by calculating the characteristic value of each grid area calculated for each type of characteristic information based on a weight for each characteristic information, and sets boundaries (t1, t2, t3, t4) based on an average value or a median value of the integrated characteristic value, thereby dividing the projection surface into a plurality of areas. In this case, the electronic device (100) can determine an output area based on the characteristic values ​​corresponding to the plurality of areas and the number of grid areas included in the plurality of areas.

[0212] For example, the electronic device (100) may determine an output area by identifying an area in which the value of other characteristic information is greater than a threshold value in an area in which the value of the characteristic information with the highest weight among each characteristic information, for example, line continuity information, is greater than a threshold value.

[0213] For example, the electronic device (100) may classify a plurality of grid areas using a decision tree based on characteristic values ​​corresponding to each of the plurality of grid areas, and may determine an output area by selecting a node among the nodes included in the decision tree that has the largest number of grid areas included in the node.

[0214] However, this is only an example, and other machine learning algorithms or ensemble methods can be used to determine new output regions. This allows the projection image to be output on a flat surface, unobstructed by other objects or uneven walls.

[0215] FIG. 16 is a diagram illustrating a method for determining a final output area according to one or more embodiments.

[0216] According to one embodiment, the electronic device (100) can perform keystone correction on the projection area (1610) in a projection environment without a dedicated screen to identify a keystone-corrected projection area (1620). Then, the electronic device (100) can identify the largest area having uniform characteristics in the keystone-corrected projection area (1620) as the final projection area (1630) through characteristic information analysis as described above for the keystone-corrected projection area (1620), if the keystone-corrected projection area (1620) is an area having non-uniform characteristics. Accordingly, the user can view a clear and crisp image.

[0217] FIG. 17 is a diagram illustrating a method for determining a final output area according to one or more embodiments.

[0218] According to one embodiment, the electronic device (100) can identify a keystone-corrected projection area in a projection environment without a dedicated screen and output a frame image (1710). Subsequently, the electronic device (100) can output a pattern image (1720) together with the frame image (1710). Subsequently, the electronic device (100) can identify a maximum-sized area having uniform characteristics within the frame image (1710) as a final projection area (1730) based on a captured image of the projection surface. Accordingly, a user can view a clear and crisp image.

[0219] Each operation according to the various embodiments described above may be performed by the processor (140), but if necessary, a module for each operation may be utilized. For example, each module may be implemented using at least one software, at least one hardware, and / or a combination thereof. Each module may be implemented to utilize a predefined algorithm, a predefined formula, and / or a learned artificial intelligence model to perform the operation. However, at least some modules may be distributed to an external device.

[0220] According to the various embodiments described above, optimal image projection is possible by searching and determining the optimal projection area even without a dedicated screen. In particular, stable projection image output is possible even on irregular projection surfaces, thereby enhancing user convenience.

[0221] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device and / or server.

[0222] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0223] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0224] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0225] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0226] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure. It will also be understood that any embodiment(s) described in the present disclosure may be used in conjunction with any other embodiment(s) described in the present disclosure.

Claims

1. In electronic devices, Video projection unit; camera; memory that stores instructions; and one or more processors including processing circuitry; One or more of the above processors, When the above instructions are executed individually or collectively, Control the image projection unit to output an image including a preset pattern to the projection surface, Obtain a video image of the projection surface captured through the above camera, Identifying characteristic information including continuity information of a line included in the preset pattern in the above-described captured image, Identifying at least a portion of the projection surface as an output area based on the above characteristic information, An electronic device configured to control the image projection unit to project an input image onto the identified output area.

2. In paragraph 1, The one or more processors, when the instructions are executed individually or collectively, Controlling the image projection unit to output the image including a plurality of grid areas defined by grid lines to the projection surface, Identifying the continuity information of the grid line in the above-mentioned captured image, An electronic device configured to identify at least a portion of the projection surface as the output area based on continuity information of the grid lines.

3. In paragraph 2, The one or more processors, when the instructions are executed individually or collectively, Identifying characteristic information including at least one of texture complexity information including fine texture information of each grid area in the above-mentioned captured image, homogeneity information including brightness change information between adjacent grid areas adjacent to each grid area, or chroma information including color change information between adjacent grid areas adjacent to each grid area, An electronic device configured to identify the output area on the projection surface based on at least one of the continuity information of the line, the texture complexity information, the flatness information, or the chroma information.

4. In paragraph 3, The one or more processors, when the instructions are executed individually or collectively, Convert the above-mentioned captured video into a YUV video, Identifying at least one of the continuity information of the line, the texture complexity information, and the flatness information based on the Y signal included in the YUV image, An electronic device configured to identify the chroma signal based on a UV signal included in the YUV image.

5. In paragraph 4, The one or more processors, when the instructions are executed individually or collectively, Separate the Y signal included in the YUV image into a reflectance component and an illumination component, By applying a straight line estimation algorithm to the above reflection component, line continuity information corresponding to each grid area is identified, Identifying texture complexity information corresponding to each grid area based on the difference between the first contour information identified in the Y signal and the second contour information identified in the reflection component, An electronic device configured to identify flatness information corresponding to each grid area based on whether the standard deviation of the flatness values ​​identified in the above reflection components is greater than or equal to a threshold value.

6. In paragraph 4, The one or more processors, by executing the instructions, Separate the UV signal included in the YUV image into a reflection component and an illumination component, An electronic device configured to identify chroma information corresponding to each grid area based on the distance between the U component and the V component identified in the above reflection component.

7. In paragraph 3, The one or more processors, when the instructions are executed individually or collectively, An electronic device configured to identify the output area on the projection surface by applying different weights to each characteristic information according to the importance of each characteristic information obtained for each grid area.

8. In paragraph 3, The one or more processors, by executing the instructions, An electronic device that identifies the output area on the projection surface by applying different weights for each characteristic information based on at least one of the types or categories of the input image.

9. In paragraph 3, The one or more processors, when the instructions are executed individually or collectively, An electronic device configured to identify the output area on the projection surface by applying a dot product operation or a machine learning clustering algorithm to characteristic information acquired for each grid area.

10. In paragraph 3, The one or more processors, when the instructions are executed individually or collectively, An electronic device configured to use at least one learned artificial intelligence model for at least one of the above characteristic information identification or the above output area identification.

11. In paragraph 1, The one or more processors, when the instructions are executed individually or collectively, An electronic device configured to control the image projection unit to adjust the input image to correspond to the size of the identified output area and project it onto the identified output area.

12. In a method for controlling an electronic device, A step of outputting an image including a preset pattern to a projection surface; A step of acquiring a photographed image of the above projection surface; A step of identifying characteristic information including continuity information of a line included in the preset pattern in the above-described captured image; A step of identifying at least a portion of the projection surface as an output area based on the above characteristic information; and A control method comprising the step of projecting an input image onto the identified output area.

13. In paragraph 12, The step of outputting an image including the above-described pattern to a projection surface is as follows: Outputting the image including a plurality of grid areas defined by grid lines to the projection surface, The step of identifying the above characteristic information is: Identifying the continuity information of the grid line in the above-mentioned captured image, The step of identifying the above output area is: A control method for identifying at least a portion of the projection surface as the output area based on continuity information of the grid line.

14. In paragraph 13, The step of identifying the above characteristic information is: Identifying characteristic information including at least one of texture complexity information including fine texture information of each grid area in the above-mentioned captured image, homogeneity information including brightness change information between adjacent grid areas adjacent to each grid area, or chroma information including color change information between adjacent grid areas adjacent to each grid area, The step of identifying the above output area is: A control method for identifying the output area on the projection surface based on at least one of the continuity information of the line, the texture complexity information, the flatness information, or the chroma information.

15. A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, The above action is, A step of outputting an image including a preset pattern to a projection surface; A step of acquiring a photographed image of the above projection surface; A step of identifying characteristic information including continuity information of a line included in the preset pattern in the above-described captured image; A step of identifying at least a portion of the projection surface as an output area based on the above characteristic information; and A non-transitory computer-readable medium comprising: a step of projecting an input image onto the identified output area;

Images