Electronic device for compensating for external area according to keystone correction, and control method therefor

Abstract

An electronic device is disclosed. The electronic device comprises: a projection unit for projecting an image of a specified resolution; a camera; and at least one processor including a processing circuit that is connected to the projection unit and the camera and controls the electronic device, wherein the at least one processor can: individually and / or collectively perform keystone correction on a first image; control the projection unit such that an image with specified resolution, in which the keystone-corrected first image is included in one area and a second image is included in the remaining area, is projected in a projection area; capture the projection area and an area outside the projection area through the camera so as to acquire a captured image; adjust a pixel value of the second image on the basis of first pixel information corresponding to the remaining area in the captured image and second pixel information corresponding to the area outside the projection area; and control the projection unit such that the image with specified resolution, in which the keystone-corrected first image is included in one area and the second image having the adjusted pixel value is included in the remaining area, is projected in the projection area.

Description

Electronic device for compensating for external areas due to keystone correction and control method thereof

[0001] The present disclosure relates to an electronic device and a control method thereof, for example, to an electronic device and a control method thereof for compensating for an external area according to keystone correction, and a control method thereof.

[0002] With the advancement of electronic technology, electronic devices offering a variety of functions are being developed. In particular, various types of projectors have recently become widespread.

[0003] A projector is a device that projects images. It has the advantage of being able to easily create large screens compared to other types of display devices by projecting light onto the projection area to create images.

[0004] Recently, projectors that provide keystone correction functions have become more widespread, improving user convenience.

[0005] According to one embodiment of the present disclosure for achieving the above object, an electronic device includes a projection unit for projecting an image of a specified resolution, a camera, and at least one processor including a processing circuit connected to the projection unit and the camera and configured to individually and / or collectively control the electronic device, wherein the at least one processor individually and / or collectively performs keystone correction on a first image, and controls the projection unit to project an image of the specified resolution, in which the keystone-corrected first image is included in one area and a second image is included in a remaining area, onto a projection area, and acquires a captured image including a photograph of the projection area and an area outside the projection area through the camera, and adjusts a pixel value of the second image based on first pixel information corresponding to the remaining area and second pixel information corresponding to the area outside the projection area in the captured image, and controls the projection unit to project an image of the specified resolution, in which the keystone-corrected first image is included in one area and a second image with the adjusted pixel value is included in the remaining area, onto the projection area.

[0006] Additionally, the image is an image of a specified color, and the at least one processor can individually and / or collectively adjust pixel values ​​of the second image so that a difference between the first pixel information and the second pixel information is less than a threshold value.

[0007] And, the at least one processor may individually and / or collectively acquire the captured image including a photograph of the projection area and the external area through the camera at a time interval specified, and adjust a pixel value of the second image based on the first pixel information corresponding to the remaining area and the second pixel information corresponding to the external area in each of the plurality of captured images acquired at the time interval specified, and when a difference between the first pixel information and the second pixel information becomes less than a threshold value, the acquisition of the captured image and the adjustment of the pixel value may be stopped.

[0008] Additionally, the at least one processor can individually and / or collectively identify a pattern corresponding to an area outside the projection area in the captured image, and adjust pixel values ​​of the second image based on the identified pattern.

[0009] And, the at least one processor can individually and / or collectively adjust pixel values ​​of the second image based on at least one of luminance or color of the keystone corrected image.

[0010] In addition, the device further includes a sensor, and the at least one processor can individually and / or collectively obtain sensing information including ambient illumination of the electronic device through the sensor, and adjust pixel values ​​of the second image further based on the sensing information.

[0011] And, further including a sensor, the at least one processor can individually and / or collectively obtain sensing information including an angle by which the electronic device is rotated in the direction of gravity and an angle by which the electronic device is rotated relative to the projection area through the sensor, and change the shape of the first image based on the sensing information to perform the keystone correction for the first image.

[0012] Additionally, the user interface further comprises at least one processor capable of individually and / or collectively adjusting pixel values ​​of the second image based on a color corresponding to a user command received through the user interface.

[0013] And, the projection area may be an area where light emitted from the projection unit is projected.

[0014] Meanwhile, according to one embodiment of the present disclosure, a control method of an electronic device may include the steps of performing keystone correction on a first image, projecting an image of a specified resolution, in which the keystone-corrected first image is included in one area and a second image is included in a remaining area, onto a projection area, obtaining a captured image including a photograph of the projection area and an area outside the projection area, adjusting a pixel value of the second image based on first pixel information corresponding to the remaining area and second pixel information corresponding to an area outside the projection area in the captured image, and projecting an image of the specified resolution, in which the keystone-corrected first image is included in one area and a second image with the adjusted pixel value is included in the remaining area, onto the projection area.

[0015] Additionally, the image is an image of a specified color, and the adjusting step may adjust a pixel value of the second image so that a difference between the first pixel information and the second pixel information is less than a threshold value.

[0016] And, the acquiring step acquires the captured image including a photograph of the projection area and the external area at a specified time interval, and the adjusting step adjusts the pixel value of the second image based on the first pixel information corresponding to the remaining area and the second pixel information corresponding to the external area in each of the plurality of captured images acquired at the specified time interval, and the method may further include a step of stopping the acquisition of the captured image and the adjustment of the pixel value when a difference between the first pixel information and the second pixel information becomes less than a threshold value.

[0017] In addition, the step of identifying a pattern corresponding to an area outside the projection area in the photographed image is further included, and the adjusting step can adjust a pixel value of the second image based on the identified pattern.

[0018] And, the adjusting step may adjust the pixel values ​​of the second image based on at least one of the brightness or color of the keystone-corrected image.

[0019] In addition, the method further includes a step of acquiring sensing information including ambient illumination of the electronic device, and the adjusting step may adjust pixel values ​​of the second image further based on the sensing information.

[0020] And, the step of obtaining sensing information including an angle at which the electronic device is rotated in the direction of gravity and an angle at which the electronic device is rotated with respect to the projection area is further included, and the performing step can perform the keystone correction for the first image by changing the shape of the first image based on the sensing information.

[0021] Additionally, the adjusting step may adjust pixel values ​​of the second image based on a color corresponding to a user command.

[0022] And, the projection area may be an area where light emitted from the electronic device is projected.

[0023] Meanwhile, according to one embodiment of the present disclosure, a non-transitory computer-readable recording medium stores a program that, when individually and / or collectively executed by one or more processors of the electronic device, causes the electronic device to perform operations including the following, wherein the operations may include the steps of performing keystone correction on a first image, projecting an image of a specified resolution, in which the keystone-corrected first image is included in one area and a second image is included in a remaining area, onto a projection area, obtaining a captured image including a photograph of the projection area and an area outside the projection area, adjusting pixel values ​​of the second image based on first pixel information corresponding to the remaining area and second pixel information corresponding to the area outside the projection area in the captured image, and projecting an image of the specified resolution, in which the keystone-corrected first image is included in one area and a second image with the adjusted pixel values ​​is included in the remaining area, onto the projection area.

[0024] The above and other aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0025] FIG. 1 is a drawing illustrating exemplary keystone correction of an image to help understand the present disclosure.

[0026] FIG. 2 is a block diagram showing an exemplary configuration of an electronic device according to various embodiments of the present disclosure.

[0027] FIG. 3 is a block diagram showing an exemplary configuration of an electronic device according to various embodiments of the present disclosure.

[0028] FIG. 4 is a drawing for explaining exemplary keystone correction according to various embodiments of the present disclosure.

[0029] FIG. 5 is a drawing illustrating an external area due to exemplary keystone correction according to various embodiments of the present disclosure.

[0030] FIG. 6 is a flowchart illustrating an exemplary method of keystone correction and outer region compensation according to various embodiments of the present disclosure.

[0031] FIG. 7 is a diagram illustrating an exemplary method for performing keystone correction and outer area compensation according to various embodiments of the present disclosure.

[0032] FIG. 8 is a flowchart illustrating an exemplary control method of an electronic device according to various embodiments of the present disclosure.

[0033] The various embodiments of the present disclosure are susceptible to various modifications. Accordingly, various embodiments are illustrated in the drawings and described in detail in the detailed description. However, it should be understood that the present disclosure is not limited to the specific embodiments, but rather encompasses all modifications, equivalents, and alternatives that do not depart from the scope and spirit of the present disclosure. Furthermore, well-known functions or components may not be described in detail if unnecessary detail would obscure the disclosure.

[0034] The present disclosure provides an electronic device and a control method thereof for minimizing and / or reducing a user's discomfort by compensating for an external area according to keystone correction.

[0035] Hereinafter, the present disclosure will be described in more detail with reference to the attached drawings.

[0036] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected, in which case their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure are not simply defined as names, but are defined based on the meanings of the terms and the overall content of this disclosure.

[0037] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0038] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0039] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0040] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0041] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0042] Various exemplary embodiments of the present disclosure are described in more detail with reference to the attached drawings below.

[0043] FIG. 1 is a drawing illustrating exemplary keystone correction of an image to help understand the present disclosure.

[0044] When the projector is placed flat and level, facing the screen, a rectangular image can be displayed on the screen. Conversely, if the projector is not placed flat and level, the image may be distorted vertically or horizontally, or may appear rotated. This distortion is called the keystone effect, and the function that eliminates it is called keystone correction.

[0045] For example, if the projector is not placed flat on a surface facing the screen, a projection area such as area 101 in Fig. 1 may be formed. That is, the projector can project an image onto area 101. The projection area may be an area onto which light emitted from the projector is projected.

[0046] A projector can project an image onto a portion of a projection area, such as area 102 in FIG. 1, through keystone correction. However, even in this case, light emitted from the projector may be projected onto areas outside of the projection area, excluding area 102. For example, the projector can project black onto areas outside of the projection area, excluding area 102. However, since a projector is a device that displays color on a screen using a light source, even if black is projected, it may provide a sense of incongruity to the user. In other words, the user may feel a sense of incongruity due to the difference between areas outside of the projection area, excluding area 102, and areas outside of the projection area.

[0047] FIG. 2 is a block diagram showing an exemplary configuration of an electronic device (100) according to various embodiments of the present disclosure.

[0048] The electronic device (100) may be a device that changes the position of the electronic device (100) and projects content. For example, the electronic device (100) may be a projector capable of moving on its own.

[0049] The electronic device (100) may be a device that projects an image onto a screen. For example, the electronic device (100) may be a projector equipped with a keystone function.

[0050] According to FIG. 1, the electronic device (100) includes a projection unit (110), a camera (120), and a processor (e.g., including a processing circuit, 130).

[0051] The projection unit (110) can project an image of a preset (e.g., designated) resolution onto a projection surface. For example, the projection unit (110) can project an image or video including at least one of content received from a source device and pre-stored content onto a projection area using a light source such as a lamp or LED.

[0052] The camera (120) is configured to capture still images or moving images. The camera (120) can capture still images at a specific point in time, but can also capture still images continuously.

[0053] The camera (120) can capture the front of the electronic device (100) to capture the area where content is projected. The processor (130) can perform image processing based on the captured image captured by the camera (120).

[0054] The camera (120) may include, for example, a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time at which light reflected from a subject enters the camera (120), and the aperture controls the amount of light incident on the lens by mechanically increasing or decreasing the size of the opening through which light enters. The solid-state image sensor outputs an image by the photocharge as an electrical signal when light reflected from a subject is accumulated as a photocharge. The TG outputs a timing signal for reading out pixel data of the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0055] The processor (130) includes various processing circuits and controls the overall operation of the electronic device (100). For example, the processor (130) may be connected to each component of the electronic device (100) and control the overall operation of the electronic device (100). For example, the processor (130) may be connected to components such as a projection unit (110), a camera (120), a sensor (not shown), a user interface (not shown), and the like and control the operation of the electronic device (100).

[0056] The one or more processors (130) may include one or more of a CPU, a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), an NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. The one or more processors (130) may control one or any combination of other components of the electronic device (100) and perform operations related to communication or data processing. The one or more processors (130) may execute one or more programs or instructions stored in a memory. For example, the one or more processors (130) may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0057] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0058] One or more processors (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (130) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0059] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0060] In various embodiments of the present disclosure, one or more processors (130) may represent, for example, a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, an NPU, a hardware accelerator, or a machine learning accelerator, but the various embodiments of the present disclosure are not limited thereto. However, for the convenience of explanation, the operation of the electronic device (100) will be described below with the expression processor (130). In other words, the processor (130) may include various processing circuits and / or multiple processors. For example, the term "processor" used in the present disclosure, including the claims, may include various processing circuits, including at least one processor, wherein one or more of the at least one processor may be configured to individually and / or collectively perform various functions described in the present disclosure in a distributed manner. When the terms "processor," "at least one processor," and "one or more processors" are used herein to describe a processor configured to perform a number of functions, these terms encompass, for example and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other of the recited functions, and situations where a single processor can perform all of the recited functions. Furthermore, the at least one processor may comprise a combination of processors that perform various recited / disclosed functions, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0061] The processor (130) can perform keystone correction on the first image and control the projection unit (110) to project an image of a preset resolution, in which the keystone-corrected first image is included in one area and the second image is included in the remaining area, onto a projection area. Here, the projection area may be an area onto which light emitted from the projection unit (110) is projected.

[0062] For example, the processor (130) can keystone correct a first image having a resolution of 1920×1080. In this case, the resolution of the keystone-corrected first image may be lower than the resolution of 1920×1080. When the projection unit (110) projects an image having a resolution of 1920×1080, an image having a resolution of 1920×1080 is required, so the processor (130) can control the projection unit (110) to obtain an image having a resolution of 1920×1080, in which the keystone-corrected first image is included in one area and the second image is included in the remaining area, and to project the obtained image onto the projection area. The second image may be a black image. However, the present invention is not limited thereto, and the second image may not be black, and may be an image having a preset color set at the time of manufacturing. In addition, the first image may be an image scaled to a resolution that the projection unit (110) can output. For example, if the resolution that the projection unit (110) can output is 1920×1080 resolution and the original image is 3840×2160 resolution, the processor (130) can downscale the original image to obtain the first image with a resolution of 1920×1080. Therefore, when the first image is keystone corrected, the resolution of the keystone-corrected first image may be smaller than the resolution that the projection unit (110) can output. Here, the downscaling may be performed for each frame after decoding the original image.

[0063] When an image including a keystone-corrected first image in one area and a second image in the remaining area is projected onto a projection area, a first area corresponding to the first image in the projection area and a second area corresponding to the second image in the projection area can be distinguished. However, even if the projection unit (110) projects a black-colored image onto the second area, the light from the projection unit (110) is not blocked, but rather light for expressing the black color is projected onto the second area, so the color of the second area may be different from that of an area outside the projection area.

[0064] Since the user may feel a sense of incongruity due to the color difference, in order to reduce the sense of incongruity, the processor (130) may acquire a photographed image by photographing the projection area and the area outside the projection area through the camera (120), adjust the pixel value of the second image based on the first pixel information corresponding to the remaining area in the photographed image and the second pixel information corresponding to the area outside the projection area, and control the projection unit (110) to project an image of a preset resolution, in which the keystone-corrected first image is included in one area and the second image with the adjusted pixel value is included in the remaining area, onto the projection area.

[0065] For example, the preset image may include an image of a preset color, and the processor (130) may control the projection unit (110) to adjust the pixel values ​​of the second image so that the difference between the first pixel information and the second pixel information is less than a threshold value, and to project an image of a preset resolution, in which the keystone-corrected first image is included in one area and the second image with the adjusted pixel values ​​is included in the remaining area, onto the projection area.

[0066] The processor (130) can reduce the difference between the second area and the external area of ​​the projection area by repeating the above-described operation.

[0067] For example, the processor (130) may acquire captured images of the projection area and the external area through the camera (120) at preset time intervals, and adjust pixel values ​​of the second image based on first pixel information corresponding to the remaining area and second pixel information corresponding to the external area in each of the plurality of captured images acquired at preset time intervals. The processor (130) may repeat this operation, and when the difference between the first pixel information and the second pixel information becomes less than a threshold value, the processor may stop acquiring the captured images and adjusting the pixel values.

[0068] The processor (130) may identify a pattern corresponding to an area outside the projection area in the captured image and adjust pixel values ​​of the second image based on the identified pattern. For example, if the screen includes a preset pattern, the area outside the projection area in the captured image may be expressed by the preset pattern. The processor (130) may identify the preset pattern in the captured image and adjust pixel values ​​so that the second image includes the preset pattern.

[0069] The processor (130) may also adjust pixel values ​​of the second image by further considering (or based on) at least one of the luminance or color of the keystone-corrected image.

[0070] The electronic device (100) further includes a sensor, and the processor (130) obtains sensing information including the ambient illuminance of the electronic device through the sensor, and may adjust the pixel values ​​of the second image by further considering the sensing information.

[0071] The electronic device (100) further includes a sensor, and the processor (130) obtains sensing information including an angle at which the electronic device is rotated in the direction of gravity and an angle at which the electronic device is rotated relative to a projection area through the sensor, and can perform keystone correction for the first image by changing the shape of the first image based on the sensing information.

[0072] The electronic device (100) further includes a user interface, and the processor (130) may adjust pixel values ​​of the second image based on a color corresponding to a user command received through the user interface.

[0073] However, the present invention is not limited thereto, and the processor (130) may perform at least one of keystone correction or adjustment of pixel values ​​of the second image based on a user command.

[0074] The processor (130) may also recommend to the user a plurality of colors for adjusting pixel values ​​of the second image. For example, the processor (130) may obtain pixel values ​​for adjusting pixel values ​​of the second image based on first pixel information corresponding to the remaining area in the captured image and second pixel information corresponding to an area outside the projection area, and may recommend to the user a plurality of pixel values ​​by modifying the obtained pixel values ​​using a plurality of preset methods. The processor (130) may obtain a plurality of pixel values ​​by changing at least one of brightness and white balance of the obtained pixel values.

[0075] The processor (130) can control the projection unit (110) to project multiple colors. The processor (130) can adjust the pixel values ​​of the second image to the selected color according to a user command to select one of the multiple colors.

[0076] However, the present invention is not limited thereto, and the electronic device (100) may further include a display, and the processor (130) may control the display to display multiple colors. The processor (130) may provide multiple colors to the user terminal device, and when the user selects one of the multiple colors using the user terminal device and information about the selected color is received from the user terminal device, the pixel values ​​of the second image may be adjusted to the selected color.

[0077] Meanwhile, the processor (130) may input the captured image into a neural network model to obtain color information for adjusting pixel values ​​of the second image. The neural network model may represent, for example, a model that has learned the relationship between a sample captured image and sample color information for adjusting pixel values ​​of the second image.

[0078] The functions related to artificial intelligence according to the present disclosure can be operated through a processor (130) and memory.

[0079] The processor (130) may include one or more processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, AP, DSP, etc., a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU.

[0080] One or more processors can be controlled to process input data according to predefined operating rules or artificial intelligence models stored in memory. If one or more processors are dedicated AI processors, the dedicated AI processors may be designed with a hardware structure specialized for processing a specific AI model. The predefined operating rules or artificial intelligence models are characterized by being created through learning.

[0081] Creating through learning may mean, for example, that a basic artificial intelligence model is learned using a learning algorithm using a large amount of learning data, thereby creating a predefined set of operating rules or an artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself where the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc.

[0082] An artificial intelligence model may include multiple neural network layers. Each of the multiple neural network layers may have multiple weight values, and perform neural network operations by calculating the results of previous layers and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated during the learning process to reduce or minimize the loss or cost values ​​obtained by the artificial intelligence model.

[0083] Artificial neural networks may include deep neural networks (DNNs), such as, but not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), or deep Q-networks.

[0084] FIG. 3 is a block diagram showing a detailed configuration of an electronic device (100) according to an embodiment of the present disclosure. The electronic device (100) may include a projection unit (110), a camera (120), and a processor (130). The electronic device (100) may further include a sensor (140), a user interface (150), a communication interface (160), a memory (170), a display (180), a microphone (185), and a speaker (190). For components illustrated in FIG. 3 that overlap with those illustrated in FIG. 2, a detailed description thereof will be omitted.

[0085] The sensor (140) may include a sensor for identifying the distance from the electronic device (100) to the projection surface. For example, the sensor (140) may include a ToF sensor. However, the present invention is not limited thereto, and the sensor (140) may be any sensor that can identify the distance from the electronic device (100) to the projection surface. In addition, the processor (130) may identify the distance from the electronic device (100) to the projection surface through the camera (120).

[0086] The user interface (150) may be implemented with buttons, a touch pad, a mouse, a keyboard, etc., or may be implemented with a touch screen capable of performing both display and operation input functions. Here, the buttons may be various types of buttons, such as mechanical buttons, touch pads, wheels, etc., formed on any area of ​​the front, side, or back of the main body of the electronic device (100).

[0087] The communication interface (160) is a component that performs communication with various types of external devices according to various types of communication methods. For example, the electronic device (100) can perform communication with a content server or a user terminal device through the communication interface (160).

[0088] The communication interface (160) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0089] Wi-Fi and Bluetooth modules communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received. Infrared communication modules use infrared data association (IrDA) technology, which wirelessly transmits data over short distances using infrared light, which lies between visible light and millimeter waves.

[0090] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0091] Alternatively, the communication interface (160) may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0092] In addition, the communication interface (160) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or an optical fiber cable.

[0093] Memory (170) may refer to hardware that stores information such as data in an electrical or magnetic form so that the processor (130) or the like can access it. To this end, the memory (170) may be implemented as at least one piece of hardware from among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), solid state drive (SSD), RAM, ROM, etc.

[0094] The memory (170) may store at least one instruction required for the operation of the electronic device (100) or the processor (130). Here, the instruction is a unit of code that instructs the operation of the electronic device (100) or the processor (130), and may be written in machine language, which is a language that a computer can understand. Alternatively, the memory (170) may store a plurality of instructions for performing a specific task of the electronic device (100) or the processor (130) as an instruction set.

[0095] The memory (170) may store data, which is information in bit or byte units that can represent characters, numbers, images, etc. For example, a keystone correction module, a photographed image analysis module, etc. may be stored in the memory (170).

[0096] The memory (170) is accessed by the processor (130), and reading / writing / modifying / deleting / updating instructions, instruction sets, or data can be performed by the processor (130).

[0097] The display (180) is a configuration that displays content and can be implemented as a variety of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (180) may also include a driving circuit, a backlight unit, etc. that can be implemented as a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (180) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0098] The microphone (185) is configured to receive sound and convert it into an audio signal. The microphone (185) is electrically connected to the processor (130) and can receive sound under the control of the processor (130).

[0099] For example, the microphone (185) may be formed as an integrated unit integrated into the upper side, front side, side side, etc. of the electronic device (100). Alternatively, the microphone (185) may be provided in a remote control separate from the electronic device (100). In this case, the remote control may receive sound through the microphone (185) and provide the received sound to the electronic device (100).

[0100] The microphone (185) may include various configurations such as a microphone that collects sound in analog form, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0101] Meanwhile, the microphone (185) may be implemented in the form of a sound sensor, and any method may be used as long as it has a configuration capable of collecting sound.

[0102] The speaker (190) is a component that outputs various audio data processed by the processor (130) as well as various notification sounds and voice messages.

[0103] As described above, the electronic device (100) can minimize the user's sense of incongruity by compensating for the external area according to keystone correction, thereby increasing the user's sense of immersion.

[0104] Hereinafter, the operation of the electronic device (100) will be described in more detail with reference to FIGS. 4 to 7. For convenience of explanation, individual embodiments are described in FIGS. 4 to 7. However, the individual embodiments of FIGS. 4 to 7 may be implemented in any combination.

[0105] FIG. 4 is a drawing for explaining keystone correction according to one embodiment of the present disclosure.

[0106] The processor (130) can obtain sensing information including the angle at which the electronic device (100) is rotated in the direction of gravity and the angle at which the electronic device (100) is rotated relative to the projection area through the sensor (140), and can identify whether keystone correction is to be performed based on the obtained sensing information.

[0107] For example, the processor (130) may obtain an angle at which the electronic device (100) is rotated in the direction of gravity through the acceleration sensor (140-1), obtain an angle at which the electronic device (100) is rotated with respect to the projection area through the plurality of distance sensors (140-2, 140-3), and if it is determined based on the obtained angles that the electronic device (100) is not placed parallel to the screen and thus the projection area is not rectangular, it may determine to perform keystone correction. Here, if the processor (130) is determined to perform keystone correction, it may perform keystone correction on the first image based on the obtained angles. For example, the processor (130) may obtain a projection matrix and a transformation matrix based on the posture of the electronic device (100) expressed as a roll angle, a pitch angle, and a yaw angle, and the distance (d1, d2) to the screen, and perform keystone correction on the first image based on the projection matrix and the transformation matrix.

[0108] However, the present invention is not limited thereto, and the processor (130) may also identify whether keystone correction is to be performed based on a user command. Alternatively, the processor (130) may acquire sensing information, and if it is determined that keystone correction is to be performed based on the acquired sensing information, it may inform the user that keystone correction needs to be performed, and perform keystone correction according to the user's confirmation command.

[0109] The processor (130) may acquire sensing information when the electronic device (100) is turned on, and determine whether to perform keystone correction based on the acquired sensing information. Alternatively, if the processor (130) determines that the posture of the electronic device (100) has changed while projecting an image, the processor (130) may acquire sensing information, and determine whether to perform keystone correction based on the acquired sensing information. In this case, the processor (130) may keep one of the acceleration sensor or multiple distance sensors turned on at all times. For example, the processor (130) may turn on the acceleration sensor to detect a change in posture, and if it is determined that the posture has changed, the processor (130) may turn on the multiple distance sensors to acquire sensing information, and determine whether to perform keystone correction based on the acquired sensing information. Alternatively, the processor (130) may turn on multiple distance sensors to obtain an angle at which the electronic device (100) is rotated relative to the screen, turn on an acceleration sensor to obtain sensing information when it is determined that the posture has changed, and determine whether to perform keystone correction based on the obtained sensing information.

[0110] The above description uses an acceleration sensor and multiple distance sensors, but is not limited thereto. For example, the processor (130) may capture a screen image via a camera (120) and determine whether keystone correction is to be performed based on the captured image.

[0111] FIG. 5 is a drawing for explaining an external area due to keystone correction according to one embodiment of the present disclosure.

[0112] The projection unit (110) can project an image with a preset resolution. For example, the projection unit (110) can project an image with a resolution of 1920×1080. When projecting an image with a resolution of 1920×1080, the processor (130) can control the projection unit (110) to project the image onto the projection area without scaling. Alternatively, when projecting an image with a resolution other than 1920×1080, the processor (130) can control the projection unit (110) to scale the image to a resolution of 1920×1080 and then project the scaled image onto the projection area. Hereinafter, it will be described that the first image is scaled to a preset resolution that the projection unit (110) can project.

[0113] If the electronic device (100) is not placed parallel to the screen, the projection area may not be rectangular. For example, if the electronic device (100) is not placed parallel to the screen, as shown in the upper part of FIG. 5, the first image of 510-1 may be projected on the screen (520) into a projection area having the same shape as 510-2. In this case, the processor (130) may perform keystone correction on the first image.

[0114] For example, the processor (130) can distort the first image of 510 as shown in the lower part of FIG. 5, such as 530-1, and include it in a portion of 540-1. Here, the area of ​​540-1 can correspond to a preset resolution that the projector (110) can project, and can be data before projection. That is, the data before projection can include the distorted first image in the area of ​​530-1 among the areas of 540-1, and the second image in the remaining area of ​​the area of ​​540-1.

[0115] The processor (130) can project data prior to projection onto the screen. Since the electronic device (100) is not placed parallel to the screen, the area of ​​540-1 can be projected onto the area of ​​540-2 on the screen (520). However, since the first image is expressed using only the area of ​​530-1 among the areas of 540-1, it can be projected as a rectangle onto the area of ​​530-2 on the screen (520). The second image can be projected onto an area of ​​540-2 excluding the area of ​​530-2.

[0116] The processor (130) adjusts the second image based on the area excluding the area of ​​540-2 on the screen (520), so that the difference in color, brightness, etc. between the area excluding the area of ​​530-2 among the areas of 540-2 and the area excluding the area of ​​540-2 on the screen (520) can be minimized, thereby reducing the user's sense of incongruity.

[0117] FIG. 6 is a flowchart illustrating keystone correction and outer area compensation according to one embodiment of the present disclosure.

[0118] First, the processor (130) can identify whether it is in automatic keystone mode (S610).

[0119] When in automatic keystone mode, the processor (130) may acquire sensor information (S620-1), perform coordinate calculation for automatic keystone (S630-1), and perform image processing for keystone correction (S640). Alternatively, when not in automatic keystone mode, the processor (130) may receive user input (S620-2), perform coordinate calculation for manual keystone (S630-2), and perform image processing for keystone correction (S640).

[0120] The processor (130) can project a processed image and identify whether it is in automatic external area correction mode (S650).

[0121] When the processor (130) is in automatic external area correction mode, it may acquire a photographed image (S660-1) and select a keystone external area correction color (S670). Alternatively, when the processor (130) is not in automatic external area correction mode, it may receive user input (S660-2) and select a keystone external area correction color (S670).

[0122] The processor (130) can correct the keystone outer area based on the selected color (S680) and output the corrected image (S690).

[0123] FIG. 7 is a diagram illustrating a method for performing keystone correction and external area compensation according to one embodiment of the present disclosure.

[0124] The processor (130) can perform keystone correction and external area compensation.

[0125] For example, the processor (130) can obtain sensor information using at least one of an acceleration sensor (710-1), a distance sensor (710-2), a 3D TOF sensor (710-3), or an image sensor (710-4), and process the obtained sensor information (720).

[0126] The processor (130) may perform image processing on the original image based on at least one of the processed sensor information or the user input (730). For example, the processor (130) may perform keystone correction on the original image based on at least one of the processed sensor information or the user input.

[0127] The processor (130) can perform external area correction based on at least one of sensor information obtained from the image sensor or user input (S740).

[0128] The processor (130) can mix the processed original image and the corrected external area (750) and output the image (760).

[0129] FIG. 8 is a flowchart for explaining a control method of an electronic device according to an embodiment of the present disclosure.

[0130] First, keystone correction is performed on the first image (S810). Then, an image of a preset resolution, in which the keystone-corrected first image is included in one area and the second image is included in the remaining area, is projected onto the projection area (S820). Then, a captured image of the projection area and an area outside the projection area is acquired (S830). Then, the pixel values ​​of the second image are adjusted based on the first pixel information corresponding to the remaining area and the second pixel information corresponding to the area outside the projection area in the captured image (S840). Then, an image of a preset resolution, in which the keystone-corrected first image is included in one area and the second image with adjusted pixel values ​​is included in the remaining area, is projected onto the projection area (S850).

[0131] Additionally, the preset image is an image of a preset color, and the adjusting step (S840) can adjust the pixel values ​​of the second image so that the difference between the first pixel information and the second pixel information becomes less than a threshold value.

[0132] And, the acquiring step (S830) acquires a photographed image in which the projection area and the external area are captured at preset time intervals, and the adjusting step (S840) adjusts the pixel value of the second image based on the first pixel information corresponding to the remaining area and the second pixel information corresponding to the external area in each of the plurality of photographed images acquired at preset time intervals, and the control method may further include a step of stopping the acquisition of the photographed image and the adjustment of the pixel value when the difference between the first pixel information and the second pixel information becomes less than a threshold value.

[0133] In addition, the step of identifying a pattern corresponding to an area outside the projection area in the captured image is further included, and the adjusting step (S840) can adjust the pixel values ​​of the second image based on the identified pattern.

[0134] And, the adjusting step (S840) can adjust the pixel values ​​of the second image by further considering at least one of the brightness or color of the keystone-corrected image.

[0135] In addition, the step of acquiring sensing information including the ambient illumination of the electronic device is further included, and the adjusting step (S840) can adjust the pixel values ​​of the second image by further considering the sensing information.

[0136] And, the step (S810) further includes a step of acquiring sensing information including an angle at which the electronic device is rotated in the direction of gravity and an angle at which the electronic device is rotated based on the projection area, and the step of performing can perform keystone correction for the first image by changing the shape of the first image based on the sensing information.

[0137] Additionally, the adjusting step (S840) can adjust the pixel values ​​of the second image based on a color corresponding to a user command.

[0138] And, the projection area may be an area where light emitted from an electronic device is projected.

[0139] According to various embodiments of the present disclosure as described above, the electronic device can minimize the user's sense of incongruity by compensating for external areas according to keystone correction, thereby enhancing the user's sense of immersion.

[0140] Meanwhile, computer instructions for performing processing operations of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in such a non-transitory computer-readable medium, when executed by a processor of a specific device, cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

[0141] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0142] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, A projection unit that projects an image of a specified resolution; camera; and At least one processor including a processing circuit connected to the projection unit and the camera and configured to individually and / or collectively control the electronic devices; The at least one processor, individually and / or collectively: Perform keystone correction on the first image, Control the projection unit to project an image of the specified resolution, in which the keystone-corrected first image is included in one area and the second image is included in the remaining area, onto the projection area; By photographing the projection area and the area outside the projection area through the above camera, a photographed image is obtained, Adjusting the pixel value of the second image based on the first pixel information corresponding to the remaining area in the above-mentioned photographed image and the second pixel information corresponding to the outer area of ​​the projection area, An electronic device that controls the projection unit to project an image of the specified resolution, wherein the first image, which has been keystone corrected, is included in the first area and the second image, whose pixel values ​​have been adjusted, is included in the remaining area, onto the projection area.

2. In paragraph 1, The image specified above is, Contains an image of a specified color, The at least one processor, individually and / or collectively: An electronic device that adjusts pixel values ​​of the second image so that a difference between the first pixel information and the second pixel information is less than a threshold value.

3. In paragraph 2, The at least one processor, individually and / or collectively: By photographing the projection area and the external area through the camera at specified time intervals, the photographed image is obtained, Adjusting the pixel value of the second image based on the first pixel information corresponding to the remaining area and the second pixel information corresponding to the external area in each of the plurality of captured images acquired at the specified time interval, An electronic device that stops acquiring the captured image and adjusting the pixel value when the difference between the first pixel information and the second pixel information becomes less than a threshold value.

4. In paragraph 1, The at least one processor, individually and / or collectively: Identify a pattern corresponding to an area outside the projection area in the above captured image, An electronic device that adjusts pixel values ​​of the second image based on the identified pattern.

5. In paragraph 1, The at least one processor, individually and / or collectively: An electronic device that adjusts pixel values ​​of the second image based on at least one of luminance or color of the keystone corrected image.

6. In paragraph 1, sensors; including more; The at least one processor, individually and / or collectively: Obtain sensing information including the ambient illuminance of the electronic device through the above sensor, An electronic device that adjusts pixel values ​​of the second image further based on the sensing information.

7. In paragraph 1, sensors; including more; The at least one processor, individually and / or collectively: Obtain sensing information including the angle at which the electronic device is rotated in the direction of gravity and the angle at which the electronic device is rotated relative to the projection area through the sensor, An electronic device that performs keystone correction for the first image by changing the shape of the first image based on the sensing information.

8. In paragraph 1, further including a user interface; The at least one processor, individually and / or collectively: An electronic device that adjusts pixel values ​​of the second image based on a color corresponding to a user command received through the user interface.

9. In paragraph 1, The above projection area is, An electronic device including an area onto which light emitted from the above projection unit is projected.

10. In a method for controlling an electronic device, Step of performing keystone correction on the first image; A step of projecting an image of a specified resolution, wherein the keystone-corrected first image is included in one area and the second image is included in the remaining area, onto a projection area; A step of obtaining a photographed image by photographing the projection area and an area outside the projection area; A step of adjusting the pixel value of the second image based on first pixel information corresponding to the remaining area in the photographed image and second pixel information corresponding to an area outside the projection area; and A control method comprising the step of projecting an image of the specified resolution onto the projection area, the first image having the keystone correction included in the first area and the second image having the pixel values ​​adjusted included in the remaining area.

11. In paragraph 10, The image specified above is, Contains an image of a specified color, The above adjustment steps are: A control method for adjusting pixel values ​​of the second image so that a difference between the first pixel information and the second pixel information is less than a threshold value.

12. In paragraph 11, The above obtaining steps are: By photographing the projection area and the external area at specified time intervals, the photographed image is obtained, The above adjustment steps are: Adjusting the pixel value of the second image based on the first pixel information corresponding to the remaining area and the second pixel information corresponding to the external area in each of the plurality of captured images acquired at the specified time interval, The above method, A control method further comprising: a step of stopping acquisition of the photographed image and adjustment of the pixel value when the difference between the first pixel information and the second pixel information becomes less than a threshold value.

13. In paragraph 10, Further comprising a step of identifying a pattern corresponding to an area outside the projection area in the above-described photographed image; The above adjustment steps are: A control method for adjusting pixel values ​​of the second image based on the identified pattern.

14. In paragraph 10, The above adjustment steps are: A control method for adjusting pixel values ​​of the second image based on at least one of luminance or color of the keystone corrected image.

15. In paragraph 10, A step of obtaining sensing information including ambient illuminance of the electronic device is further included; The above adjustment steps are: A control method for adjusting pixel values ​​of the second image based further on the sensing information.

Images