Electronic device and control method therefor

Abstract

An electronic device according to one embodiment of the present disclosure comprises: a memory for storing at least one instruction; a communication unit; a camera; and at least one processor, wherein the at least one processor executes the at least one instruction so as to: acquire a projection area and a non-projection area of an external device from an acquired image when the image provided from the external device is present in the image acquired through the camera; acquire, on the basis of the position of each of a plurality of vertices corresponding to the acquired projection area, a corrected area by correcting the projection area; acquire correction coordinates for the projection area on the basis of the position of at least one object included in the correction area; and control the communication unit so that the acquired correction coordinates are provided to the external device.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for obtaining a projection area from a test image and a control method thereof.

[0002] Recent advancements in electronic and optical technology have led to the development of a variety of projectors. A projector is an electronic device that projects light onto a projection surface (or projection surface) to form an image on the surface.

[0003] When projecting an image using a projector, if the projector is placed upright on a flat surface facing the projection surface, a rectangular image is displayed on the projection surface. Otherwise, the image may appear distorted vertically or horizontally, or rotated. This distortion is called the keystone effect.

[0004] To correct this keystone effect, the projector uses an image sensor to capture a picture of the projection surface, and processes the captured image to calculate the angle of deviation between the projection surface and the projector.

[0005] According to one embodiment of the present disclosure, an electronic device includes a memory storing at least one instruction, a communication unit, a camera, and at least one processor, wherein the at least one processor executes the at least one instruction to, when an image provided from an external device is included in an image acquired through the camera, acquire a projection area and a non-projection area of ​​the external device from the acquired image, acquire a correction area that corrects the projection area based on the positions of each of a plurality of vertices corresponding to the acquired projection area, acquire correction coordinates for the projection area based on the position of at least one object included in the correction area, and control the communication unit so that the acquired correction coordinates are provided to the external device.

[0006] The at least one processor, when a test image provided from the external device is confirmed in a first image acquired through the camera, identifies a projection surface on which the test image is projected in the first image among at least one image different from the first image, and a second image acquired in a state where the test image is not projected, and can acquire a projection area and a non-projection area including the plurality of vertices within the first image based on the first image and the second image.

[0007] The at least one processor can identify the second image if there is a test image including a plurality of markers for correcting the projection area in the first image, and can obtain a projection area and a non-projection area including a plurality of vertices located outside the area corresponding to the plurality of markers based on the first image.

[0008] The at least one processor may obtain a plurality of pixel difference values ​​based on a plurality of pixel values ​​included in each of the first image and the second image, and obtain the projection area and the non-projection area based on a difference image including the obtained plurality of pixel difference values.

[0009] The at least one processor can obtain the projection area and the non-projection area based on a plurality of valid pixels corresponding to each of a plurality of pixel difference values ​​that are greater than or equal to a threshold value among a plurality of pixel difference values ​​included in the obtained difference image.

[0010] The at least one processor can obtain an area including a first area occupied by the plurality of valid pixels and at least one second area surrounded by the first area among the plurality of pixels included in the difference image as the projection area, and obtain an area excluding the projection area within the first image as the non-projection area.

[0011] Each of the at least one object may be one of a first type object for projecting an image by the external device and a second type object different from the first type object.

[0012] The at least one processor can identify a plurality of edges in a first correction area obtained by adjusting the position of at least one of the plurality of vertices, identify whether a first type object exists based on whether a preset number of edges among the identified plurality of edges are identified, and obtain the correction coordinates based on whether a second type object exists in an area corresponding to the preset number of edges.

[0013] The at least one processor may obtain correction coordinates corresponding to a second correction area having a maximum size according to the screen ratio of the first correction area included in the remaining area of ​​the area excluding the second type object if the second type object is present in the area, and may obtain the correction coordinates based on whether a difference between the screen ratio of the first type object, which is a rectangle, and the screen ratio of the first correction area is greater than or equal to a threshold value if the second type object is not present in the area.

[0014] The at least one processor may obtain correction coordinates corresponding to a third correction area having a maximum size according to a screen ratio of the first correction area among the areas if there is no second type object in the area and the difference is greater than or equal to the threshold value, and may obtain a plurality of correction coordinates corresponding to the area if there is no second type object in the area and the difference is less than the threshold value.

[0015] A method for controlling an electronic device according to an embodiment of the present disclosure includes the steps of: acquiring a projection area and a non-projection area of ​​an external device from an image acquired through a camera of the electronic device if there is an image provided from an external device in the acquired image; acquiring a correction area by correcting the projection area based on the positions of each of a plurality of vertices corresponding to the acquired projection area; acquiring correction coordinates for the projection area based on the position of at least one object within the correction area; and controlling the acquired correction coordinates to be provided to the external device.

[0016] The step of acquiring the projection area and the non-projection area may include, when a test image provided from the external device is confirmed in the first image acquired through the camera, a step of identifying a projection surface on which the test image is projected in the first image among at least one image different from the first image, and a step of acquiring the projection area and the non-projection area including the plurality of vertices within the first image based on the first image and the second image.

[0017] The step of identifying the second image may include a step of identifying the second image if there is a test image including a plurality of markers for correcting the projection area in the first image, and the step of obtaining the projection area and the non-projection area may include a step of obtaining the projection area and the non-projection area including a plurality of vertices located outside the area corresponding to the plurality of markers based on the first image.

[0018] The step of obtaining the projection area and the non-projection area may include the step of obtaining a plurality of pixel difference values ​​obtained based on a plurality of pixel values ​​included in each of the first image and the second image, and the step of obtaining the projection area and the non-projection area based on a difference image including the obtained plurality of pixel difference values.

[0019] The step of obtaining the projection area and the non-projection area may include a step of obtaining the projection area and the non-projection area based on a plurality of valid pixels corresponding to each of a plurality of pixel difference values ​​that are greater than or equal to a threshold value among a plurality of pixel difference values ​​included in the obtained difference image.

[0020] The step of obtaining the projection area and the non-projection area may include a step of obtaining an area including a first area occupied by the plurality of effective pixels and at least one second area surrounded by the first area among the plurality of pixels included in the difference image as the projection area, and a step of obtaining an area excluding the projection area within the first image as the non-projection area.

[0021] A control method, wherein each of the at least one object is one of a first type object for projecting an image by the external device and a second type object different from the first type object.

[0022] The step of obtaining the above correction coordinates may include a step of identifying a plurality of edges in a first correction area obtained by adjusting the position of at least one of the plurality of vertices, a step of identifying whether a first type object exists based on whether a preset number of edges among the identified plurality of edges are identified, and a step of obtaining the correction coordinates based on whether a second type object exists in the area corresponding to the preset number of edges.

[0023] The step of obtaining the correction coordinates based on whether the second type object exists may include the step of obtaining correction coordinates corresponding to a second correction area having a maximum size according to an aspect ratio of the first correction area included in the remaining area of ​​the area excluding the second type object if the second type object exists in the area, and the step of obtaining the correction coordinates based on whether a difference between an aspect ratio of the first type object, which is a rectangle, and an aspect ratio of the first correction area is greater than or equal to a threshold value if the second type object does not exist in the area.

[0024] In one embodiment of the present disclosure, a non-transitory computer-readable recording medium storing computer instructions that cause an electronic device to perform an operation when executed by a processor of the electronic device, the operation includes the steps of: if an image provided from an external device is present in an image acquired through a camera of the electronic device, acquiring a projection area and a non-projection area of ​​the external device from the acquired image; acquiring a correction area that corrects the projection area based on the positions of each of a plurality of vertices corresponding to the acquired projection area; acquiring correction coordinates for the projection area based on the position of at least one object within the correction area; and controlling the acquired correction coordinates to be provided to the external device.

[0025] FIG. 1 is a diagram illustrating the operation of an electronic device and an external device according to one or more embodiments of the present disclosure.

[0026] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0027] FIG. 3 is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure.

[0028] FIG. 4 is a drawing for explaining a first image and a second image according to one or more embodiments of the present disclosure.

[0029] FIG. 5 is a drawing illustrating a projection area according to one or more embodiments of the present disclosure.

[0030] FIG. 6 is a drawing for explaining a vertex according to one or more embodiments of the present disclosure.

[0031] FIG. 7 is a drawing for explaining correction coordinates according to one or more embodiments of the present disclosure.

[0032] FIG. 8 is a drawing for explaining correction coordinates according to one or more embodiments of the present disclosure.

[0033] FIG. 9 is a drawing for explaining correction coordinates according to one or more embodiments of the present disclosure.

[0034] FIG. 10 is a drawing for explaining correction coordinates according to one or more embodiments of the present disclosure.

[0035] FIG. 11 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0036] FIG. 12 is a flowchart illustrating a method for outputting final coordinates according to one or more embodiments of the present disclosure.

[0037] FIG. 13 is a flowchart illustrating a method for outputting final coordinates based on keystone correction values ​​according to one or more embodiments of the present disclosure.

[0038] FIG. 14 is a flowchart illustrating a method for outputting final coordinates according to one or more embodiments of the present disclosure.

[0039] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0040] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0041] The terminology used in this disclosure has been selected from widely used, current terms, taking into account the functions of the disclosure. However, this may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0042] Terms like "first" and "second" may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0043] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0044] The expression "at least one of A or B" should be understood to mean either "A" or "B" or "A and B".

[0045] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented by one or more processors (not shown), excluding any "modules" or "parts" that need to be implemented in specific hardware.

[0046] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description are omitted for clarity of description of the present disclosure, and similar parts are designated with similar reference numerals throughout the specification.

[0047] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0048] FIG. 1 is a diagram illustrating the operation of an electronic device and an external device according to one or more embodiments of the present disclosure.

[0049] According to FIG. 1, an electronic device (100), an external device (10-1, 10-2) and a projection image (20) are illustrated.

[0050] Each of the electronic device (100) and the external device (10-1, 10-2) may be implemented in a form as shown in FIG. 1, but is not necessarily limited thereto and may be implemented in various forms.

[0051] Electronic devices (100) include smartphones, tablet PCs (personal computers), desktop PCs, laptop PCs, PCs, set-top boxes, OTT service (over-the-top media service) servers, consoles (video game consoles), Blu-ray players, DVD (Digital Video Disc or Digital Versatile Disc) players, home automation control panels, security control panels, media boxes (e.g., Samsung HomeSync). TM , Apple TV TM , or Google TV TM ), game consoles (e.g. Xbox TM , PlayStation TM ) can be implemented as at least one of the following, but is not limited thereto.

[0052] Meanwhile, the electronic device (100) may be a device capable of capturing images of the external environment. For example, the electronic device (100) may capture images of the space surrounding the electronic device (100) through a camera. The camera and its related operations will be described later in FIG. 2.

[0053] Here, the external device (10-1, 10-2) may be implemented as a smartphone, tablet PC (personal computer), desktop PC, laptop PC, PC, set-top box, etc., similar to the electronic device (100). However, the present invention is not limited thereto.

[0054] For example, the external device (10-1, 10-2) may be implemented as a device having a projector function. Here, the projector function may mean a function of projecting an image.

[0055] Here, the external device (10-1, 10-2) may correspond to a mobile projector directly equipped with a moving part, or a mobile projector without a moving part that must be carried and moved by the user. Here, the moving part may correspond to a configuration for moving the electronic device (100).

[0056] For this purpose, the moving part (160) includes a motor, wheels, etc., and can move the electronic device (100) through the movement of the wheels. Meanwhile, when implemented, a cat filter, etc. may be used instead of the wheels, and when the electronic device (100) is implemented as a drone, etc., a propeller may be used instead of the wheels.

[0057] In this case, the external device (10-1, 10-2) can project the projection image (20) onto the wall. When the external device (10-1, 10-2) projects the projection image (20) onto the wall, a projection area can appear on the wall. Here, the projection area can mean a space on a screen or surface on which the projector projects light to display an image.

[0058] External devices (10-1, 10-2) display a screen with a relatively accurate ratio when the projector is positioned in a straight line with the projection surface, but when this is not satisfied due to space conditions, the screen is projected out of the projection surface or in a diamond-shaped shape that is distorted up, down, left, and right.

[0059] In this case, keystone correction may be necessary. Here, keystone correction refers to a function that adjusts the projection so that the corners of the displayed screen (i.e., the projected screen) are forcibly moved so that they are closer to the original rectangular shape.

[0060] For example, keystone correction can be performed through an electronic device (100) other than an external device (10-1, 10-2). Here, a projection area on which a projection image (20) is projected is captured using a camera provided in the electronic device (100), and keystone correction can be performed based on the captured image.

[0061] Here, Projective Transformation can be utilized. Projective Transformation is a transformation that projects an image in 3D space into 2D space. In other words, this transformation method transforms two images viewed from two different viewpoints in 3D space.

[0062] At this time, the matrix expressing the relationship between two different images is called a homography matrix (hereinafter referred to as the H matrix). For example, the size of the H matrix can be 3x3. To obtain the H matrix, four corresponding pairs of coordinates are required. For example, the four corresponding pairs of coordinates can be coordinates in the world coordinate system.

[0063] Below, we will explain the concepts of world coordinate system and other coordinate systems.

[0064] There are four coordinate systems in image geometry: world coordinates, camera coordinates, normal coordinates, and pixel coordinates. The world coordinates and camera coordinates are three-dimensional coordinate systems, while the normal coordinates and pixel coordinates are two-dimensional coordinate systems.

[0065] The World Coordinate System (WCS) is a coordinate system used as a reference when expressing the position of an object. The World Coordinate System is a coordinate system that can be arbitrarily selected and used. For example, one corner of space can be taken as the origin, the direction toward one wall can be set as the X-axis, the direction toward the other wall can be set as the Y-axis, and the direction looking toward the sky can be set as the Z-axis. A point in the World Coordinate System can be expressed as P(X, Y, Z).

[0066] The camera coordinate system is a coordinate system based on the camera. The camera coordinate system takes the camera's focus (the center of the lens) as the origin, the camera's front optical axis direction as the Z-axis, the downward direction of the camera as the Y-axis, and the rightward direction as the X-axis. A point in the camera coordinate system can be expressed as Pc(Xc, Yc, Zc).

[0067] The Pixel Image Coordinate System (PICS) is also known as the Image Coordinate System. The PCS is a coordinate system for images seen with the naked eye, with the upper left corner of the image as the origin, the right direction as the increasing x-axis, and the downward direction as the increasing y-axis. The plane determined by the x-axis and y-axis of the PCS is called the image plane.

[0068] Geometrically, a point P = (X, Y, Z) in 3D space is projected to a point pimg = (x, y) on the image plane through the focal point of the camera (or the focal point of the lens). All 3D points on the ray connecting point P and point pimg are also projected to pimg. Therefore, pimg can be uniquely determined from a 3D point P, but conversely, obtaining P from an image pixel pimg is impossible without additional information. The unit of the pixel coordinate system is the pixel, and it can be written as pimg = (x, y).

[0069] The normalized image coordinate system can be viewed as an image coordinate system that removes the influence of the camera's internal parameters. Furthermore, the normalized coordinate system is a coordinate system that eliminates the units of the coordinate system (normalized) and defines a virtual image plane whose distance from the camera focal point is 1. In other words, it can be an image plane that has been moved by translating the original image plane to a point with a distance of 1 from the camera focal point. The origin of the normalized coordinate system is the center of the image plane (the intersection with the optical axis Zc). A point in the normalized coordinate system can be expressed as p'=(u, v). Even if the same scene is captured from the same location and angle, different images are obtained depending on the camera used or camera settings. Because it is more effective to analyze common geometric characteristics and establish theories in a normalized image plane that removes these factors, the normalized image plane can be used.

[0070] Meanwhile, the shape and material of the projection surface have a significant impact on the distortion and quality of the output image, and it is difficult for the projector to correct keystone distortion without distortion.

[0071] Accordingly, below, various embodiments of performing accurate keystone correction so that an image captured by an electronic device (100) is projected to an optimal area desired by the user will be described.

[0072] The electronic device (100) can acquire an image by photographing a projection surface on which an external device (10-1, 10-2) projects a projection image (20). The acquired image can be captured by a camera, and the acquired image can be stored in the electronic device (10-1, 10-2).

[0073] The electronic device (100) can perform keystone correction using the acquired image. Here, the keystone correction may include a step of acquiring the projection area before adjusting the projection area.

[0074] Here, the electronic device (100) can obtain a projection area using the acquired image.

[0075] For example, the electronic device (100) can perform keystone correction using a single image in the area where the projection image (20) is projected, but this is not limited thereto.

[0076] The electronic device (100) can acquire a projection area using multiple images. Here, the multiple images may correspond to images acquired by photographing a projection surface under the same shooting conditions.

[0077] Meanwhile, the plurality of images may correspond to images obtained by photographing the projection surface under different photographing conditions. For example, the plurality of images may include an image in which a projection image (20) projected by an external device (10-1, 10-2) is photographed, and an image in which the projection image (20) does not exist.

[0078] Here, the image in which the projection image (20) does not exist may mean an image captured in a space where the external device (10-1, 10-2) does not project the projection image (20).

[0079] For example, when an external device (10-1, 10-2) projects a projection image (20) onto a wall and the projection image (20) is captured by an electronic device (100), the external device (10-1, 10-2) can deactivate the projection image (20).

[0080] Disabling the projected image (20) here may mean preventing an external device (10-1, 10-2) such as a projector from projecting an image onto the screen. For example, if the projector is in standby mode, the image may not be projected. Additionally, if the projector is powered off, the image may not be projected.

[0081] However, the present invention is not limited to the above-described examples, and the external device (10-1, 10-2) can receive a control signal to deactivate the projected image (20). The external device (10-1, 10-2) can deactivate the projected image (20) being projected based on the control signal.

[0082] Here, the control signal may correspond to a signal provided by the electronic device (100), but is not limited thereto.

[0083] The specific details of the multiple images that the electronic device (100) can acquire and the projection area acquired based thereon will be described in detail later in FIG. 2.

[0084] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0085] According to FIG. 2, the electronic device (100) may include a memory (110), a communication unit (120), a camera (130), and a processor (140).

[0086] The memory (110) is electrically connected to the processor (140) and can store data required for various embodiments of the present disclosure. For example, the memory (110) may be implemented as an internal memory such as a ROM (Read-Only Memory) (e.g., an EEPROM (electrically erasable programmable read-only memory)), a RAM (Random Access Memory)) included in the processor (140), or may be implemented as a separate memory from the processor (140).

[0087] The memory (110) may be implemented in the form of memory embedded in the electronic device (100) or may be implemented in the form of memory that can be attached or detached from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be attached or detached from the electronic device (100). When implemented as a memory embedded in an electronic device (100), the memory (110) may be at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0088] Meanwhile, in the illustrated example, the electronic device (100) is depicted as being composed of one memory, but when referring to volatile memory and non-volatile memory separately, the electronic device (100) may be referred to as including multiple memories.

[0089] The memory (110) according to one or more embodiments may store at least one instruction, wherein the at least one instruction may correspond to at least one command for the electronic device (100) to obtain correction coordinates.

[0090] Here, the correction coordinates may refer to the corrected coordinates when a distorted projected image is converted into a rectangle through the aforementioned keystone correction. For example, the correction coordinates may refer to the coordinates of the four corners of a trapezoidally projected image that are repositioned into a correct rectangular shape on the screen by correcting them.

[0091] The operation of obtaining the correction coordinates here will be described in detail in the following section.

[0092] Meanwhile, the memory (110) can store a neural network model.

[0093] Here, the neural network model is a computer system or software module that implements human-level intelligence, and has the characteristics of a machine learning and making judgments on its own, and its recognition rate improving with use.

[0094] A neural network model is composed of machine learning (deep learning) technology that uses an algorithm that classifies / learns the characteristics of input data on its own, and element technologies that use machine learning algorithms to simulate the cognitive and judgment functions of the human brain.

[0095] Here, the neural network model may also be referred to as a learning model, an AI (Artificial Intelligence) model, or a deep learning model.

[0096] The element technologies may include at least one of, for example, a linguistic understanding technology that recognizes human language / characters, a visual understanding technology that recognizes objects as if they were human vision, an inference / prediction technology that judges information and logically infers and predicts, and a knowledge representation technology that processes human experience information into knowledge data.

[0097] Here, a neural network model can be trained to perform natural language processing (NLP). NLP can be defined as a technology that enables computers to understand and process human language, and can be implemented using a large language model (LLM). This can also be applied to the various types of neural network models described below.

[0098] For example, the neural network model here is a neural network model that detects and classifies specific objects in images or videos, and can identify the location and type of each object. For example, a neural network model could be a model trained to identify objects within a captured video.

[0099] The image captured here may correspond to an image projected by an external device. In this case, a neural network model can be trained to identify the location and type of objects contained in the image (e.g., a test image) projected by the external device as input data. The objects and test images will be described in detail in the following sections.

[0100] In addition, the memory (110) can store information necessary for the operation of the electronic device (100). For example, the memory (110) can store information for identifying a screen for projecting an image among a plurality of object types.

[0101] For example, information for identifying a screen may include information on the number and size of pre-stored edges. In this case, the electronic device (100) may identify the presence of a screen if a predetermined number of identified edges are detected and the size of the area formed by the edges is greater than the predetermined size.

[0102] Here, an edge can refer to a linear boundary formed by abrupt changes in brightness or color in an image composed of multiple pixels. Multiple edges can represent the shape or outline of an object within an image and distinguish different areas.

[0103] Here, a screen can be a flat object that receives and displays images projected by an external device, such as a projector. For example, a screen can be a surface that reflects light, allowing viewers to view images.

[0104] Here, the operation of the electronic device (100) to identify an object based on information stored in the memory will be described in detail in the following section.

[0105] The communication unit (120) is a component that performs communication with various types of external devices according to various types of communication methods. The communication unit (120) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0106] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID and session key, is first transmitted and received. This information is then used to establish a connection before various other information can be transmitted and received.

[0107] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which lies between visible light and millimeter waves.

[0108] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0109] In addition, the communication unit (120) may include at least one of wired communication modules that perform communication using a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, an optical fiber cable, or a UWB (Ultra Wide-Band) module. This communication unit (120) may also be referred to as a transceiver.

[0110] Meanwhile, the electronic device (100) can receive a signal requesting information or correction coordinates for the projection area from a server or the like through the communication unit (120).

[0111] Alternatively, the electronic device (100) may transmit a signal requesting information about the projection area or correction coordinates, etc., to a server or the like via the communication unit (120). For example, the electronic device (100) may obtain correction coordinates based on the location of an object identified by the electronic device (100), etc., based on the received information about the projection area, etc.

[0112] Meanwhile, the electronic device (100) may receive a control signal to obtain a projection area from a server device or the like, or a control signal to obtain correction coordinates based on the projection area. However, the present invention is not limited thereto.

[0113] Meanwhile, the electronic device (100) may further include an interface such as an HDMI port, DP, RGB, DVI, USB, Thunderbolt, etc. for receiving video / audio signals by being connected to external devices or servers. HMDI, DP, and Thunderbolt are ports that can transmit video and audio signals simultaneously. The electronic device (100) may perform various processing such as demuxing, decoding, and scaling on various signals received from external devices, servers that communicate with external devices, etc. through the communication unit (120) and these various interfaces, to obtain a projection area or output correction coordinates based on the obtained projection area.

[0114] According to one or more embodiments, the electronic device (100) can control the communication unit (120) to provide the acquired correction coordinates to an external device.

[0115] For example, the electronic device (100) can provide the acquired correction coordinates to an external device through the communication unit (120). In addition, the electronic device (100) can provide the acquired correction coordinates to a server device, etc., through the communication unit (120). The server device, etc. can provide the received correction coordinates to the external device.

[0116] The camera (130) can be turned on and perform shooting according to a preset event. The camera (130) can convert the captured image into an electrical signal and generate image data based on the converted signal. For example, the subject can be converted into an electrical image signal through a semiconductor optical element (CCD; Charge Coupled Device), and the converted image signal can be amplified and converted into a digital signal and then signal processed. For example, the camera (130) can be implemented as a general camera, a stereo camera, a depth camera, etc.

[0117] The processor (140) can perform overall control operations of the electronic device (100).

[0118] The processor (140) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an ARM processor, or may be defined by the relevant terminology. In addition, the processor (140) may be implemented as a system on chip (SoC), a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA) having a built-in processing algorithm. In addition, the processor (140) may perform various functions by executing computer executable instructions stored in a memory. Meanwhile, in FIG. 2, only one processor is illustrated in the electronic device (100), but the implementation A poem may contain multiple processors (e.g., CPU + GPU, CPU + DSP).

[0119] According to one or more embodiments, the processor (140) can identify whether an image provided from an external device is included in the image acquired through the camera (130). If the provided image is included in the acquired image, the processor (140) can acquire a projection area and a non-projection area of ​​the external device from the acquired image.

[0120] Here, the image acquired through the camera may correspond to an image captured by the camera. Here, the image provided by an external device may correspond to an image projected from the external device. However, this is not limited to this.

[0121] For example, the processor (140) can analyze features (e.g., markers, patterns) of the image from the acquired image. By recognizing the features, the processor (140) can determine whether a specific image is included.

[0122] That is, the processor (140) can compare the main feature points (e.g., corners, markers, patterns) of the image with the feature points of the captured image to determine whether a specific image is included. Here, the feature point (picture point) may mean a point that has a unique pattern or shape in the image, is distinguished from other parts, and is useful for tracking and comparison.

[0123] Meanwhile, the non-projection area can refer to the space outside the screen where the projector's light cannot reach and therefore the image cannot be projected. For example, the non-projection area could be an area where the projector does not project images, such as the walls or ceiling surrounding the screen.

[0124] According to one embodiment, the processor (140) can identify a second image among at least one image that is different from the first image when there is a test image provided from an external device in the first image captured by the camera (130).

[0125] Here, the test image may be an image used to check and adjust the distortion correction of the projected image. For example, the test image may include multiple (reference) markers (e.g., squares, grids, dots, etc.).

[0126] Here, the test image provided from an external device may refer to an image projected from an external device, but is not limited thereto.

[0127] Here, multiple markers can be used to visually check screen distortion and track each corner or center of the image for accurate correction. For example, a test image containing multiple square markers in a grid pattern can be used to correct keystone distortion in a projector.

[0128] Here, the second image may correspond to an image obtained in a state where the test image is not projected on the projection surface on which the test image is projected in the first image.

[0129] For example, the second image may correspond to an image acquired after the test image projected by the external device has been deactivated. Here, the second image may correspond to an image of the same space in which the first image was captured. In other words, the second image may correspond to an image captured of the same space captured by the camera (130). However, the present invention is not limited thereto.

[0130] Meanwhile, the processor (140) can identify the second image if there is a test image including a plurality of markers for correcting the projection area in the first image.

[0131] For example, the processor (140) can identify whether a shape identical to or similar to the shape of each of the plurality of markers exists in the first image. At this time, the processor (140) can determine whether a test image exists in the first image based on whether a shape similar to the shape of the plurality of markers exists among the feature points extracted from the first image.

[0132] According to one embodiment, the processor (140) can obtain a projection area and a non-projection area including a plurality of vertices in the first image based on the first image and the second image.

[0133] Here, a vertex can correspond to a point representing the location (or boundary) of the projection area. For example, the vertices can correspond to points located at the four corners of the projection area. In a rectangularly projected test image, each vertex could represent the upper-left, upper-right, lower-left, and lower-right positions of the screen.

[0134] Here, a vertex may correspond to a single pixel or an area formed by multiple pixels. For example, a vertex may correspond to a circular (or elliptical, rectangular, etc.) area formed by a fixed number of pixels within the projection area.

[0135] For example, the processor (140) may obtain a projection area and a non-projection area including a plurality of vertices located outside an area corresponding to a plurality of markers based on the first image and the second image. Here, the area corresponding to a plurality of markers may correspond to an area formed by a plurality of markers.

[0136] Specifically, the area formed by multiple markers may refer to a polygonal space formed by each marker as a vertex. For example, if four markers are arranged in a rectangular shape, the area formed by these markers may correspond to a rectangular shape with each marker as a vertex.

[0137] Here, the plurality of vertices may be located outside the area formed by the plurality of markers. That is, the plurality of vertices may be located between the projection area and the area formed by the plurality of markers.

[0138] For example, the plurality of markers may be positioned at predefined locations, for example, corresponding to the four corners of a test image of the image, or in an area inside a threshold distance from the four corners. For example, the plurality of markers may be positioned in an area inside a predefined ratio based on the size of the entire image based on the four corners of the test image.

[0139] Meanwhile, the processor (140) can obtain a plurality of pixel difference values ​​based on a plurality of pixel values ​​included in each of the first image and the second image.

[0140] The processor (140) may compare a plurality of pixel values ​​of the first image with a plurality of pixel values ​​of the second image. For example, the processor (140) may compare the pixel values ​​of each position in the plurality of pixel values ​​with the pixel values ​​of the respective positions in the previous image based on the pixel positions of each of the plurality of pixel values.

[0141] The processor (140) can obtain a pixel difference value for each of a plurality of pixels. Here, the pixel difference value may correspond to a value (absolute value) calculated by calculating the difference in RGB or grayscale values ​​between pixels of the same coordinates.

[0142] The processor (140) can obtain a difference image in which each of a plurality of pixel difference values ​​is a pixel value.

[0143] Here, the difference image is a new image created by calculating the difference between corresponding pixel values ​​of two images, where each pixel can represent the brightness or color difference between the two images.

[0144] For example, if a difference image is created to emphasize parts of two images that have movement or changes, the parts with significant changes can be expressed as bright, and the parts with no differences can be expressed as dark.

[0145] Meanwhile, the processor (140) may align the first image and the second image to obtain a difference image. Here, the alignment may mean compensating for the movement of the electronic device (100) (or the camera (130)) while each of the first image and the second image is being captured, for at least one of the first image and the second image.

[0146] For example, the processor (140) can extract feature points from each of the first image and the second image, and identify the degree of movement (displacement) of an object corresponding to the extracted feature points. The processor (140) can correct the degree of movement for at least one of the first image and the second image.

[0147] However, it is not limited to this, and various correction methods may be used to correct hand shake and movement of electronic devices (100) that may occur between the time the first image is captured and the time the second image is captured.

[0148] Meanwhile, the processor (140) can adjust the time intervals at which each of the first and second images is captured. For example, the processor (140) can minimize the effects of the aforementioned hand shake and movement of the electronic device (100) by reducing the time intervals.

[0149] Specifically, the processor (140) may provide an external device with a control signal to deactivate a test image after a first image of a specific space has been captured. If the external device deactivates the test image (e.g., stops projection) according to the control signal, the processor (140) may acquire an image of the corresponding space captured with the test image removed.

[0150] At this time, a certain amount of time may be required for the control signal to be provided to the external device to deactivate the test image.

[0151] Meanwhile, the processor (140) can obtain the first image or the second image by a user operation input (such as a shooting button) entered through the communication unit (120).

[0152] However, the present invention is not limited thereto, and the processor (140) may acquire a second image that has acquired the same space after the first image has been acquired by a user operation input at a set time interval based on a user setting input, etc. Here, the second image that has acquired the same space may correspond to an image of the same space captured by the camera (130).

[0153] The time interval set here can be adjusted to minimize the specific time required as described above. Accordingly, the processor (140) can acquire the first and second images based on the reduced time interval, thereby minimizing errors that may occur when capturing the first and second images, respectively.

[0154] The processor (140) can obtain a projection area and a non-projection area based on a difference image including a plurality of acquired pixel difference values.

[0155] For example, the processor (140) can distinguish between a projected area and a non-projected area based on brightness (pixel difference value) in the difference image. Here, the processor (140) can identify an area occupied by a plurality of pixels having a specific brightness or higher as a projected area.

[0156] The processor (140) can obtain a projection area and a non-projection area based on a plurality of valid pixels corresponding to each of a plurality of pixel difference values ​​that are greater than or equal to a threshold value among a plurality of pixel difference values ​​included in the obtained difference image.

[0157] Here, a valid pixel can mean a pixel whose pixel value is greater than or equal to a threshold value.

[0158] Here, the threshold value may correspond to a reference value for distinguishing pixels above and below a certain brightness in the difference image. For example, if the threshold value is set to 100, the processor (140) may identify an area with a brightness value of 100 or more as a projection area, and process an area with a brightness value of 100 or less as a non-projection area.

[0159] That is, the processor (140) can identify an area occupied by a plurality of valid pixels as a projection area, and can identify the remaining area as a non-projection area. However, this is not limited to this, and an area other than the area occupied by a plurality of valid pixels can also be identified as being included in the projection area.

[0160] The processor (140) can identify a region including a first region occupied by a plurality of valid pixels and at least one second region surrounded by the first region among the plurality of pixels included in the difference image as a projection region.

[0161] For example, when a test image containing multiple markers is projected, a first image may be captured with some of the markers and at least one object overlapping. If the color and shape of the markers are similar to the color and shape of the overlapping object, the pixel difference value of that portion (the overlapping portion) may be calculated to be lower than that of the rest of the image.

[0162] Accordingly, a low calculated pixel difference value may correspond to a value below the threshold. In this case, the pixels in the overlapping area cannot be identified as valid pixels and cannot be included in the first region.

[0163] However, if the overlapping portion corresponds to a second area surrounded by the first area (i.e., an area occupied by valid pixels), the processor (140) can identify both the first area and the second area as projection areas.

[0164] Accordingly, even if an island-shaped second region exists inside the outer border of the identified first region, all regions inside the outer border of the first region can be identified as projection regions.

[0165] Through this, the processor (140) can perform subsequent corrections (such as keystone correction) using multiple vertices included in the projection area.

[0166] Meanwhile, the processor (140) can acquire an area excluding the projection area from the first image as a non-projection area. Since the non-projection area has been described above, a redundant description will be omitted.

[0167] According to one or more embodiments, the processor (140) may obtain a correction area by correcting the projection area based on the positions of each of a plurality of vertices corresponding to the acquired projection area.

[0168] Here, the multiple vertices corresponding to the projection area may correspond to points located at the four corners of the acquired projection area. Since the vertices have been described above, a redundant description will be omitted.

[0169] Here, the position of the vertex may refer to the above-described coordinates. For example, the processor (140) may perform keystone correction based on a plurality of coordinates representing a plurality of vertex positions.

[0170] According to one or more embodiments, the processor (140) can obtain correction coordinates for the projection area based on the position of at least one object included in the correction area.

[0171] For example, the processor (140) may obtain correction coordinates by performing screen fit. Here, screen fit may refer to a function that automatically adjusts the size of an image on the screen to fit the screen. Here, screen fit may include a function that adjusts the projected image so that it does not go beyond the boundaries of the screen, and functions that adjust the ratio and resolution.

[0172] In one embodiment, each of the at least one object may correspond to one of a first type object and a second type object that is different from the first type object.

[0173] Here, the first type object may correspond to an object for an external device to project an image.

[0174] For example, the first type object may correspond to a planar object recognized by the external device (either the electronic device (100) or the external device) to project an image. That is, the first type object may correspond to a surface used to accurately display the projected image.

[0175] For example, a Type 1 object could be a projection screen, a wall, or any flat surface on which projection is possible.

[0176] A Type 2 object may be an obstacle that interferes with or obstructs the projection of an image by the projector. For example, a Type 2 object may be a person, furniture, or other object blocking the projection path.

[0177] If a second type object exists on the projection area, it may distort or obscure part or all of the projected image.

[0178] For example, a first-type object and a second-type object may be positioned separately without overlapping each other. Furthermore, a second-type object (e.g., a picture frame) may be positioned on a first-type object (e.g., a wall). However, this is not limited to this.

[0179] Meanwhile, the processor (140) can identify a plurality of edges in a first correction area. Here, the first correction area may correspond to an area obtained by adjusting the position of at least one of a plurality of vertices.

[0180] Here, the position can mean the coordinates (of the vertex) as described above. Here, adjusting the position can mean moving the position of the vertex (transforming the coordinates) when the projection area is an irregular area (e.g., an area shaped like a rhombus or trapezoid) through the positions of multiple vertices.

[0181] Moving the position or transforming the coordinates here may mean performing keystone correction with respect to the vertex positions (coordinates).

[0182] Meanwhile, the processor (140) can identify whether a first type object exists based on the presence of a preset number of edges among the identified plurality of edges.

[0183] Here, the preset number may correspond to the standard number of edges for identifying a first-type object. For example, the preset number may be set to the minimum number (three) that can form a two-dimensional area on a plane (a first-type object). However, this is not limited to this, and the preset number may be set to a number exceeding three.

[0184] Meanwhile, the processor (140) can identify whether a first type object exists based on a predetermined size. That is, the processor (140) can identify whether a first type object exists based on the presence of a preset number of edges forming an area of ​​a predetermined size.

[0185] The size determined here may correspond to a size criterion established to identify a first-type object. The determined size may refer to the size of the projection area within which the external device can appropriately project an image. For example, the range of projection areas that can be projected may be determined based on the projection capabilities of the external device.

[0186] Meanwhile, the processor (140) can obtain correction coordinates based on whether a second type object exists in the above area. Here, the area may correspond to a screen area. The screen area may refer to the size of a projection area in which an external device can appropriately project an image.

[0187] Here, the screen area may correspond to an area corresponding to a preset number of edges. Here, the area corresponding to a preset number of edges may refer to an area formed by a preset number of edges. For example, if four edges form a rectangle, that rectangle may correspond to the screen area.

[0188] If there is a second type object in the area, the processor (140) can obtain a second correction area included in the remaining area excluding the second type object. In addition, the processor (140) can obtain correction coordinates corresponding to the second correction area.

[0189] Specifically, the processor (140) can obtain a second correction area having a maximum size according to the screen ratio of the first correction area included in the remaining area excluding the second type object.

[0190] Here, the second correction area may correspond to an area having the same screen ratio as the screen ratio of the first correction area among the areas and having the maximum size.

[0191] For example, if an object such as a protruding shelf exists in the area, the processor (140) can obtain a final area (second correction area) with the same screen ratio as the first correction area and the maximum size in the area excluding the area occupied by the shelf.

[0192] Accordingly, the processor (140) can obtain the second correction area while maintaining the screen ratio of the first correction area and avoiding the second object that interferes with projecting a flat image.

[0193] Meanwhile, the processor (140) may acquire correction coordinates corresponding to the acquired second correction area. Here, the correction coordinates corresponding to the second correction area may correspond to the vertices of the second correction area. However, this is not limited thereto, and the correction coordinates may correspond to the coordinates of a plurality of points located within the second correction area at a predetermined ratio.

[0194] Meanwhile, the processor (140) may obtain correction coordinates based on whether the difference between the screen ratio of the first type object and the screen ratio of the first correction area is greater than or equal to a threshold value if there is no second type object in the area. Here, the first type object may correspond to a rectangular object.

[0195] Here, the aspect ratio may correspond to a value representing the ratio of the width and height of the screen. The difference in aspect ratio may include at least one of the difference in the width and height ratios in each of the two aspect ratios.

[0196] Meanwhile, the threshold value may correspond to a reference value for determining whether the screen ratios are similar. Here, the threshold value may be determined by at least one of the horizontal or vertical lengths of the first type object or the first correction area, but is not limited thereto.

[0197] Specifically, the processor (140) can obtain a third correction area among the areas where there is no second type object in the area and the difference is greater than a threshold value.

[0198] Here, the third compensation area may correspond to an area with the same screen ratio as the first compensation area and the largest size among the areas. For example, the processor (140) may acquire the third compensation area, which is the largest possible area among the areas while maintaining the screen ratio of the first compensation area.

[0199] Thereafter, the processor (140) can obtain correction coordinates for the acquired third correction area. Here, the correction coordinates may mean coordinates of multiple vertices, similar to the second correction area, but are not limited thereto.

[0200] The processor (140) can obtain a plurality of correction coordinates corresponding to the area if there is no second type object in the area and the difference is less than a threshold value.

[0201] Here, the correction coordinates may refer to the coordinates of multiple vertices of the area, similar to the third correction area, but are not limited thereto. For example, if the difference in screen ratio between the first correction area and the first type object is less than a threshold value and thus a slight difference exists, the coordinates corresponding to the area can be acquired as the correction coordinates.

[0202] That is, when the screen ratio of the first correction area obtained through keystone correction and the first type object are almost identical, the coordinates corresponding to the area occupied by the first type object (screen area) can be the correction coordinates.

[0203] Meanwhile, according to one or more embodiments, the processor (140) may control the communication unit (120) to provide the acquired correction coordinates to an external device. The external device may adjust the projection area based on the provided correction coordinates.

[0204] For example, keystone correction can be performed by comparing the correction coordinates with existing reference coordinates (e.g., marker coordinates, vertex coordinates, etc.).

[0205] If an obstacle (type 2 object) included in the photographed space interferes with the marker (e.g., is adjacent or overlapping), the projection area is identified by the image photographing the space and keystone correction is performed, inaccurate correction coordinates may be obtained.

[0206] However, by using multiple images (first image and second image) acquired under two shooting conditions (e.g., whether a test image is projected), the projection area (vertex corresponding to the projection area) can be more accurately identified, and by performing keystone correction and screen fit operations through the vertex, etc., more accurate correction coordinates can be obtained.

[0207] In FIG. 2, the electronic device (100) is illustrated as including only basic components (i.e., memory, communication unit, processor, etc.), but the electronic device (100) may include various components in addition to the above-described components.

[0208] For example, the electronic device (100) may include a display.

[0209] A display is a configuration for displaying the operating status, notification messages, UI screens, etc. of an electronic device (100). The display may be implemented in various forms, such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display may also include a driving circuit, a backlight unit, etc., which may be implemented in forms such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional display), etc. Alternatively, the display may be implemented with only one or a plurality of light-emitting elements.

[0210] The electronic device (100) can change the display status according to various states, such as when the electronic device (100) is turned on, when it is operating normally, when it is low on power or in an error state, so that the user can intuitively understand the status of the electronic device (100).

[0211] However, the display configuration is only one of various embodiments, and the display configuration may be omitted. That is, the electronic device (100) may be a device directly equipped with a display, or a device connected to an external device.

[0212] For example, when the electronic device (100) is implemented as a set-top box, a one-connect box, a projector, etc., the operations of the electronic device (100) described above may be performed in an electronic device that does not include a display.

[0213] Meanwhile, the electronic device (100) may include a microphone. The microphone can receive the user's voice when activated. For example, the microphone may be integrally formed on the upper side, front side, or side of the electronic device (100).

[0214] The microphone may include various configurations such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0215] The microphone can transmit the received user voice to the electronic device (100). Subsequently, the electronic device (100) can input the received user voice into a voice recognition model to perform voice recognition. For example, the electronic device (100) can perform STT (Speech to Text) on the user voice to perform voice recognition on the user voice. The microphone can receive the user voice and transmit the received user voice to the electronic device (100). Subsequently, the electronic device (100) can input the received user voice into a voice recognition model to perform voice recognition. For example, the electronic device (100) can perform STT (Speech to Text) on the user voice to perform voice recognition on the user voice.

[0216] The electronic device (100) may include various other configurations in addition to the configurations described above, and some of the configurations described as being included in the electronic device (100) may be omitted.

[0217] For example, the electronic device (100) can directly obtain an image through a camera (130), but if the electronic device (100) is not equipped with a separate camera (130), the image can be obtained through a separate external device equipped outside the electronic device (100).

[0218] In this case, the processor (140) can receive an image acquired by an external device through a camera equipped in the external device through the communication unit (120). Here, the image may correspond to an image captured by an external device equipped with a projection function, such as a projector. However, the present invention is not limited thereto.

[0219] The electronic device (100) may be implemented without other components (display, microphone, etc.), and may receive information necessary for the operation of the electronic device (100) from an external device, etc.

[0220] FIG. 3 is a diagram illustrating the operation of an electronic device according to one or more embodiments of the present disclosure.

[0221] According to FIG. 3, a test image (310) and a captured image (320) are shown.

[0222] The test image (310) may include multiple markers (311-1, 311-2, 311-3, 311-4).

[0223] In FIG. 3, for convenience, the test image (310) is illustrated as having a rectangular shape and including four markers (first to fourth markers) (311-1, 311-2, 311-3, 311-4), but this is merely an example, and the test image (310) can be implemented in various other shapes and can be implemented as including various other numbers of markers.

[0224] Meanwhile, the captured image (320) may correspond to an image captured by the electronic device (100). The captured image (320) may include an object (322). Here, the object (322) may correspond to a second type object among the first type object and the second type object described above.

[0225] The electronic device (100) can obtain a captured image (320) that captures a test image (310) projected by an external device.

[0226] The captured image (320) may include four markers (311-1, 311-2, 311-3, 311-4) included in the test image (310). At this time, the electronic device (100) can detect the shape (feature) of each of the four markers (311-1, 311-2, 311-3, 311-4) included in the captured image (320) and confirm that the test image (310) is included in the captured image (320).

[0227] If the third marker (311-3) among the four markers (311-1, 311-2, 311-3, 311-4) overlaps with the object (322), the electronic device (100) may not detect the third marker among the four markers (311-1, 311-2, 311-3, 311-4).

[0228] Accordingly, the electronic device (100) may not be able to confirm that the test image (310) exists in the captured image (320).

[0229] In addition, when the electronic device (100) performs keystone correction and screen fit based on the position coordinates of each of the four markers (311-1, 311-2, 311-3, 311-4), if the third marker (311-3) is not recognized, there may be an error in the correction coordinates acquired by the electronic device (100).

[0230] Accordingly, the electronic device (100) may obtain the projection area and the non-projection area by using the captured image (320) that captures the scene projecting the test image (310). However, in this method, the projection area and the non-projection area may not be clearly distinguished due to the color, shape, etc. of the object (322) and the background object.

[0231] The electronic device (100) can obtain the projection area and the non-projection area by additionally using an image (320) captured in a state where the test image (310) is projected, as well as an image captured in a state where the test image (310) is inactive.

[0232] FIG. 4 is a drawing for explaining a first image and a second image according to one or more embodiments of the present disclosure.

[0233] According to FIG. 4, a first image (411), a second image (412), a difference image (420), a first binary image (430), and a second binary image (440) are illustrated.

[0234] Here, the first image (411) may correspond to an image captured while the test image is projected, and the second image (412) may correspond to an image captured while the test image is not projected.

[0235] The difference image (420) may correspond to an image including pixel difference values ​​between a plurality of pixel values ​​included in each of the first image (411) and the second image (412). Each of the plurality of pixels included in the difference image (420) may correspond to a pixel difference value, and the pixel difference value may be displayed in grayscale.

[0236] However, it is not limited thereto, and the plurality of pixel difference values ​​may each include R, G, and B difference values, and the difference image (420) may be implemented as a color image based on the R, G, and B difference values.

[0237] The first binary image (430) may correspond to an image in which each of a plurality of pixel values ​​corresponds to one of two brightness levels. Here, a binary image may correspond to an image in which each pixel value has only one of these two values ​​(mainly 0 and 1).

[0238] For example, it may correspond to an image in which each of a plurality of pixel values ​​(pixel difference values) included in each difference image (420) is mapped to one of two brightness levels (black and white) based on a threshold value.

[0239] The first binary image (430) may include a first region (431) and a second region (432). The first region and the second region have been described in FIG. 2, so a redundant description thereof will be omitted.

[0240] Here, the second region (432) may correspond to a portion where the third marker and the object overlap in the first image (411). The electronic device (100) may obtain a difference image (420) through the first image (411) and the second image (412), and then detect the portion where the marker and the object overlap through the first binary image (430) obtained by analyzing the difference image.

[0241] At this time, if the electronic device (100) confirms the overlapping portion of the marker and the object through the second area (432), the electronic device (100) can use the vertices (441-1 to 411-4) of the projection area instead of using the marker in the subsequent keystone correction step.

[0242] The electronic device (100) can obtain a projection area that includes not only the first area (431) but also the second area (432).

[0243] For example, the electronic device (100) can obtain a second binary image (440) by converting all of the plurality of pixel values ​​with brightness 0 included in the second region (432) to match the pixel values ​​(e.g., 1) of the first region (431).

[0244] Here, the electronic device (100) can obtain a projection area including all areas within the outer border of the first area (431) of the second binary image (440). The electronic device (100) can identify the remaining areas excluding the projection area as non-projection areas.

[0245] The second binary image (440) may include multiple vertices (441-1 to 441-4). In FIG. 4, four vertices (441-1 to 441-4) are shown, but this is only an example, and a different number of vertices may be included depending on the shape of the projection area.

[0246] Specifically, the electronic device (100) can extract a plurality of vertices (441-1 to 441-4) using an edge detection algorithm (e.g., Canny algorithm). Thereafter, the electronic device (100) can identify a plurality of vertices corresponding to a plurality of edges by applying a corner detection algorithm (e.g., Harris corner detection algorithm).

[0247] Here, the Canny algorithm may correspond to an algorithm for detecting edges in an image. When the Canny algorithm is applied, the electronic device (100) can detect edges by applying a Gaussian filter to control noise, distinguishing between strong and weak edges, and ultimately leaving only sharp edges.

[0248] Here, the Harris Corner Detection algorithm may correspond to an algorithm for finding corners (vertices) in an image. When the Harris Corner Detection algorithm is applied, the electronic device (100) can identify points corresponding to corners by analyzing the rate of change of surrounding pixels.

[0249] By utilizing both the first image (411) and the second image (412), the electronic device (100) can more accurately distinguish between the projection area and the non-projection area even when it is difficult to recognize the marker of the pattern image due to an object (obstacle, etc.) on the projection surface.

[0250] Additionally, the electronic device (100) can recognize that the marker has not been accurately recognized by more accurately recognizing the overlapping portion of the marker and the object. Through this, the electronic device (100) can then perform screen fit or keystone correction using multiple vertices (441-1 to 441-2) of the projection area instead of using the marker.

[0251] FIG. 5 is a drawing illustrating a projection area according to one or more embodiments of the present disclosure.

[0252] According to FIG. 5, a first captured image (510) and a second captured image (520) are shown.

[0253] Here, the first captured image (510) may correspond to an image captured while the test image is projected when multiple markers are included in the test image. The electronic device (100) can identify the projection area (511) through the first captured image (510) and can identify the screen area (512) included in the projection area (511).

[0254] Here, the second shooting image (520) may correspond to an image shot while the test image is projected when the test image does not include multiple markers.

[0255] Meanwhile, as described in FIG. 4, the electronic device (100) can identify the projection area (511) using the difference image obtained through the first image and the second image.

[0256] And the electronic device (100) can also identify a screen area (512) included in the projection area (511). Since the operation of identifying the screen area has been described above, a duplicate description will be omitted.

[0257] However, even if the test image does not include multiple markers, the electronic device (100) can identify the same projection area (521) and screen area (522) as when the first captured image (510) is used.

[0258] When the electronic device (100) identifies a projection area or performs keystone correction, etc. through a plurality of markers, such as when some of the markers overlap with an object, an error may occur in the correction coordinates. Instead of using a plurality of markers, the electronic device (100) can identify a projection area through a difference image and perform keystone correction, etc. through the vertices (A, B, C, D) of the projection area.

[0259] Accordingly, even if a marker is included in the test image, the electronic device (100) can identify the projection area without being affected by the marker. In other words, even if a marker is included in the test image, the electronic device (100) can perform the same function as when the second captured image (520) is used.

[0260] Additionally, the electronic device (100) can perform other operations (screen fit, etc.) such as performing keystone correction or recognizing the screen area (512) through the vertex obtained from the projection area (511).

[0261] For convenience of explanation, the following description will illustrate a test image that does not include multiple markers. However, the test image depicted in the following drawings is not necessarily limited to an image that does not include multiple markers.

[0262] FIG. 6 is a drawing for explaining a vertex according to one or more embodiments of the present disclosure.

[0263] According to FIG. 6, an electronic device (100), an external device (10), a captured image (610), a vertex (611), a projection area (612), a screen area (613), and a correction area (612') are illustrated.

[0264] The electronic device (100) can obtain a captured image (610) in which an external device (10) projects an image.

[0265] The electronic device (100) can identify a projection area (612) using an image captured with the projection image deactivated along with the captured image (610). The operation of the electronic device (100) to identify the projection area has been described above, so a duplicate description will be omitted.

[0266] The electronic device (100) can detect a plurality of vertices (611) from the acquired projection area (612). The electronic device (100) can perform keystone correction using the coordinates of the plurality of vertices (611) as reference coordinates. The electronic device (100) can acquire a correction area (612') through keystone correction.

[0267] Here, keystone correction may mean an operation in which the electronic device (100) obtains keystone correction coordinates based on the vertex coordinates. That is, the keystone correction here may not mean controlling the electronic device (100) to directly adjust the projection area of ​​the external device (10), but may mean an operation in which the external device (10) obtains correction coordinates necessary for adjusting the projection area.

[0268] The electronic device (100) can identify a screen area (613) included in a correction area (612') and perform screen fitting. The electronic device (100) can identify the screen area (613) based on the coordinates of a plurality of vertices and perform screen fitting.

[0269] That is, the electronic device (100) can perform keystone correction using multiple vertices obtained from the projection area (612) instead of using multiple markers, and can more effectively identify the screen area, etc. included in the correction area (612').

[0270] FIG. 7 is a drawing for explaining correction coordinates according to one or more embodiments of the present disclosure.

[0271] According to FIG. 7, it may include an image (710), a projection area (711), a screen area (712), an object (713), and a screen fit area (714).

[0272] Here, the image (710) may correspond to an image on which the above-described keystone correction has been performed, but is not limited thereto.

[0273] Here, the electronic device (100) can identify the projection area (711) included in the image (710).

[0274] Thereafter, the electronic device (100) can identify a screen area (712) among the projection areas (711). And the electronic device (100) can identify an object (713) included in the screen area (712).

[0275] Here, the object (713) may correspond to the second type object described above. However, it is not limited thereto.

[0276] The electronic device (100) can obtain a screen fit area (714) of the largest size among the remaining areas excluding the object (713) among the screen areas (712). The screen fit area (714) may correspond to an image obtained by the electronic device (100) by performing screen fit.

[0277] For example, the screen fit area (714) may correspond to the maximum area that matches the screen ratio of the projection area (712) without overlapping with the object (713), but is not limited thereto.

[0278] Accordingly, the electronic device (100) can optimize the projection area by adjusting the projection area to the maximum size that does not overlap with the object without changing the screen ratio of the original image (projected image).

[0279] The electronic device (100) can obtain a plurality of correction coordinates corresponding to the optimized image as above.

[0280] FIG. 8 is a drawing for explaining correction coordinates according to one or more embodiments of the present disclosure.

[0281] According to FIG. 8, an image (810), a screen area (811), an object (812), a first screen fit area (813), and a second screen fit area (814) are illustrated.

[0282] Here, the image (810) may correspond to an image on which keystone correction has been performed, but is not limited thereto.

[0283] The electronic device (100) can perform screen fit on the projection area included in the image (810). As illustrated in FIG. 7, the electronic device (100) can obtain the maximum area with the same screen ratio as the projection area, among the areas excluding the object (812) in the screen area (811), as the first screen fit area (813).

[0284] For example, the first screen fit area (813) may correspond to an area adjacent to the top of the screen area (811). In this case, the electronic device (100) may adjust the position of the first screen fit area (813) to the center to obtain the second screen fit area (814).

[0285] Accordingly, the electronic device (100) can perform screen fitting to the maximum area that does not overlap with an object, and can also adjust the screen fit area if there is a free area other than the first screen fit area (813) in the screen area (811). Through this, the final projection area can be obtained as an area that can improve the viewing experience of a user of an external device.

[0286] The electronic device (100) can obtain a plurality of correction coordinates corresponding to the optimized image as above.

[0287] FIG. 9 is a drawing for explaining correction coordinates according to one or more embodiments of the present disclosure.

[0288] According to FIG. 9, a projection area (910), an object (911), a first screen fit area (912), and a second screen fit area (913) are illustrated.

[0289] Here, the projection area (910) may correspond to an area where an external device projects an image (test image). However, this is not limited thereto, and the projection area (910) may correspond to an area (correction area) obtained by the electronic device (100) performing keystone correction on an area where an external device projects an image.

[0290] According to FIG. 9, unlike those illustrated in FIGS. 7 and 8, the projection area (910) may not include a screen area. Even in this case, the electronic device (100) may obtain an area with the same ratio as the projection area (910) in the remaining area excluding the object (911) among the projection areas (910) as the first screen fit area (912).

[0291] Here, the first screen fit area (912) may correspond to the maximum size area with the same screen ratio as the projection area (910) in the remaining area, but is not limited thereto.

[0292] Meanwhile, the first screen fit area (912) may correspond to an area adjacent to the top of the projection area (910). In this case, the electronic device (100) may adjust the position of the first screen fit area (912) to the center to obtain the second screen fit area (913).

[0293] That is, even if the projection area (910) does not include the screen area, the electronic device (100) can perform screen fitting to the maximum area that does not overlap with an object, and if there is a free area excluding the first screen fit area (912) among the projection areas (910), the electronic device (100) can adjust the screen fit area. Through this, the final projection area can be obtained as an area that can improve the viewing experience of the user of the external device.

[0294] The electronic device (100) can obtain a plurality of correction coordinates corresponding to the optimized image as above.

[0295] FIG. 10 is a drawing for explaining correction coordinates according to one or more embodiments of the present disclosure.

[0296] According to FIG. 10, an original image (1010), a projection area (1020), a screen area (1021), a first screen fit area (1022), and a second screen fit area (1023) are illustrated.

[0297] Here, the original image (1010) may refer to an image projected by an external device. For example, the original image (1010) may correspond to an image with a screen ratio of 16:9.

[0298] The electronic device (100) can identify a projection area (1020) and a screen area (1021) included in the projection area (1020). Here, the screen ratio of the screen area (1021) may not match the screen ratio of the original image (1010).

[0299] The electronic device (100) can identify whether an object (an object other than a first type object such as a screen) exists in the screen area (1021). If it is identified that no object exists in the screen area (1021), the electronic device (100) can perform screen fitting based on the screen ratio of the screen area (1021).

[0300] The electronic device (100) compares the screen ratio of the screen area (1021) with the ratio of the original image (1010) (or the ratio of the projection area (1020), hereinafter the same), and if the difference in the screen ratio is less than a threshold value, the coordinates corresponding to the screen area (1020) can be obtained as correction coordinates.

[0301] On the other hand, as illustrated in FIG. 10, the electronic device (100) can perform screen fitting while maintaining the ratio of the original image (1010) (e.g., 16:9) when the difference between the screen ratio of the screen area (1021) and the ratio of the original image (1010) is greater than a threshold value.

[0302] For example, the electronic device (100) can obtain a first screen fit area (1022) by performing screen fitting so that the size within the screen area (1021) is maximized while having the same aspect ratio as the original image (1010).

[0303] Meanwhile, as illustrated in FIG. 10, if the center of the first screen fit area (1022) is not located at the center of the screen area (1022), the electronic device (100) can adjust the position of the first screen fit area (1022) to the center to obtain the second screen fit area (1023).

[0304] That is, even if the projection area (1020) does not include an object, the electronic device (100) can perform screen fitting to the maximum area within the screen area (1021), and if there is a free area excluding the first screen fit area (1022) among the screen areas (1021), the screen fit area can be adjusted. Through this, the final projection area can be obtained as an area that can improve the viewing experience of the user of the external device.

[0305] The electronic device (100) can obtain a plurality of correction coordinates corresponding to the optimized image as above.

[0306] FIG. 11 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0307] The electronic device (100) can obtain a projection area and a non-projection area (S1110).

[0308] According to one or more embodiments, the electronic device (100) can obtain a projection area and a non-projection area of ​​the external device from the captured image if there is an image provided from an external device in the captured image.

[0309] According to one or more embodiments, the electronic device (100) can obtain a correction area by correcting the projection area based on the positions of each of a plurality of vertices corresponding to the acquired projection area.

[0310] Next, the electronic device (100) can obtain correction coordinates based on the object position (S1120).

[0311] According to one or more embodiments, the electronic device (100) can obtain correction coordinates for the projection area based on the position of at least one object within the correction area.

[0312] Next, the electronic device (100) can provide the correction coordinates to an external device (S1130).

[0313] Hereinafter, a more specific operation of the electronic device (100) identifying the projection area and obtaining correction coordinates will be described.

[0314] FIG. 12 is a flowchart illustrating a method for outputting final coordinates according to one or more embodiments of the present disclosure.

[0315] An external device, such as a projector, can project a pattern (such as a pattern image or test image) at a maximum (default) size (S1210). The maximum size here may correspond to the maximum size that the external device can project. For example, the maximum size may be determined by the projection specifications of the external device. Furthermore, the maximum size here may correspond to the size set as the default size by the external device.

[0316] Next, the electronic device (100) can acquire an image of a pattern projected by an external device based on a user manipulation input (S1220). Here, the user manipulation input may correspond to a manipulation input for capturing a pattern. The image acquired here may correspond to the first image described above.

[0317] And the electronic device (100) can acquire an image of a space captured after pattern projection is deactivated by the projector based on a user operation input (S1230). Here, the user operation input may correspond to an operation input for capturing a space in which pattern projection is deactivated. The image acquired here may correspond to the second image described above.

[0318] Here, "general environment shooting" can mean shooting the area where the pattern was projected, with the projector's pattern projection disabled. Since we've already explained why pattern projection is disabled, we'll omit a repeat explanation.

[0319] The electronic device (100) can identify the projection area based on an image acquired by photographing the projected pattern and an image acquired through general environmental photography. Since the operation of identifying the projection area by the electronic device (100) has been described above, a redundant description will be omitted.

[0320] The electronic device (100) can identify whether a pattern (or marker) is detected in an image in which a pattern image is captured (S1240). If a pattern is detected in the image, the electronic device (100) can identify whether a keystone correction value is calculated (S1250).

[0321] Calculating a keystone correction value here can mean that valid correction coordinates can be derived from the extracted feature points in the captured image, which is a distorted image projected by the projector. The fact that valid correction coordinates are derived here can mean that the extracted correction coordinates are within the normal range.

[0322] Additionally, the fact that valid correction coordinates are produced can mean that these coordinates can be used to accurately correct distortions in the image into a rectangular shape.

[0323] For example, if markers are properly recognized and their positions are within the normal range within the projection screen, the keystone correction value can be said to have been successfully calculated. Conversely, if the coordinates are too far apart or distorted to be meaningful, the correction value can be said to have not been calculated.

[0324] Here, coordinates that are too far apart may mean that the coordinates are located outside the normal range described above. The normal range is used as a criterion for determining whether valid correction coordinates have been calculated, and can be set by factors such as the maximum size of the projector's projection image and user settings.

[0325] If the keystone correction value is identified as calculated, the electronic device (100) can map the marker or corner (vertex) location to a rectangle (S1260). Here, mapping may mean adjusting the coordinates corresponding to the existing marker or corner location according to the keystone correction value. However, this is not limited thereto.

[0326] Here, if the electronic device (100) detects a pattern, the position of the marker can be mapped to a rectangle, and if the electronic device (100) detects a projection screen instead of a pattern, the position of the corner can be mapped to a rectangle.

[0327] This can also be applied to the operation of the electronic device (100) to map the positions of markers or corners into a rectangle when no keystone correction value is calculated.

[0328] Next, the electronic device (100) can output final coordinates based on at least one of the screen boundary and the object screen boundary, and the object and screen boundary ratio within the mapping image (S1270).

[0329] Here, the screen boundary may correspond to a plurality of edges forming the screen area described above. Here, the object may correspond to the second type object described above (i.e., an object other than the first type object, such as an obstacle, such as a screen). Here, the screen boundary ratio may refer to the aspect ratio of the screen area described above.

[0330] The operation of the electronic device (100) to output final coordinates by various elements such as the screen boundary and objects will be described later in FIG. 13.

[0331] Meanwhile, if a pattern is not detected in the above-described image (an image in which a pattern is captured), the electronic device (100) can identify whether a projection screen (projection area) is detected in the image (S1241).

[0332] If the projection screen is not detected, the electronic device (100) may recognize that the final coordinate calculation has failed and terminate the process for calculating the final coordinates (S1280). For example, if the electronic device (100) cannot recognize the projection screen, keystone correction and screen fit cannot be performed, and thus the process for calculating the final coordinates may be interrupted and terminated.

[0333] If a projection screen is detected in the above-described image, the electronic device (100) can identify whether a keystone correction value is calculated (S1250).

[0334] For example, if a test image includes multiple markers and some of the markers overlap with a specific object, the electronic device (100) may not be able to identify all of the markers included in the test image. In this case, the electronic device (100) may identify the projection area and the non-projection area instead of the multiple markers and perform subsequent processes.

[0335] Meanwhile, if the electronic device (100) determines that a keystone correction value is not calculated, the position of the marker or corner (vertex) can be mapped into a rectangle (S1551). For example, the electronic device (100) can extract the corner coordinates of the photographed pattern or projected area and correct them into a rectangle.

[0336] To this end, the electronic device (100) can use a corner detection algorithm (e.g., Hough transform, Canny edge detection, etc.) to find the four corners of the projection area, and apply a homography transform based on the coordinates to correct the distorted image into a rectangle. Here, the Hough transform may correspond to an algorithm that mathematically recognizes and detects specific shapes, such as straight lines and circles, in an image.

[0337] Next, the electronic device (100) can output final coordinates based on the screen boundary and objects within the mapping image (S1252). The operation of the electronic device (100) to output final coordinates based on various factors such as the screen boundary when the keystone correction value is not calculated will be described later in FIG. 14.

[0338] FIG. 13 is a flowchart illustrating a method for outputting final coordinates based on keystone correction values ​​according to one or more embodiments of the present disclosure.

[0339] According to FIG. 13, the operation of obtaining final coordinates (corrected coordinates) after the electronic device (100) maps the position of a marker or corner into a rectangle is shown step by step.

[0340] The electronic device (100) can map the positions of markers or corners into a rectangle (S1310) and identify whether a screen boundary is detected within the mapping image (S1320). Here, the screen boundary may refer to a plurality of edges forming a screen area as described above. Here, the mapping image may refer to an image whose projection area is adjusted based on a keystone correction value.

[0341] If the screen boundary is not detected within the mapping image, the electronic device (100) can identify whether an object exists within the mapping image (S1321).

[0342] If an object is identified as present in the mapping image, the electronic device (100) can calculate the coordinates of the maximum area, such as the projection image ratio, within the projection image boundary, in an object-free area (S1322). Here, the maximum area may refer to the aforementioned screen fit image. Since this has been explained in detail above, a redundant explanation will be omitted.

[0343] If an object is identified as not existing in the mapping image, the electronic device (100) can calculate the coordinates of the boundary of the projection area (S1323). Here, the boundary may mean multiple vertices or multiple edges included in the projection area.

[0344] Meanwhile, when a screen boundary is detected within a mapping image, the electronic device (100) can identify whether an object exists within the screen boundary (S1330). Here, the area occupied by the mapping image may correspond to the first correction area described above. However, this is not limited thereto.

[0345] If an object is identified as existing within the screen boundary, the coordinates of the maximum area, such as the projection image ratio, within the detected boundary and in the area without the object can be calculated (S1331). Here, the maximum area may refer to the aforementioned screen fit area (or second correction area).

[0346] If it is determined that no object exists within the screen boundary, the electronic device (100) can determine whether the screen boundary ratio and the ratio of the projected image match (S1340). Here, matching ratios (screen ratios) may mean that the difference in the ratios of each of the two images is less than the threshold value for the aforementioned screen ratio.

[0347] If it is identified that the screen boundary ratio and the projection image ratio match, the electronic device (100) can calculate the coordinates of the detected boundary (S1341).

[0348] The detected boundary here may refer to multiple edges forming a screen area, and the coordinates of the detected boundary may correspond to the coordinates of the vertices of the screen area formed by the detected boundary, but are not limited thereto.

[0349] If it is determined that the screen boundary ratio and the projection image ratio do not match, the electronic device (100) can calculate the coordinates of the maximum area equal to the projection image ratio within the detected boundary (S1342). Here, the maximum area may refer to the third area described above, but is not limited thereto.

[0350] The electronic device (100) can output coordinates (corrected coordinates) calculated through the various paths described above.

[0351] As described above, the electronic device (100) can obtain final coordinates based on various factors such as the detected boundary, the ratio of the projected image, etc. when the keystone correction value is calculated. However, even when the keystone correction value is not calculated, the electronic device (100) can obtain final coordinates based on various factors such as the screen boundary and the position of an object within the boundary.

[0352] FIG. 14 is a flowchart illustrating a method for outputting final coordinates according to one or more embodiments of the present disclosure.

[0353] According to FIG. 14, after a user takes a picture of a pattern with an electronic device (100), the electronic device (100) shows a step-by-step operation of outputting the final coordinates when a keystone correction value is not calculated.

[0354] The electronic device (100) can acquire two images (first image and second image) by pattern shooting and general environment shooting, respectively, based on user operation input (S1410). In addition, the electronic device (100) can map the positions of markers or corners into a rectangle (S1420).

[0355] Since the electronic device (100) has already described the operation of adjusting the projection area by mapping the positions of markers or corners when the keystone correction value is not calculated, a redundant description will be omitted.

[0356] Thereafter, the electronic device (100) can identify whether a screen boundary is detected in the projection area (S1430). If the screen boundary is not detected, the electronic device (100) can recognize that the final coordinate calculation has failed if the projection screen is not detected, and can terminate the process for calculating the final coordinates (S1480).

[0357] For example, if the electronic device (100) cannot recognize the projection screen and cannot detect the screen boundary, keystone correction and screen fit cannot be performed, and thus the process for calculating the final coordinates can be terminated.

[0358] If a screen boundary is detected, the electronic device (100) can identify whether an object exists within the boundary (S1440). If it is determined that an object does not exist within the screen boundary, the electronic device (100) can calculate the coordinates of the detected boundary (S1450).

[0359] If an object is identified as existing within the screen boundary, the electronic device (100) can calculate the coordinates of the maximum area such as the projection image ratio within the detected boundary (S1460).

[0360] For example, if a keystone correction value is not calculated, the electronic device (100) can identify the screen boundary and objects, etc., using the pattern (or vertex) coordinates of the projection area. Unlike when a keystone correction value is calculated, the electronic device (100) can calculate the coordinates of the maximum area without comparing the screen boundary ratio and the projection image ratio.

[0361] The electronic device (100) can output coordinates calculated through the various paths described above (S1470).

[0362] When an external device such as a projector projects a test image, if a marker included in the projected test image is overlapped or distorted by some object, the electronic device (100) may obtain somewhat inaccurate correction coordinates.

[0363] Instead of identifying a projection area or performing keystone correction using a marker (or pattern) included in a test image, the electronic device (100) can first identify a projection area on which a test image is projected and perform keystone correction and screen fit, etc. based on this.

[0364] The electronic device (100) can identify a more accurate projection area (vertex coordinates of the projection area, etc.) by comparing images (first image and second image) captured under different conditions to accurately identify the criteria for keystone correction and screen fit.

[0365] Meanwhile, in Figures 11 to 14, the order of all steps is mapped for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0366] Meanwhile, the methods according to at least some of the various embodiments of the present disclosure described above can be implemented in the form of applications that can be installed on existing electronic devices.

[0367] Additionally, the methods according to at least some of the various embodiments of the present disclosure described above can be implemented with only a software upgrade or a hardware upgrade for an existing electronic device.

[0368] Additionally, the methods according to at least some of the various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic devices.

[0369] Meanwhile, according to one embodiment of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The machine is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the 'non-transitory storage medium' only means that it is a tangible device and does not include a signal (e.g., electromagnetic wave), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is temporarily stored in the storage medium. For example, a 'non-transitory storage medium' may include a buffer in which data is temporarily stored. According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones).In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0370] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored in the storage medium and operate according to the called instructions, and may include an electronic device (e.g., an electronic device (100-1)) according to the disclosed embodiments.

[0371] When the above-described instruction is executed by the processor, the processor may perform the function corresponding to the instruction directly or by utilizing other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter.

[0372] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, Memory that stores at least one instruction; Department of Communications; camera; and comprising at least one processor; The at least one processor, by executing the at least one instruction, If there is an image provided from an external device in the image acquired through the above camera, the projection area and non-projection area of ​​the external device are acquired from the acquired image, Obtain a correction area by correcting the projection area based on the positions of each of the plurality of vertices corresponding to the acquired projection area, Obtaining correction coordinates for the projection area based on the position of at least one object included in the correction area, An electronic device that controls the communication unit so that the acquired correction coordinates are provided to the external device.

2. In paragraph 1, At least one processor, When a test image provided from the external device is confirmed in the first image acquired through the camera, the projection surface on which the test image is projected in the first image among at least one image different from the first image is identified as a second image acquired in a state where the test image is not projected. An electronic device that obtains a projection area and a non-projection area including the plurality of vertices within the first image based on the first image and the second image.

3. In paragraph 2, At least one processor, If there is a test image including a plurality of markers for correcting the projection area in the first image, the second image is identified, An electronic device that obtains a projection area and a non-projection area including a plurality of vertices located outside an area corresponding to the plurality of markers based on the first image and the first image.

4. In paragraph 3, At least one processor, Obtaining a plurality of pixel difference values ​​based on a plurality of pixel values ​​included in each of the first image and the second image, An electronic device that obtains the projection area and the non-projection area based on a difference image including the obtained plurality of pixel difference values.

5. In paragraph 4, At least one processor, An electronic device that acquires the projection area and the non-projection area based on a plurality of valid pixels corresponding to each of a plurality of pixel difference values ​​greater than or equal to a threshold value among a plurality of pixel difference values ​​included in the acquired difference image.

6. In paragraph 5, At least one processor, Acquire an area including a first area occupied by the plurality of valid pixels and at least one second area surrounded by the first area among the plurality of pixels included in the difference image as the projection area, An electronic device that acquires an area excluding the projection area within the first image as the non-projection area.

7. In paragraph 1, An electronic device, wherein each of the at least one object is one of a first type object for projecting an image by the external device and a second type object different from the first type object.

8. In paragraph 7, At least one processor, Identifying a plurality of edges in a first correction area obtained by adjusting the position of at least one of the plurality of vertices, Identifying whether a first type object exists based on whether a preset number of edges among the identified plurality of edges are identified, An electronic device that obtains the correction coordinates based on the presence or absence of the second type object in an area corresponding to the above-described number of edges.

9. In paragraph 8, At least one processor, If there is a second type object in the above area, the correction coordinates corresponding to the second correction area having the maximum size according to the screen ratio of the first correction area included in the remaining area excluding the second type object among the above areas are obtained, An electronic device that obtains the correction coordinates based on whether the difference between the screen ratio of the first type object, which is a rectangle, and the screen ratio of the first correction area is greater than or equal to a threshold value when there is no second type object in the area.

10. In paragraph 9, At least one processor, If there is no second type object in the above area and the difference is greater than or equal to the threshold value, a correction coordinate corresponding to a third correction area having a maximum size according to the screen ratio of the first correction area among the above areas is obtained, An electronic device that obtains a plurality of correction coordinates corresponding to the area when the second type object is not present in the area and the difference is less than the threshold value.

11. In a method for controlling an electronic device, If there is an image provided from an external device in an image acquired through a camera of the electronic device, a step of acquiring a projection area and a non-projection area of ​​the external device from the acquired image; A step of obtaining a correction area by correcting the projection area based on the positions of each of a plurality of vertices corresponding to the acquired projection area; A step of obtaining correction coordinates for the projection area based on the position of at least one object within the correction area; and A control method, comprising: a step of controlling the acquired correction coordinates to be provided to the external device; 12. In paragraph 11, The step of obtaining the above projection area and non-projection area is: When a test image provided from the external device is confirmed in the first image acquired through the camera, a step of identifying a projection surface on which the test image is projected in the first image among at least one image different from the first image, and a second image acquired in a state where the test image is not projected; and A control method comprising: a step of obtaining a projection area and a non-projection area including the plurality of vertices within the first image based on the first image and the second image.

13. In paragraph 12, The step of identifying the second image is: A step of identifying the second image if there is a test image including a plurality of markers for correcting the projection area in the first image; The step of obtaining the above projection area and non-projection area is: A control method comprising: a step of obtaining a projection area and a non-projection area including a plurality of vertices located outside an area corresponding to the plurality of markers based on the first image and the first image.

14. In paragraph 13, The step of obtaining the above projection area and non-projection area is: A step of obtaining a plurality of pixel difference values ​​based on a plurality of pixel values ​​included in each of the first image and the second image; and A control method comprising: a step of obtaining the projection area and the non-projection area based on a difference image including the obtained plurality of pixel difference values; 15. A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation comprising: A step of obtaining a projection area and a non-projection area of ​​the external device from an image obtained through a camera of the electronic device, if there is an image provided from an external device; A step of obtaining a correction area by correcting the projection area based on the positions of each of a plurality of vertices corresponding to the acquired projection area; A step of obtaining correction coordinates for the projection area based on the position of at least one object within the correction area; and A non-transitory computer-readable recording medium, comprising: a step of controlling the acquired correction coordinates to be provided to the external device;

Images