Electronic device and control method thereof

Abstract

An electronic device includes an image projection unit, a memory configured to store at least one instruction, and one or more processors connected to the image projection unit and the memory to control the electronic device, and the one or more processors are configured to execute, by executing the at least one instruction, an image of a projection surface based on capturing the image of the projection surface. And a control unit that recognizes feature information of the projection surface, recognizes first format information corresponding to a plurality of images provided in a multi-view mode, obtains layout information of a plurality of screens corresponding to the plurality of images based on the feature information of the projection surface and the first format information, and controls the projection unit such that the plurality of images are provided based on the layout information.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that projects an image and a control method thereof. Background Art

[0002] With the development of electronic technology, various types of electronic devices are being developed and popularized. In particular, electronic devices used in various places such as homes, offices, and public spaces have continued to develop in recent years.

[0003] In particular, beam projectors are used in various places such as offices, theaters, homes, and shops, and the corresponding market continues to grow. For home / portable beam projectors, the number of products with various form factors is increasing due to the high efficiency and low power consumption of LED light sources, and the market is growing by more than 10% to 15% per year.

[0004] In addition, with the recent increase in wired and wireless interface capabilities between beam projector devices and external devices, various contents can be selected and output in real time via Wi-Fi and 5G networks, and the contents can be shared and controlled between multiple devices. In this scenario, a multi-view function that outputs various inputs to a screen from a single projector device is necessary to expand the user experience. Summary of the invention

[0005] Solution

[0006] According to one or more embodiments, an electronic device includes an image projection unit, a memory configured to store at least one instruction, and one or more processors, wherein the one or more processors are configured to: identify characteristic information of a projection surface based on capturing an image of the projection surface by executing at least one instruction; identify first format information corresponding to a plurality of images provided in a multi-view mode; obtain layout information of a plurality of screens corresponding to the plurality of images based on the characteristic information of the projection surface and the first format information; and control the projection unit so that the plurality of images are projected onto the projection surface based on the layout information.

[0007] According to one or more embodiments, the feature information of the projection surface may include at least one of shape information, structure information, size information, color information, reflectivity information, or obstacle information.

[0008] According to one or more embodiments, the first format information corresponding to the plurality of images may include at least one of resolution information or aspect ratio information of the plurality of images.

[0009] According to one or more embodiments, the layout information of the plurality of screens may include at least one of resolution information, aspect ratio information, position information, or direction information of the plurality of screens.

[0010] According to one or more embodiments, at least one processor may be configured to perform trapezoidal correction on a projection surface based on feature information of the projection surface by executing at least one instruction, identify multiple second format information corresponding to the projection surface on which the trapezoidal correction is performed, identify loss information based on the first format information and the multiple second format information, and obtain layout information of multiple screens based on the loss information.

[0011] According to one or more embodiments, one or more processors may be configured to identify loss information corresponding to each of multiple second format information based on a similarity between the first format information and the multiple second format information by executing at least one instruction, and obtain layout information of multiple screens based on the second format information having the smallest loss information among the multiple second format information.

[0012] According to one or more embodiments, one or more processors may be configured to: identify loss information corresponding to each of resolution information and aspect ratio information included in multiple second format information based on a similarity between the first format information and the multiple second format information; identify cumulative loss information based on the loss information corresponding to each of the resolution information and the aspect ratio information; and identify second format information having the smallest cumulative loss information among the multiple second format information by executing at least one instruction.

[0013] According to one or more embodiments, one or more processors may be configured to execute at least one instruction to: identify multiple first loss information between first resolution information included in first format information and multiple second resolution information included in multiple second format information, identify multiple second loss information between first aspect ratio information included in the first format information and multiple second aspect ratio information included in multiple second format information, identify cumulative loss information corresponding to each of the multiple second format information based on the multiple first loss information and the multiple second loss information, and identify second format information having the smallest cumulative loss information among the multiple second format information based on the identified cumulative loss information.

[0014] According to one or more embodiments, one or more processors may be configured to control the image projection unit to project UI information indicating the placement status of multiple screens based on the layout information of the multiple screens by executing at least one instruction, and provide a multi-view mode based on the layout information of the multiple screens based on the UI information selected to be projected in response to a user command.

[0015] According to one or more embodiments, the UI information may include a guide UI for adjusting at least one of the size, direction, or angle of the plurality of screens.

[0016] According to one or more embodiments, one or more processors may be configured to, by executing at least one instruction, identify first format information corresponding to remaining images among the multiple images based on a position of one of the multiple images selected in response to a user command, obtain layout information of a screen corresponding to the remaining images based on feature information of a projection surface and the first format information, and control an image projection unit to provide the remaining images based on the obtained layout information.

[0017] According to one or more embodiments, one or more processors may be configured to identify the number of planes included in an image capturing a projection surface, identify an area, direction, and scale of a projection image that is valid for each identified plane, provide a guidance UI for a multi-view mode based on the identified area, direction, and scale of the valid projection image, and obtain layout information of multiple screens corresponding to multiple images based on the projection surface layout selected through the guidance UI by executing at least one instruction.

[0018] According to one or more embodiments, a control method of an electronic device includes: based on capturing an image of a projection surface, identifying characteristic information of the projection surface, identifying first format information corresponding to a plurality of images provided in a multi-view mode, obtaining layout information of a plurality of screens corresponding to the plurality of images based on the characteristic information of the projection surface and the first format information, and projecting the plurality of images onto the projection surface based on the layout information.

[0019] In a non-transitory computer-readable storage medium storing computer instructions, the computer instructions, when executed by a processor of an electronic device according to one or more embodiments, cause the electronic device to perform operations, the operations including: based on capturing an image of a projection surface, identifying feature information of the projection surface, identifying first format information corresponding to a plurality of images provided in a multi-view mode, obtaining layout information of a plurality of screens corresponding to the plurality of images based on the feature information of the projection surface and the first format information, and projecting the plurality of images onto the projection surface based on the layout information. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figures 1A to 1C are views provided to explain a method of providing a multi-view mode according to one or more embodiments;

[0021] Figure 2A is a block diagram showing a configuration of an electronic device according to an embodiment;

[0022] Figure 2B is a block diagram illustrating in detail a configuration of an electronic device according to one or more embodiments;

[0023] Figure 3 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments;

[0024] FIG. 4A to FIG. 4C are views provided to explain a method of recognizing feature information of a projection surface according to one or more embodiments;

[0025] Figure 5 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments;

[0026] Figure 6 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments;

[0027] Figure 7 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments;

[0028] FIG. 8A to FIG. 8E is a view provided to explain a method of providing a multi-view mode in a projection environment according to one or more embodiments; and

[0029] Fig. 9 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments. DETAILED DESCRIPTION

[0030] Hereinafter, the present disclosure is described in detail with reference to the accompanying drawings.

[0031] Considering their functions in the present disclosure, currently widely used general terms are selected as the terms used in the embodiments of the present disclosure, and may be changed based on the intention of those skilled in the art or the emergence of judicial precedents, new technologies, etc. In addition, in specific cases, there may be terms arbitrarily selected by the applicant. In this case, the meanings of these terms are mentioned in detail in the corresponding description of the present disclosure. Therefore, the terms used in the embodiments of the present disclosure need to be defined based on the meanings of the terms and contents throughout the present disclosure rather than the simple names of the terms.

[0032] In the present disclosure, expressions “having”, “may have”, “including”, “may include”, etc. indicate the existence of corresponding features (e.g., values, functions, operations, or components such as parts), and do not exclude the existence of additional features.

[0033] In the present disclosure, the expressions "A or B", "at least one of A or / and B", "one or more of A or / and B", "at least one of A or B", "at least one of A and B", "at least one of A, B and C", or "at least one of A, B or C" and the like may include any and all combinations of one or more items listed together. For example, the expressions "A or B", "at least one of A and B", or "at least one of A or B" may refer to all of the following cases: (1) including at least one A, case (2) including at least one B, case (3) including both at least one A and at least one B. As additional examples, the expressions "at least one of A, B or C" or "at least one of A, B and C" may refer to all of the following cases: (1) including at least one A, case (2) including at least one B, case (3) including at least one C, case (4) including both at least one A and at least one B, case (5) including both at least one A and at least one C. Case (6) including both at least one B and at least one C, case (7) including at least one of A, at least one B and at least one C.

[0034] The expressions "first", "second", etc. used in the present disclosure may indicate various components regardless of the order and / or importance of the components. These expressions are only used to distinguish one component from another component and do not limit the corresponding components.

[0035] In the case where any component (e.g., a first component) is mentioned as being "coupled with" (operably or communicatively) "to" another component (e.g., a second component) or "connected to" another component (e.g., a second component), it should be understood that any component is directly coupled to another component or may be coupled to another component through yet another component (e.g., a third component).

[0036] Depending on the circumstances, the expression "configured (or set) to" used in the present disclosure may be replaced by expressions such as "suitable for", "capable of", "designed to", "adapted to", "manufactured to", or "capable of". The term "configured (or set) to" may not necessarily mean "specially designed to" in hardware.

[0037] In some cases, the expression "the device is configured to" may mean that the device is "capable" of working with other devices or components. For example, "a processor configured (or set) to perform A, B, and C" may represent a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by executing one or more software programs stored in a memory device.

[0038] Unless otherwise clearly stated in the context, singular terms may include their plural forms. It should be understood that the terms "comprising", "formed by...", etc. used in this application specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof mentioned in the specification, and do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0039] In an embodiment, a "~ module" or "~ device" may perform at least one function or operation and may be implemented by hardware or software, or by a combination of hardware and software. In addition, multiple "~ modules" or multiple "~ devices" may be integrated into at least one module and may be implemented by at least one processor (not shown) except for the "~ modules" or "~ devices" that need to be implemented by specific hardware.

[0040] Meanwhile, various elements and regions in the drawings are schematically drawn. Therefore, the technical concept of the present disclosure is not limited by the relative sizes or intervals drawn in the drawings.

[0041] Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

[0042] Figures 1A to 1C is a view provided to explain a method of providing a multi-view mode according to one or more embodiments.

[0043] When a multi-view function is performed using a projector, a user can perform a multi-view mode (or a multi-view function) by selecting a layout of the multi-view and a plurality of input sources desired to be output to the layout using a remote controller. Here, the multi-view mode may be a mode for providing a plurality of images received via a plurality of input sources on a plurality of split screens.

[0044] In the case of a TV, an input source corresponding to a fixed aspect ratio and size is selected according to the size of the TV screen and the corresponding layout, while in the case of a projector, the screen size can vary according to the projection environment.

[0045] For example, Figure 1A As shown, depending on the environment in which the projection surface is located, there may be a situation where obstacles 20 and 30 exist on the projection surface 10.

[0046] In this case, the problem is that when adjusting the size of the projection surface 10 to avoid obstacles 20, 30, such as Figure 1B As shown, the user may not be able to view the screen at a desired resolution for the plurality of images 11 , 12 provided in the multi-view mode.

[0047] In this case, by considering the position and size of the screen suitable for providing images in the multi-view mode, it is considered that Figure 1CThe illustrated environment of the projection surface may provide an optimal multi-view mode when providing a plurality of images 11 , 12 .

[0048] Therefore, hereinafter, various embodiments will be described that can enhance the UX experience of a user by providing a multi-view layout suitable for the projection environment whenever the projection environment changes.

[0049] Figure 2A is a block diagram showing a configuration of an electronic device according to an embodiment.

[0050] refer to Figure 2A , the electronic device 100 may include an image projection unit 110, a memory 120, and one or more processors 130. The electronic device 100 may be implemented as a projector that projects an image onto a wall or a projection surface, or any other type of device having image projection capabilities. For example, the electronic device 100 may be implemented as a movable projector that can be moved.

[0051] The image projection unit 110 may perform a function of projecting light to the outside to represent an image and outputting the image to a projection surface. Here, the projection surface may be a part of a physical space where the image is output, or may be a separate projection surface. The image projection unit 110 may include various detailed configurations, such as a light source of at least one of a lamp, an LED or a laser, a projection lens, a reflector, and the like.

[0052] The image projection unit 110 may project an image in one of various projection methods, such as a cathode ray tube (CRT) method, a liquid crystal display (LCD) method, a digital light processing (DLP) method, a laser method, etc. The image projection unit 110 may include at least one light source.

[0053] The image projection unit 110 can output images with a screen ratio of 4:3, a screen ratio of 5:4, and a wide screen ratio of 16:9 depending on the purpose of the electronic device 100 and the user's settings, and can output images with various resolutions such as WVGA (854*480), SVGA (800*600), XGA (1024*768), WXGA (1280*720), WXGA (1280*800), SXGA (1280*1024), UXGA (1600*1200), full HD (1920*1080), etc. according to the screen ratio.

[0054] In addition, the image projection unit 110 may perform various functions to adjust the projected image under the control of the processor 130. For example, the image projection unit 110 may perform a zoom function, a lens shift function, and the like.

[0055] The memory 120 may store information required by various embodiments. The memory 120 may be implemented in the form of a memory embedded in the electronic device 100' or in the form of a memory detachable from the electronic device 100 according to the data storage purpose. For example, data for driving the electronic device 100 may be stored in a memory embedded in the electronic device 100', and data for the extended functions of the electronic device 100 may be stored in a memory detachable from the electronic device 100. Meanwhile, the memory embedded in the electronic device 100 may be implemented as at least one of a volatile memory (e.g., a dynamic RAM (DRAM), a static RAM (SRAM), or a synchronous dynamic RAM (SDRAM)) and a non-volatile memory (e.g., a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g., a NAND flash memory or a NOR flash memory), a hard disk drive, or a solid state drive (SSD)). In addition, the memory that can be removed from the electronic device 100' can be implemented in the form of a memory card (e.g., Compact Flash (CF), Secure Digital (SD), Micro Secure Digital (Micro-SD), Mini Secure Digital (Mini-SD), Extreme Digital (xD), or Multimedia Card (MMC)), an external memory that can be connected to a USB port (e.g., a USB memory), etc.

[0056] The one or more processors 130 control the overall operation of the electronic device 100. Specifically, the one or more processors 130 may be connected to each configuration of the electronic device 100 to control the overall operation of the electronic device 100. For example, the one or more processors 130 may be electrically connected to the display 110 and the memory 120 to control the overall operation of the electronic device 100. The one or more processors 130 may be composed of one or more processors.

[0057] The one or more processors 130 may execute at least one instruction stored in the memory 120 to perform operations of the electronic device 100 according to various embodiments.

[0058] The one or more processors 130 may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors 130 may control one or any combination of other components of the electronic device, and may perform operations or data processing related to communication. The one or more processors 130 may execute one or more programs or instructions stored in a memory. For example, the one or more processors may execute a method according to an embodiment by executing one or more instructions stored in a memory.

[0059] In the case where the method according to an embodiment of the present disclosure includes multiple operations, the multiple operations can be performed by one processor or multiple processors. For example, in the case where the first operation, the second operation, and the third operation are performed by the method according to the embodiment, the first operation, the second operation, and the third operation can all be performed by the first processor, or the first operation and the second operation can be performed by the first processor (e.g., a general-purpose processor), and the third operation can be performed by the second processor (e.g., an artificial intelligence dedicated processor).

[0060] One or more processors 130 may be implemented as a single-core processor including one core, or may be implemented as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). In the case where one or more processors 130 are implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include a processor internal memory, such as a cache memory or an on-chip memory, and the multi-core processor may include a public cache shared by multiple cores. In addition, each of the multiple cores (or some of the multiple cores) included in the multi-core processor may independently read and execute program instructions for implementing the method according to one or more embodiments, or all (or some) of the multiple cores may be linked to each other to read and execute program instructions for implementing the method according to one or more embodiments.

[0061] In the case where the method according to the embodiment includes multiple operations, the multiple operations may be performed by one of the multiple cores included in the multi-core processor, or may be performed by multiple cores. For example, in the case where the first operation, the second operation, and the third operation are performed by the method according to the embodiment, the first operation, the second operation, and the third operation may all be performed by the first core included in the multi-core processor, or the first operation and the second operation may be performed by the first core included in the multi-core processor, and the third operation may be performed by the second core included in the multi-core processor.

[0062] In addition, the electronic device 100 can perform calculations of functions related to artificial intelligence by using multiple cores (e.g., dual cores or quad cores) included in one processor. Specifically, the electronic device 100 can perform artificial intelligence calculations such as convolution calculations and matrix multiplication calculations in parallel using the multiple cores included in the processor.

[0063] In an embodiment of the present disclosure, a processor may refer to a system on chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor. Here, the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, a machine learning accelerator, etc., but the embodiments of the present disclosure are not limited thereto. Hereinafter, for ease of explanation, one or more processors 130 will be referred to as processor 130.

[0064] Figure 2B is a block diagram illustrating in detail the configuration of an electronic device according to one or more embodiments.

[0065] refer to Figure 2B , the electronic device 100' may include an image projection unit 110, a memory 120, one or more processors 130, a camera 140, a communication interface 150, a user interface 160, and a sensor 170. Figure 2B As shown in Figure 2A The configuration shown in the figure is redundant for any configuration.

[0066] The camera 140 may be turned on in response to a preset event to perform shooting. The camera 140 may convert a captured image into an electrical signal and generate image data based on the converted signal. For example, an object may be converted into an electrical image signal via a semiconductor optical device (CCD), and the converted image signal may be amplified, converted into a digital signal, and then signal processed. For example, the camera 140 may include at least one of a general (or basic) camera or an ultra-wide-angle camera.

[0067] The communication interface 150 may also be implemented as various interfaces according to exemplary embodiments of the electronic device 100'. For example, the communication interface 150 may communicate with an external device, an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through communication methods such as Bluetooth, AP-based Wi-Fi (wireless LAN network), Zigbee, wired / wireless local area network (LAN), wide area network (WAN), Ethernet, IEEE 1394, high-definition multimedia interface (HDMI), universal serial bus (USB), mobile high-definition link (MHL), audio engineering society / European Broadcasting Union (AES / EBU), optical, coaxial, etc. According to an embodiment, the communication interface 150 may perform communication with other electronic devices, external servers, and / or remote control devices, etc.

[0068] The user interface 160 may be implemented as devices such as a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen capable of performing the above-described display function and manipulation input function.

[0069] The sensor 170 may include various types of sensors, such as a touch sensor, a proximity sensor, an acceleration sensor (or a gravity sensor), a geomagnetic sensor, a gyroscope sensor, a pressure sensor, a position sensor, a distance sensor, an illumination sensor, etc. The distance sensor is configured to detect the distance from the projection surface. The distance sensor may be implemented as various types of sensors, such as an ultrasonic sensor, an infrared sensor, a LIDAR sensor, a RADAR sensor, and a photodiode sensor, etc. A geomagnetic sensor or a gyroscope sensor may be used to obtain yaw information.

[0070] Figure 3 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments.

[0071] according to Figure 3 In the illustrated embodiment, the processor 130 may identify feature information of the projection surface based on capturing an image of the projection surface (S310). For example, the feature information of the projection surface may include at least one of shape information, structure information, size information, color information, reflectivity information, or obstacle information of the projection surface.

[0072] For example, the processor 130 may obtain feature information of the projection surface by using a predetermined analysis method such as edge analysis, texture analysis, color analysis, etc. For example, the processor 130 may use at least one of object recognition, object detection, object tracking, or image segmentation to identify obstacle information included in the captured image. For example, the processor 130 may use techniques such as semantic segmentation (which classifies and extracts objects included in the image by type as needed), instance segmentation (which recognizes objects by classifying objects of the same type), and when an object included in the image is detected, a rectangular bounding box including the detected object may be used to identify obstacle information.

[0073] For example, the processor 130 may obtain feature information of the projection surface by inputting the captured image into the trained first neural network model. For example, when the captured image is input, the first neural network model may be trained to output at least one of shape information, structure information, size information, color information, reflectivity information, or obstacle information of the projection surface.

[0074] The processor 130 may identify first format information corresponding to a plurality of images provided in a multi-view mode (S320). Here, the multi-view mode may be a mode in which a plurality of images received via a plurality of input sources are provided on a plurality of split screens. For example, the first format information corresponding to the plurality of images may include at least one of resolution information or aspect ratio information of each of the plurality of images. For example, the processor 130 may obtain resolution information and aspect ratio information of each of the plurality of images based on at least one of the characteristics of each of the plurality of input sources available in the multi-view mode or at least one of the characteristics of the images provided via each of the plurality of input sources.

[0075] The processor 130 may obtain layout information of a plurality of screens corresponding to a plurality of images based on the characteristic information of the projection surface and the first format information ( S330 ).

[0076] For example, the layout information of the multiple screens may include at least one of resolution information, aspect ratio information, position information, or direction information of the multiple screens. For example, the processor 130 may obtain the layout information of the multiple screens suitable for the first format information based on the feature information of the projection surface.

[0077] For example, the processor 130 may obtain layout information of the multiple screens by inputting feature information of the projection surface and the first format information into the trained second neural network model. For example, when the feature information of the projection surface and the first format information are input, the second neural network model may be trained to output at least one of resolution information, aspect ratio information, position information, or direction information of the multiple screens.

[0078] For example, the processor 130 may obtain feature information of the projection surface by inputting the captured image and the first format information into the trained third neural network model. For example, when the captured image and the first format information are input, the second neural network model may be trained to output at least one of resolution information, aspect ratio information, position information, or direction information of the plurality of screens.

[0079] The processor 130 may control the image projection unit 110 to provide a plurality of images based on the layout information of the plurality of screens (S340). For example, the processor 130 may control the image projection unit 110 to provide a plurality of images based on the resolution information and the aspect ratio information of the plurality of screens.

[0080] FIG. 4A to FIG. 4C is a view provided to explain a method of recognizing feature information of a projection surface according to one or more embodiments.

[0081] According to one or more embodiments, the processor 130 may identify at least one of shape information, structure information, size information, color information, reflectivity information, or obstacle information of the projection surface, and calculate the number of regions available on the projection surface based on the identified information. For example, the processor 130 may identify multiple regions (or segmented surfaces) that can be provided on the projection surface based on the features of the projection surface. For example, the processor 140 may identify the projection surface as multiple regions through straight structure recognition, color recognition, reflectivity difference recognition, etc. within the projection surface.

[0082] For example, when identifying a planar projection surface, such as Figure 4A As shown, a plurality of areas 411 , 412 that may be provided based on at least one of a position or a size of an obstacle included in the projection surface 410 may be identified.

[0083] For example, when identifying a planar projection surface, such as Figure 4B As shown, a plurality of areas 413 , 414 may be identified that may be provided based on at least one of a location or a size of an obstacle included in the projection surface 410 .

[0084] For example, when identifying a planar projection surface, such as Figure 4C As shown, a plurality of areas 415 , 416 may be identified that may be provided based on at least one of a location or a size of an obstacle included in the projection surface 410 .

[0085] Figure 5 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments.

[0086] according to Figure 5In the illustrated embodiment, the processor 130 may identify first format information corresponding to a plurality of images provided in a multi-view mode (S510). For example, the first format information corresponding to the plurality of images may include at least one of resolution information or aspect ratio information of each of the plurality of images. For example, the processor 130 may obtain resolution information and aspect ratio information of each of the plurality of images based on the characteristics of each of the plurality of input sources available in the multi-view mode.

[0087] The processor 130 may identify feature information of the projection surface based on capturing the image of the projection surface (S520). For example, the feature information of the projection surface may include at least one of shape information, structure information, size information, color information, reflectivity information, or obstacle information of the projection surface.

[0088] The processor 130 may perform a trapezoidal correction on the projection surface based on the characteristic information of the projection surface (S530). For example, the electronic device 100 having the ability to project an image (i.e., a projector function) may display the screen in a relatively accurate proportion when the projector is in a straight line with the projection surface, but may project a screen that deviates from the projection surface, or may project a diamond-shaped screen that is distorted upward, downward, leftward, rightward, or leftward when the above conditions are not met due to space limitations. In this case, trapezoidal correction may be required. Trapezoidal correction refers to an adjustment function that is like forcibly moving the corners of the displayed screen (i.e., the projection screen) through the projection screen to be closer to the original shape of the square. Various conventional methods can be used as trapezoidal correction methods, so their detailed description will be omitted.

[0089] The processor 130 may identify second format information corresponding to the projection surface on which the trapezoidal correction is performed (S540). For example, the second format information corresponding to the plurality of images may include at least one of resolution information or aspect ratio information of each segmented surface included in the projection surface on which the trapezoidal correction is performed. For example, the second format information may include resolution information and aspect ratio information of a projection effective area after the trapezoidal correction. Here, the projection effective area may mean an area that can be projected by avoiding obstacles in the projection surface.

[0090] For example, the processor 130 may be based on FIG. 4A to FIG. 4C The available screen configurations of the projection surfaces are shown to identify resolution information and aspect ratio information of each split screen included in each projection surface.

[0091] The processor 130 may identify loss information based on the first format information and the second format information (S550).For example, the first format information corresponding to the plurality of images may include at least one of resolution information or aspect ratio information of each of the plurality of images.

[0092] The processor 130 may obtain layout information of the plurality of screens based on the loss information (S560). For example, the layout information of the plurality of screens may include at least one of resolution information, aspect ratio information, position information, or direction information of the plurality of screens.

[0093] According to an embodiment, the processor 130 may identify loss information (or difference information) corresponding to each of the plurality of second format information based on the similarity between the first format information and the plurality of second format information. Here, the loss information may be information based on the magnitude (or amount) of the difference between each of the plurality of second format information relative to the first format information. For example, the processor 130 may obtain layout information of the plurality of screens based on the second format information having the smallest loss information among the plurality of second format information.

[0094] For example, the processor 130 may identify loss information corresponding to each of the resolution information and the aspect ratio information included in the plurality of second format information based on the similarity between the first format information and the plurality of second format information. The processor 130 may identify cumulative loss information based on the loss information corresponding to each of the resolution information and the aspect ratio information, and identify the second format information having the smallest cumulative loss information among the plurality of second format information.

[0095] For example, the processor 130 may identify the resolution loss information based on the similarity between the first resolution information included in the first format information and the second resolution information included in each of the plurality of second format information. For example, the resolution loss information may be calculated as resolution loss=(Loss1+Loss2+...+LossN). Here, Loss1(resolution)=input source #1 resolution-resolution that can be displayed in segmented area #1, Loss2(resolution)=input source #2 resolution-resolution that can be displayed in segmented area #2, and LossN(resolution)=input source #N resolution-resolution that can be displayed in segmented area #N.

[0096] For example, the processor 130 may identify aspect ratio loss information based on the similarity between the first aspect ratio information included in the first format information and the second aspect ratio information included in each of the plurality of second format information. For example, the aspect ratio loss information may be calculated as R aspect ratio loss = (Loss1 + Loss2 + ... + LossN). Here, Loss1 (aspect ratio) = (H, V) ratio of input source #1 / (H, V) ratio of segmented region #1, Loss2 (aspect ratio) = (H, V) ratio of input source #2 / (H, V) ratio of segmented region #2, and LossN (aspect ratio) = (H, V) ratio of input source #N / ​​(H, V) ratio of segmented region #N.

[0097] For example, the processor 130 may identify the cumulative loss information based on the resolution loss information and the aspect ratio loss information, and identify the second format information having the minimum cumulative loss information among the plurality of second format information. For example, the processor 130 may identify the second format information capable of expressing the information of the input image source with the minimum loss.

[0098] For example, the cumulative loss information may be calculated as arg_min(a*resolution loss+b*aspect ratio loss) (where a=1-b).

[0099] The processor 130 may project a plurality of images based on the layout information of the plurality of screens (S570).

[0100] According to an embodiment, the processor 130 may control the image projection unit 110 to project UI information indicating the placement status of the multiple screens based on the layout information of the multiple screens. Subsequently, when the projected UI information is selected in response to a user command, the processor 130 may provide a multi-view mode based on the layout information of the multiple screens. Here, the UI information may include a guide UI for adjusting at least one of the size, direction, or angle of the multiple screens.

[0101] Figure 6 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments.

[0102] according to Figure 6 In the illustrated embodiment, the processor 130 may identify feature information of the projection surface based on capturing an image of the projection surface (S310). For example, the feature information of the projection surface may include at least one of shape information, structure information, size information, color information, reflectivity information, or obstacle information of the projection surface.

[0103] The processor 130 may identify first format information corresponding to a plurality of images provided in a multi-view mode. For example, the first format information corresponding to the plurality of images may include at least one of resolution information or aspect ratio information of each of the plurality of images.

[0104] When the position of one of the plurality of images is selected in response to a user command (S630: Yes), the processor 130 may identify first format information corresponding to the remaining images of the plurality of images (S640). For example, the processor 130 may identify at least one of resolution information, aspect ratio information, position information, or direction information corresponding to the remaining images.

[0105] The processor 130 may obtain layout information of the screen corresponding to the remaining image based on the feature information of the projection surface and the first format information (S650). For example, when the resolution and aspect ratio of one of the multiple images are selected in response to a user command, the processor 130 may fix the projection area of ​​the corresponding image and obtain layout information of the screen corresponding to the remaining image based on the size of the remaining area of ​​the projection surface and the resolution and aspect ratio of the remaining images of the multiple images.

[0106] The processor 130 may control the image projecting unit 110 so that the image selected by the user is provided in the selected area and the remaining images are provided based on the layout information of the plurality of screens (S660).

[0107] Figure 7 is a flowchart provided to explain a control method of an electronic device according to one or more embodiments.

[0108] according to Figure 7 In the illustrated embodiment, the processor 130 may identify the number of planes included in the projection surface based on capturing an image of the projection surface (S710). For example, the processor 130 may identify the number of planes by detecting corner points in the image of the projection surface.

[0109] The processor 130 may identify an area, a direction, and an aspect ratio (or scale) of a valid projection image for each identified plane (S720).

[0110] The processor 130 may provide a guide UI for the multi-view mode based on the area, direction, and aspect ratio (or ratio) of the identified effective projection image (S730). For example, the processor 130 may provide a guide UI in the form of thumbnails that allows the user to recognize at a glance the placement status of multiple screens available in the multi-view mode.

[0111] The processor 130 may obtain layout information of a plurality of screens corresponding to a plurality of images based on the projection surface layout selected via the guide UI (S740).

[0112] The processor 130 may project a plurality of images based on the layout information of the plurality of screens (S750).

[0113] FIG. 8A to FIG. 8E is a view provided to explain a method of providing a multi-view mode in a projection environment according to one or more embodiments.

[0114] refer to Fig. 8A , the processor 130 may identify the number of planes in the image capturing the projection surface through corner detection.

[0115] For example, Fig. 8A As shown, if the number of angular coordinates is identified as four, it can be identified as a single-side projection. In addition, if the number of angular coordinates is identified as six, it can be identified as a two-side projection. In addition, if the number of angular coordinates is identified as seven, it can be identified as a three-side projection.

[0116] refer to Figure 8B , the processor 130 may estimate the projection area, projection direction, and aspect ratio (or scale) of the effective image based on the individual surface by performing geometric calibration.

[0117] For example, Figure 8B As shown in The ratio can be estimated as front / left / 400×300 / approximately 4:3, the surface The ratio can be estimated as front / right / 300×300 / approximately 1:1, and the surface The ratio can be estimated as bottom / center / 400×300 / approximately 4:3.

[0118] refer to Figure 8C and Fig.8D , the processor 130 may provide the user with information about the multi-view configurations available in the projection surface and may allow the user to select a multi-view configuration. Figure 8C and Fig.8D As shown, the processor 130 can provide guidance on the projection surface to allow the user to select three surfaces. or two surfaces (ab, ac or bc).

[0119] For example, if the user selects fewer surfaces than the number of detected valid surfaces, the processor 130 may process the unselected surfaces differently from the other surfaces. For example, the processor 130 may process the unselected surfaces with a specific color, such as black (or a similar color). For example, if the user does not want the bottom surface to be available as a projection surface, the processor 130 may process the bottom surface with black so that only the surface and surface Can be used as a projection surface.

[0120] For example, the processor 130 may provide an adjustment guide for adjusting the projection surface to reduce the number of effective projection surfaces and increase the effective area. For example, an arrow-shaped adjustment guide may be provided.

[0121] refer to Fig. 8E When entering the multi-view mode based on a preset event, the processor 130 may automatically select a plurality of image sources to be provided to the plurality of surfaces selected by the user, or may automatically recommend a multi-view screen configuration based on information about the plurality of image sources selected by the user. For example, the preset event may include an event of selecting the multi-view mode or an event of selecting a plurality of image sources.

[0122] For example, Fig. 8E As shown, the processor 130 may automatically select the two surfaces (surface ), or can automatically recommend based on information about the two image sources selected by the user. Fig. 8E For example, the processor 130 may preferentially place the input source with the highest resolution or the input source best suited for the 16:9 ratio on the surface. and the input source (e.g., smartphone mirroring) that best fits the 9:16 ratio may be preferably placed on the surface In this case, the processor 130 may calculate the loss information based on the combination. For example, the loss function may be calculated as loss=Σ(recommended placement image specification-input source specification). For example, the resolution loss may be calculated as resolution loss=Σ(input resolution-resolution that can be output by the projector), and the ratio loss may be calculated as ratio loss=Σ(input source ratio / ratio that can be output by the projector).

[0123] According to an embodiment, when there is no input source for a specific screen ratio, for example, when there is only an input source for an image ratio of 4:3 or 16:9, a UI for guiding projector screen adjustment may be provided. For example, the UI may include a UI for adjusting a display direction and / or a display angle.

[0124] Fig. 9is a flowchart provided to explain a control method of an electronic device according to one or more embodiments.

[0125] according to Fig. 9 In the illustrated embodiment, when the features of the projection surface are identified ( S910 ), the processor 130 may perform keystone correction on the projection screen based on the identified feature information ( S920 ).

[0126] For example, the processor 130 may extract factors that allow a single plane to be perceived as multiple planes by human visual characteristics. For example, the processor 130 may calculate multiple segmented surfaces by considering the features of the projection surface in the projector and the factors that the human eye can recognize as separate segmented surfaces based on the features. According to the characteristics of human vision, when a single content screen is output, one or more of corners, some linear shapes / structures, colors, and reflectivity differences may be detected within the projection surface. This may cause visibility distortions that interfere with the perception of a single screen. These characteristics may be used as a hint to activate a multi-view function that organizes multiple input image sources into multiple planes, even if they are projected onto a single plane.

[0127] For example, the processor 130 may perform a distortion correction operation, such as a keystone correction, to output a rectangular or square shaped screen on multiple planes. Once the keystone correction is completed, the projection surface on which the keystone correction is performed may be divided into multiple plane areas.

[0128] The processor 130 may identify specification information of the plurality of input sources. For example, the processor may identify resolutions and aspect ratios of the plurality of input sources and store the identified information in the memory 120 (S930).

[0129] The processor 130 may identify effective area information that may be output to the projection surface after keystone correction, and store the identified information in the memory 120 (S940).

[0130] The processor 130 may calculate the loss information based on the specification information of the plurality of input sources and the effective area information that can be output to the projection surface stored in the memory 120 (S950). For example, the processor 130 may identify the specification information of the plurality of input sources (e.g., the first format information) and the plurality of effective area information (e.g., FIG. 4A to FIG. 4C loss information between the second format information shown).

[0131] Based on the calculated loss information, the processor 130 may divide the projection surface area into regions (S960) and match each input source with the divided region (S970). For example, when a divided region with the minimum loss is identified, the processor 130 may match each input source with the corresponding divided region.

[0132] When each input source provides a UI matching the divided area and performs user confirmation and / or adjustment via the UI (S980), the processor 130 may perform a multi-view operation (S990). For example, when the user changes the position of a specific image to a specific divided area, the processor 130 may re-identify a layout that minimizes the loss while keeping the corresponding image in the corresponding divided area, and provide it to the user for confirmation.

[0133] According to an embodiment, the processor 130 may provide a UI that recommends a multi-view layout with minimal loss and source placement for display on each surface. For example, the processor 130 may provide a source placement for each surface in the form of a thumbnail, and when a user selects the provided thumbnail, a multi-view mode may be provided based on the corresponding source placement.

[0134] According to an embodiment, when receiving a user command to reposition an image source, the processor 130 may guide viewing constraint information such as screen size, color, etc. based on the repositioning, and provide a multi-view mode based on the repositioning once the user confirms the repositioning.

[0135] According to the various embodiments described above, when a multi-view mode is provided using a projector, the multi-view mode can be provided in a layout suitable for the viewing space, thereby improving user convenience. In addition, the UX experience can be improved not only in a home environment, but also in a business-to-business environment such as inside a cafe.

[0136] Meanwhile, the methods according to the above various embodiments may be implemented only by software upgrade or hardware upgrade of the existing electronic device.

[0137] Alternatively, the various embodiments described above may be performed by an embedded server included in the electronic device or an external server of the electronic device.

[0138] At the same time, according to an embodiment, the various embodiments described above may be implemented in software including instructions stored in a machine-readable storage medium, which may be read by a machine (e.g., a computer). The machine may be a device that calls the stored instructions from a storage medium and operates based on the called instructions, and may include a display device (e.g., a display device (A)) according to an embodiment. In the case where the instructions are executed by a processor, the processor may directly execute the functions corresponding to the instructions, or other components may execute the functions corresponding to the instructions under the control of the processor. The instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" indicates that the storage medium is tangible and does not include a signal, and does not distinguish whether the data is stored in the storage medium semi-permanently or temporarily.

[0139] In addition, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided in a computer program product. The computer program product is a commodity and may be traded between a seller and a buyer. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed online through an application store (e.g., PlayStoreTM). In the case of online distribution, at least a portion of the computer program product may be stored in or at least temporarily generated in a storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0140] In addition, each component (e.g., module or program) according to the various embodiments described above may include singular or plural objects, and some of the corresponding subcomponents described above may be omitted, or other subcomponents may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single object that performs the same or similar functions as those performed by each of the corresponding components before integration. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be performed in a different order, omitted, or other operations may be added.

[0141] Although preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the above-mentioned specific embodiments, and various modifications may be made by a person skilled in the art without departing from the spirit of the present disclosure as claimed in the claims, and these modifications should not be understood as being isolated from the technical concept or prospect of the present disclosure.

Claims

1. An electronic device, comprising: Image projection unit; a memory configured to store at least one instruction; as well as One or more processors configured to, by executing at least one instruction: Based on capturing the image of the projection surface, identifying feature information of the projection surface, identifying first format information corresponding to a plurality of images provided in a multi-view mode, obtaining layout information of a plurality of screens corresponding to a plurality of images based on the characteristic information of the projection surface and the first format information, and The image projection unit is controlled so that the plurality of images are projected onto the projection surface based on the layout information.

2. The electronic device according to claim 1, wherein: The feature information of the projection surface includes at least one of shape information, structure information, size information, color information, reflectivity information or obstacle information. The first format information corresponding to the plurality of images includes at least one of resolution information or aspect ratio information of the plurality of images, and The layout information of the plurality of screens includes at least one of resolution information, aspect ratio information, position information, or direction information of the plurality of screens.

3. The electronic device according to claim 1, wherein: The one or more processors are configured to, by executing at least one instruction: performing keystone correction on the projection surface based on feature information of the projection surface; identifying a plurality of second format information corresponding to a projection surface on which keystone correction is performed; identifying loss information based on the first format information and the plurality of second format information; and Layout information of the plurality of screens is obtained based on the loss information.

4. The electronic device according to claim 3, wherein: The one or more processors are configured to, by executing at least one instruction: identifying loss information corresponding to each of the plurality of second format information based on a similarity between the first format information and the plurality of second format information; and Layout information of the plurality of screens is obtained based on the second format information having the least loss information among the plurality of second format information.

5. The electronic device according to claim 3, wherein: The one or more processors are configured to, by executing at least one instruction: identifying loss information corresponding to each of resolution information and aspect ratio information included in the plurality of second format information based on a similarity between the first format information and the plurality of second format information; identifying cumulative loss information based on loss information corresponding to each of the resolution information and the aspect ratio information; as well as Second format information having the smallest cumulative loss information among the plurality of second format information is identified.

6. The electronic device according to claim 3, wherein: The one or more processors are configured to, by executing at least one instruction: identifying a plurality of first loss information between the first resolution information included in the first format information and a plurality of second resolution information included in the plurality of second format information; identifying a plurality of second loss information between first aspect ratio information included in the first format information and a plurality of second aspect ratio information included in the plurality of second format information; identifying cumulative loss information corresponding to each of the plurality of second format information based on the plurality of first loss information and the plurality of second loss information; as well as Second format information having the smallest cumulative loss information among the plurality of second format information is identified based on the identified cumulative loss information.

7. The electronic device according to claim 1, wherein: The one or more processors are configured to, by executing at least one instruction: controlling the image projection unit to project UI information indicating placement states of the plurality of screens based on layout information of the plurality of screens; and A multi-view mode is provided based on layout information of the plurality of screens based on the projected UI information being selected in response to a user command.

8. The electronic device according to claim 7, wherein: The UI information includes a guide UI for adjusting at least one of sizes, directions, or angles of the plurality of screens.

9. The electronic device according to claim 1, wherein: The one or more processors are configured to, by executing at least one instruction: identifying first format information corresponding to remaining images of the plurality of images based on a position of one of the plurality of images being selected in response to a user command; Based on the feature information of the projection surface and the first format information, obtaining layout information of a screen corresponding to the remaining image; as well as The image projection unit is controlled to provide the remaining images based on the obtained layout information.

10. The electronic device according to claim 1, wherein: The one or more processors are configured to, by executing at least one instruction: identifying a plurality of planes included in an image capturing the projection surface; identifying an area, orientation, and scale of a projected image valid for each of the identified planes; providing a guide UI for a multi-view mode based on the identified area, direction, and scale of the valid projection image; as well as Based on the projection surface layout selected through the guide UI, layout information of a plurality of screens corresponding to a plurality of images is obtained.

11. A method for controlling an electronic device, comprising: Based on the captured image of the projection surface, identifying feature information of the projection surface; identifying first format information corresponding to a plurality of images provided in a multi-view mode; obtaining layout information of a plurality of screens corresponding to a plurality of images based on the feature information of the projection surface and the first format information; as well as A plurality of images are projected onto a projection surface based on the layout information.

12. The control method according to claim 11, wherein: The feature information of the projection surface includes at least one of shape information, structure information, size information, color information, reflectivity information or obstacle information. The first format information corresponding to the plurality of images includes at least one of resolution information or aspect ratio information of the plurality of images, and The layout information of the plurality of screens includes at least one of resolution information, aspect ratio information, position information, or direction information of the plurality of screens.

13. The control method according to claim 11, wherein: Obtaining layout information for multiple screens includes: performing keystone correction on the projection surface based on feature information of the projection surface; identifying a plurality of second format information corresponding to a projection surface on which keystone correction is performed; identifying loss information based on the first format information and the plurality of second format information; and Layout information of the plurality of screens is obtained based on the loss information.

14. The control method according to claim 13, wherein: identifying the loss information includes identifying the loss information corresponding to each of the plurality of second format information based on similarities between the first format information and the plurality of second format information, and Obtaining layout information of the plurality of screens includes obtaining the layout information of the plurality of screens based on second format information having the least loss information among the plurality of second format information.

15. A non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform operations comprising: Based on the captured image of the projection surface, identifying feature information of the projection surface; identifying first format information corresponding to a plurality of images provided in a multi-view mode; obtaining layout information of a plurality of screens corresponding to a plurality of images based on the feature information of the projection surface and the first format information; as well as A plurality of images are projected onto a projection surface based on the layout information.

Images